Building Machine Training Models

Training a panda to jump might need 2 or 3 sessions before it learn to move from one bin to other with accuracy without falling. After this irrespective of shape, type or form of bin it can move using the skills it has learned. Machines on the other hand to learn something simple will need lot of data and scenarios exposure before they can predict the outcome with some accuracy.

This is where machine training models play in ..

For building machine models, we ran experiments using Amazon Sandbox and then trained using different ML models such as Sklearn, lgbm and gradient boosting algorithm - XGBoost. We used Model as service paradigm exposing the service using a docker container through Amazon ELB and orchestrated using Kubernetes (Rancher).

Irrespective of what you are trying to achieve from the model there are six steps to build a machine learning model -

1. Define Problem - Convert business problem that we are trying to solve to a machine learning problem. Understanding business and applying structured thinking is key in this phase.
   
2. Generate Hypothesis - Create possible business hypothesis and features needed for the model
3. Collect Data - Collect the data to test your hypothesis created in the previous step and build the model
4. Transform Data  - Clean the data to remove any outliers, remove data that is not complete from the data set and then transform data into the required format
5. Build - Build machine learning models using your preferred language, algorithm and visualization tools
   
6. Deploy - Build an MLOps process to deploy the models

We can enable 80% of MLOps principles in the current process and the 20% can be controlled by how we setup the data to start updating the models based on latest data almost real-time. 

We are using 32GB standalone compute instances for our model building. We did try out the designer tool, but with lot of custom work that is needed for our use cases we decided to use Jupyter notebooks to train and build our model. 

For building a real-time predictions or batch predictions it is best to use cluster instances which has the capability to auto scale.

More to come ... Still in progress learning.