Machine Learning on Google Cloud using the AI Platform (Custom Model)
Train and evaluate a neural network classification model on the MNIST dataset using Keras and Tensorflow on AI Platform
This is the final story in a 3-part series to perform Machine Learning on Google Cloud. In this story, we will focus on training a custom neural network classification model on the MNIST Dataset on the GCP AI Platform. In order to get started, I would encourage you to review the first story to ensure you meet the pre-requisities and to know more about the MNIST Dataset. The first story can be found here.
AI Platform: It is fully managed, end-to-end platform for data science and machine learning on Google Cloud.
The diagram below gives a high-level overview of the stages in an ML workflow. The blue-filled boxes indicate where AI Platform provides managed services and APIs. Here is more information on AI Platform.
What is covered in this story?
In this tutorial, we will setup a Jupyter Notebook on the AI Platform and train a neural network classification model on a labeled MNIST dataset using Keras and Tensorflow. Later, we will evaluate its performance and review a prediction on the test dataset generated by this custom model.
Step 1: Download the MNIST Notebook from GitHub.
Go ahead and download the MNIST Notebook from GitHub onto your local machine. We will later review the details of this notebook and also upload it to a Jupiter Notebook instance on AI Platform.
Step 2: Setup Jupyter Notebook on AI Platform.
Log on the GCP console and confirm that you have a $300 trial credit from the Billing page.
Look up AI Platform in products and resources search bar. Click on AI Platform.
Click on Notebooks.
Click on New Intance to create a new compute instance.
We will use Tensorflow 2.4. This instance comes pre-loaded with all the python libraries needed to train this model. We are training a simple model on a dataset with 60K samples. Therefore, we should be good without using GPUs.
On the below screen, use the default options as listed and click on Create.
It may take a few minutes to spin up the instance. Once the instance is ready, the Open Jupyterlab link shown below should get activated.
Step 3: Upload the MNIST notebook.
Click on Open Jupyterlab link.
Let’s upload the MNIST notebook previously downloaded on your local machine from GitHub.
Click on the uploaded notebook.
Now, go ahead and click on Run from the top menu and select Run All Cells to execute all the cells in this notebook.
Step 4: Quick review of the Model.
- Import MNIST Dataset: To start with we import the MNIST dataset directly from Keras and then split it into training and test sets.
- Normalize: Looking at one of the training instances, we can observe that the pixel intensities for an image range from 0 and 255. We should normalize the dataset so that the values are scaled between 0 and 1. Normalizing the data speeds up the learning and leads to faster convergence in a neural network.
- Configure the Model: The next step is to build and configure the model. In this stage, we specify the different layers of the neural network. We define the optimizer and the loss function our neural network should optimize. Note that during training, the primary goal of a neural network is to minimize the loss function. We also tell the model that we would like to evaluate accuracy as the metric during the training process.
- Model Training: Once the model is configured, we are ready to begin the training phase. We are running 4 epochs. Running one epoch is like training the neural network with all of its training data exactly once. We are training with 60,000 samples. With every epoch, we can observe that the loss is minimized which in turn results into improving the accuracy.
Please spend some time to review the notebook as it contains additional information with regards to the configuration of the model.
Step 5: Model Evaluation & Predictions.
Accuracy, Precision and Recall: The results are quite promising! The training accuracy is ~98.3% with precision and recall both at ~97.4%.
Confusion Matrix: From the confusion matrix, we can observe that the model is fairly consistent in its predictions except in the case of #3 and #7 where the accuracy is marginally lower.
With regards to predictions, we have displayed one of the handwritten test samples. It was correctly predicted as 7. You may experiment with other samples as well by simply changing the index of x_test and y_pred in cell #12.
Step 6: Delete the Notebook Instance.
Do not forget to delete the instance after the completion of this tutorial.
Conclusion:
AI Platform enables you train, evaluate and generate predictions from your custom model. It also gives you more control in terms of defining the hyper-parameters and tuning them to optimize your model’s performance.
Thank you again for taking time to review this story. Hope you liked it!
Here are links to the first two stories in this series:
- Machine Learning on Google Cloud using Big Query ML
- Machine Learning on Google Cloud using AutoML Vision
Word of Caution:
Please be careful while using the services on Google Cloud to stay within the $300 trial budget. Do not forget to turn off APIs, deleting any un-used instances or clearing off the cloud storage bucket after use so as to avoid any additional costs billed to your account.
