Comparing deployment options for Machine Learning Models in Azure
Last week I delivered a training session on how to deploy machine learning models in Azure. We only focused one example, but there are a load more. I was asked the following question:
“What is the best way to deploy Machine Learning models in Azure? ”
That question is the classic “it depends”. I wanted to take this blog to talk through some of the approaches I have used or I think could be a good option for you and your business. We are going to explore deployment methods for three main use cases.
Batch Machine Learning
This is where you have a table, file, DataFrame, some kind of data that you wish to score in batch, i.e. All at once. This kind of processing is typically done a few times a monthly, possibly daily and not used when you need to make a quick decision. Latency is not a huge factor here. We may use this output to load a decision support system. Analysing customer behaviour is a common batch operation.
2. Interactive Machine Learning
Latency is really important here. Interactive machine learning is a term which describes making an inference (score/prediction) in real-time. Imagine a scenario where you are buying a product online. When you go to complete your purchase, the website will try to offer you an incentive to buy more (alternative product, discounts, multi-buys etc). You have put hot dogs and mustard in your basket, do you want buns? If that recommendation takes a long time then the user is at risk of leaving the service, which we do no want! We need <100 millisecond response times here, it need to be so fast the user does not know that their behaviours have been through a model.
3. Interactive Batch Machine Learning
Sometimes the prediction you're making is batch prediction, but it is completed as an interactive request. This is ok, but interactive machine learning relies on a more complicated deployment and hosting architecture, which typically costs more. You may have an example where you're looking to perform micro-segmentation of your customers to better understand how to improve their interaction with your services. The calculation behind that will most likely be something you can do in batch, Person A comes to your site and always browses books, a batch process can identify them as a book customer and offer them incentives, or try to up-sell them also to an audio-book customer. Now rather than making that decision each time they visit the site, lets make it once a week and use another service to serve the score to the application. This is a batch output behaving like an interactive model, but reducing the complexity of interactive models. For a lot of customers who want interactive machine learning, most actually require interactive batch.
Our options:
With all the options presented here there is a maturity and experience curve. Depending on where your business is are and your individual needs / modelling requirements. Do you need to go straight for the most complicated, maybe, but try to understand the complexities first and decide which option is right for your team. This blog is a quick comparison between all the options in Azure - If you think I have missed something, add a comment and I will respond.
Batch:
Running scripts on an IAAS Virtual Machine
SQL Server 2017 - Running on an IAAS Virtual Machine
Azure SQL Database PAAS offering
Azure Databricks
Interactive:
Azure ML Studio
Azure ML Services
Azure Functions
Docker & Kubernetes
Interactive Batch:
Redis Cache
Cosmos DB
Batch Machine Learning
Running scripts on an IAAS Virtual Machine.
This is a pretty common scenario. You build a model in Python and deploy the scripts to a server. You use something to schedule the scripts and leave it running. This might work, but is not an ideal solution. There is a lot that you need to manage in this kind of deployment. You own the server, the OS , the network etc. That is a lot. The OS will need to be patched, you will inevitably hit problems with dependency management. Even if you use a package manager such as Conda, dependency management is hard and problematic. If you models run 24/7 then you need to pay for this 24/7. It does however give you a lot of flexibility, you can run in any language, in any way you want. Not an option which we typically recommend.
SQL Server 2017 - Running on an IAAS Virtual Machine.
When SQL Server was released in 2017, it came bundled with Machine Learning Services. This supported both R and Python. You can build a models in any tool you want then serialise it with Pickle and get SQL Server to run it in batch. The major benefit that you get with this, is that you do not move the data outside of SQL Server - a huge win for operational teams! Your code and data are stored in the same place. You do however need to think if this is a good option for you. Is all your data in the same database? Will you need to grab additional data from flat-files, other databases or call additional APIs? If so then this might not be a great option. If you find you're having to write long ETL processes to load data before you can build anything then there is a problem. This also suffers from the same dependency management problem as IAAS VM. What is does however offer is an easy way to deploy models and to run models which out incurring additional costs. It supports both R an Python, but nothing else. This is not recommended for training either.
Azure SQL Database. PAAS offering.
This now supports R but not Python. It has the same pros and cons as above, but it is PAAS. You do not have to manage a VM. There is only the cost of running the PAAS service. I imagine this will eventually support Python. If it does, then it is a great option for Batch ML, where all your data in in the same database. An interesting option, but not for us today.
Azure Databricks.
Azure Databricks is a scale out computation service which makes it very easy to build Apache Spark applications. Apache Spark comes bundled with loads of great Machine Learning packages. MLLib is part of Spark core and is fantastic for batch machine learning at scale. If you have a lot of data, then this is the option for you! Databricks also has the Spark ML run-time, which simplifies the process of managing dependencies. You can do both shallow and deep learning at scale. But you have to remember, it is not an interactive Machine learning tool. You do not want to have to distribute your data before making a decision. But Databricks is a great option for training and then porting the model in to an interactive architecture. Because of the scaled out approach, it is also fantastic for hyper-parameter tuning and cross-validation.
Interactive Machine Learning
Azure Machine Learning Studio
Studio is Microsoft's "draggy droppy, clicky clicky" Machine Learning development environment. Yes it is limited, but it is fantastic for those who want to prove out an idea and get an API which will allow a user to submit a REST request and get a response. If you're new to machine learning or you're new to deployment then this is a solid option. But you need to go in with eyes open. You will build it online, there is no option to source control and the range of functionality is really limited. But it is ideal for those getting started. It does support both R and Python, but nothing else. Deep learning is also out of the question!
Azure Machine Learning Services
Machine Learning Services is the evolution of many ML tools in Azure. It does a load which will help the ML developer. You can build models only in Python at this point, quite a big limiting factor - Most teams want to develop in the tools they are used to and not be locked down to a single language. ML Services does help you with experimentation tracking, deployment and model management. While this seems like a lot, there is still more work required, however out-of-the-box this is the only option in Azure which has anything which even looks like model management. As this tool improves I can see it becoming our deployment tool of choice, however, today this is not it. It is great again for teams who are newer to Machine Learning. You can do a lot, without a huge investment in skills. You have full blown Python, so you can use all the frameworks for deep learning. ML Services does also bring with it Project Brainwave! I have a blog on Brainwave, in a nutshell it is a card built for deep learning inference. Machine Learning Services, is one to start learning in my opinion.
Azure Functions
With Functions 2.0 we can now build functions natively in Python. When we want to give a piece of software access to our models, then a common option is via a REST API. Azure functions expose REST APIs so this seems like a fairly obvious choice for serving Machine Learning models. Well kind of. Azure Functions can take a while to spin up, once they are up they will fly, however that initial wait time may be too much for an end user. Getting dependencies deployed is also painful. The development process is not where I would like it to be. But it could be considered for some scenarios. As the development process improves this will become a decent option for deployment. Functions support other languages and not just Python, you can do JS, C# so other frameworks will also work.
Docker & Kubernetes
This option give you the most flexibility. If you can run it on Linux, you can run it on Docker. That means any language you want to write in is supported. This is key, too often I see teams being forced to be Python Developers or R developers. If you see something in a different language or a different way of working, you should have an architecture which will support this! The complexity is significantly increased. You need to know a lot of different technologies, and most importantly, you need to know how to make them all work together. When doe right this pattern gives the most about of flexibility. I run a three day training course on all the tech required to get started with Docker and Kubernetes for Machine Learning and often that is not enough time to bring a team up-to-speed. This requires a significant investment of time. But when done right it allows a team to find and provide return-on-investment for their teams development.
Interactive Batch
Redis
What Redis does it does well! If you have a batch processed file which you want to put in front of an end user, Redis is a great option. You will need to build additional functionality to host a REST API, but this is very well documented and the are multiple options in Azure for doing this. In Azure we also have a PAAS offering of Redis, so no need to manage lots of hardware etc.
Cosmos DB
Cosmos is fantastic and this is another great option for fast retrieval. If the output of your model is quite large or complex then this is a great option. Again you will need to wrap additional functionality to get a REST API.
Winners for Advancing analytics:
Batch: Databricks
Ease, Scale and choice of languages make this an easy winner.
Interactive: Docker & Kubernetes.
Build it the way you want to! The possibilities are unlimited. Yes it is harder to get started with, but when done right this approach just works and should be inline with the latest trends in application deployment.
Interactive Batch: Redis.
Redis is so popular for a reason. For interactive batch models, this is our go to.
If you need support with your Machine Learning project, or support getting a model in to production. Then get in touch to find out how we can help.