To most people in the world war is something they hear about on the news and feel bad for. This was never the case for me.

I was born in the beginning of 90s in post-Soviet Armenia. Blockade and war, no electricity, no food, shelling.

Two years ago in November right in the midst of a pandemic another war broke out in my country. 44 days, over 5000 dead on our side, drone alerts and sirens in the capital. My parents were there. My Grandma as well.

Some wars are more talked about than others. Regardless of the fact, it is the most inhumane experience for anyone involved. Whether you are on the ground, know people on the ground, have family or friends or live in the region.

Fast forward a week ago today: Ukraine. Friends, relatives hiding in bomb shelters or traveling to Polish border to fight for their country. Same for Russia. My brother lives there. My co-founder, Ilia, is Russian. &nbsp;Clear skies for now, but ostracized and scared people who did not choose their kleptocratic government, who are fed Orwellian style propaganda and put to jail for using the word “war” instead of state infused “special operation”.

Everyone I talk to, those who have left and are now refugees or those living in Kharkiv or Kyiv underground, &nbsp;everyone needs and wants to work. @Ntropy we are creating open roles for those affected by this war, Ukrainians and Russians, both short and longer term. &nbsp;It will be a privilege to support you and we would be so lucky to get the talent.

Thanks to partners like Deel we can hire and pay anyone anywhere in minutes. Please share and spread the word .

Leave your details&nbsp;<a href="https://docs.google.com/forms/d/e/1FAIpQLSfZREeIcfdRg7vJkTHE0SoSbENMNsFX3pax_i4VSD5POaP6AA/viewform">here!</a>

War in Ukraine

While tech companies have embraced&nbsp;Machine Learning (ML)&nbsp;for critical tasks in the past decade, running&nbsp;ML pipelines in production is still a pain point. It’s a cross-team effort that often starts in notebooks and simple scripts, that are expected to run efficiently and robustly in production. The tale of productionizing ML models or pipelines is a long one; from models to code, data and dependencies, there are a lot of ways (and tools) to streamline the ML development lifecycle, but at the end it’s always necessary to deploy the models, provision the essential infrastructure and monitor the system.

At the core of Ntropy API there are multiple large ML models that process all incoming transactions. Some of our customers use the API in real time for time-critical processes such as approving bank transactions, and need to enrich millions of transactions per day. These use-cases bring both latency, throughput and reliability requirements that make it challenging to run our own production-scale ML system.

In this post we will delve into&nbsp;how we provisioned our ML system to optimize for low latency&nbsp;while maintaining reasonable costs. We encourage you to follow our future blog posts, as this is the first article of a series that will cover the details of our production system — infrastructure, deployment, monitoring, backend, data management and so on.

To better understand our system it is important to have some context on what we do. The core of Ntropy’s system is processing bank transactions. We receive raw transaction data from customers (descriptions, dates, amounts, etc.) and&nbsp;run the transactions through a complex pipeline of large transformer models, database searches and various other operations. The system outputs structured information about each transaction such as the merchant name and transaction category. Although our system has more functionalities, the bulk of compute is spent in processing transactions, and other features built on top of the extracted data.

Our service is provided as a REST API served by a stateless backend server. All the transaction processing happens on a separate worker pool. In the diagram below you can get an idea of the components that a worker must execute to process a single transaction:

image.png

Each component requires one or more models to run, and all of these must run in sequence since they operate on the outputs of the previous model. The whole pipeline should take at most&nbsp;200ms per transaction. The&nbsp;M1..Mn&nbsp;models are a variable subset of models that may run for each transaction depending on needed additional predictions such as the MCC code of the transaction or calculating insights at the account holder level (i.e., if a transaction is recurrent according to previously submitted transactions for that account).

While the focus of this article is the ML infrastructure component of our production system, as you can see above, the ML portion is closely linked to our server and worker system. As such, in some sections of this article we will provide some details about the rest of the system to contextualize our decisions and thought process.

One of the core principles we follow is to&nbsp;avoid over-engineering and premature optimization at all cost.&nbsp;A good start to avoid over-engineering is clearly defining both the requirements you are going to address with a particular system, and the ones you are not trying to address. Internally, we do this process through the&nbsp;use of design documents&nbsp;that&nbsp;require a clear specification of the requirements and non-requirements for any particular system, which must then be reflected in the solution.

For our ML system, besides the standard requirements for this type of systems, we’ve focused on the following:

In the actual journey to build our system, we went through several iterations, always trying to match our needs with the requirements that were understood at the time. What we present in this article is the current iteration (surely not the last) of the system.

As the backbone of all our deployments we use kubernetes (k8s), particularly a managed deployment using&nbsp;AWS EKS.&nbsp;EKS&nbsp;simplifies the cluster deployment and upgrades, as well as provisioning new nodes, while retaining control over the system. We use&nbsp;k8s&nbsp;since it fulfills all our requirements, and most importantly we have plenty of in-house experience with it.

Our backend server is implemented in&nbsp;Python, with some performance-critical code written in&nbsp;Rust,&nbsp;and most of our code is asynchronous where possible. The&nbsp;worker pool is also Python based, implemented with&nbsp;celery, and shares a lot of the code with the backend. We use&nbsp;Redis&nbsp;as both the broker and result storage for celery.

Each model we run in production is the output of a training pipeline. It is serialized to the&nbsp;ONNX&nbsp;format and made available in our main repository (we use&nbsp;git lfs&nbsp;for keeping artifacts in our repository, allowing them to be easily synchronized with code). This is particularly useful since we build our container images to deploy to production on our&nbsp;CI/CD&nbsp;pipelines, allowing us to containerize the models and generate reproducible images with all available code and models.

For the ML infrastructure of our production models,&nbsp;we chose to utilize a pre-existing service, but deploy and manage it ourselves. Particularly, we use a model inference server in nodes with&nbsp;GPUs.&nbsp;By separating model inference from the rest of the computation we can optimize the resource usage, not wasting CPU during inference and ensuring the&nbsp;GPUs&nbsp;we are paying for are actively used for what they’re optimized for.

All our models are placed in a version-controlled model repository that is replaced with the newer versions every time the models are updated.&nbsp;The model repository is embedded in the container image of the inference server&nbsp;which is swapped on every deployment, ensuring all the components are kept in sync. The inference service is then exposed internally within the cluster for any component that requires model predictions.

You can take a look at the simplified view of our transaction processing system below:

There are many available tools for model serving, with different approaches to what they are most concerned about. Some of them are focused on ease of use and standardization (<a href="https://www.bentoml.com/" target="_blank">BentoML</a>), ease of deployment (<a href="https://github.com/kserve/kserve" target="_blank">KServe</a>,&nbsp;<a href="https://www.cortex.dev/" target="_blank">Cortex</a>), deep integration with a particular framework (<a href="https://github.com/pytorch/serve" target="_blank">TorchServe</a>,&nbsp;<a href="https://github.com/tensorflow/serving" target="_blank">Tensorflow Serving</a>) and so on.

As you can see on the image, we opted for using&nbsp;<a href="https://github.com/triton-inference-server/server" target="_blank">Triton Inference Server</a>&nbsp;from&nbsp;Nvidia.&nbsp;The main reasons were:

Choosing a higher ownership solution as we did to to deploy your models is cheaper and provides more control&nbsp;but doesn’t come with batteries included.

To scale the Triton deployment we use the&nbsp;k8s&nbsp;horizontal auto-scaler&nbsp;with a combination of&nbsp;inference server queue sizes and average GPU utilization as metrics. In simple terms, scaling horizontally means increasing the throughput of the system by increasing the number of instances, whereas scaling vertically means increasing the capability of existing instances to increase throughput. The horizontal auto-scaler continuously monitors these metrics and&nbsp;requests a new Triton instance when the metrics go above certain defined thresholds.

When you deploy a service that scales automatically you need a method of load balancing to ensure that the workloads are distributed equally between all nodes that are online at any given time. Since by default we use the&nbsp;GRPC&nbsp;API&nbsp;of Triton, the standard load balancer provided by&nbsp;k8s&nbsp;<a href="https://kubernetes.io/blog/2018/11/07/grpc-load-balancing-on-kubernetes-without-tears/" target="_blank">doesn’t work correctly</a>. It works at the connection level, but&nbsp;GRPC&nbsp;uses long-lasting connections, meaning that existing&nbsp;workers&nbsp;will not connect to any&nbsp;Triton nodes&nbsp;that are dynamically scheduled.

We introduced an additional service to load balance properly: we use&nbsp;istio&nbsp;(there are other methods that work). We add an&nbsp;istio&nbsp;side-cart to each triton pod and then define an&nbsp;istio&nbsp;service to route the requests between available triton pods. This load-balancing works at the application layer, meaning that&nbsp;workers&nbsp;connect to the istio service instead of the triton pods directly. In turn,&nbsp;istio&nbsp;routes any requests to a triton pod in the pool, while being aware of any new node that is added, ensuring that we can seamlessly scale Triton up and down in realtime.

To guarantee high availability of our Triton deployment we always run more than one Triton instance in&nbsp;on-demand compute nodes. We use a simple script to regularly check the health of each triton instance through k8s health probes, which automatically attempt to resolve the issue if the service is deemed unhealthy. This setup is pretty standard for any kubernetes service and we thought it would be enough. However, we had two instances of small downtime when particular availability zones in&nbsp;AWS&nbsp;had networking issues or went down which taught us a valuable lesson to&nbsp;always spread our instances over different availability zones&nbsp;if we’re chasing multiple 9s of uptime. We also added the capability to our workers to fallback to running the models locally if Triton is out of reach in order to ensure that the system slows down but doesn’t stop completely in case our ML infrastructure goes down temporarily.

Putting it all together, we get this zoomed-in diagram of the structure of our Triton deployment:

63eea7190c16f751033c02d2_i3.png

As you’ve seen,&nbsp;Ntropy runs entirely on cloud infrastructure. There are many ways to accomplish the same task using even the same cloud provider. For example, In AWS if you want to serve your backend service, your options range from from fully serverless function engines (Lambda), to serverless compute engines (Fargate), managed cluster services (EKS) or self-managed compute virtual / bare metal nodes (EC2) and everything in between.

The right decision for the cloud infrastructure varies from company to company, and what works for us might not work for you. A good starting point to making any choice in cloud infrastructure to consider it as a two-dimensional plane of&nbsp;ownership vs cost. The&nbsp;less ownership you have the more expensive the solutions tend to be&nbsp;(managed is generally more expensive than self hosted). Having less ownership has both pros and cons. Running managed services requires less in-house expertise and less resources invested to run and maintain a service, but it limits configurability and access to low level features which might be key. Eventually, you will need to consider other factors that matter when you have a specific use case in mind (i.e., latency of each solution).

The same rule of thumb applies to ML infrastructure.&nbsp;Just on AWS, you have services that provide fully managed models (i.e. Forecast, Rekognition), to managed platforms to deploy your own models (i.e. Sagemaker). Or, on the other end of the spectrum you can roll your own solution deploying the models in CPU or GPU instances provisioned by EC2 as we do. Within this framework you should strongly consider two factors before making any choice:

You can then optimize your choice to fit with your own needs. Do you have a small team to manage the ML systems and your use-cases are standard? You should probably go for a fully managed solution. However, do you have an experienced devops team and require full control of your ML system? Then you might want to follow an approach closer to what we described here.

Simplified plot of cost vs ownership placing some popular tools for model serving on the scale. The position of each tool is an approximate suggestion, particularly because some tools like Sagemaker can be used in different ways (with different degrees of ownership), and others are limited in scope (Rekognition is focused on ML models for image and video tasks)

Moving ML models from development notebooks to production isn’t trivial. However, a lot of tooling has appeared in the last years to assist in it, ranging from managed solutions to self-hosted ones.&nbsp;What works best for each company is an exercise that requires understanding of both the use-case at hand and internal limitations. We suggest you consider the resources you have available when choosing between ownership and cost, and to not design systems for requirements that don’t exist yet.

For our system we went with a Triton inference server and a version controlled model repository, deployed with k8s on AWS GPU instances. We scaled, monitored and made the system reliable using standard k8s tools, and used istio for application level-load balancing.

We hope we could shed some light on one of the many ways you can serve your models in production, as well as the thought process behind making some of these choices. Stay tuned for more posts on our production system in the near future. If we’ve peaked your interest or you’re interested in using our API you can visit our website at&nbsp;<a href="https://ntropy.com/" target="_blank">ntropy.com</a>.

Before wrapping, I’d like to add a big thank you to&nbsp;<a href="https://www.linkedin.com/in/thymoterdoest/" target="_blank">Thymo ter Doest</a>&nbsp;and&nbsp;<a href="https://www.linkedin.com/in/iliazintchenko/" target="_blank">Ilia Zintchenko&nbsp;</a>for exhaustive proof-reading and improving the article, as well as everyone else from Ntropy that helped.

The Anatomy of our ML Production System: The Infrastructure (Part 1)

The&nbsp;<a href="https://ntropy.com/developers" target="_blank">Ntropy API</a>&nbsp;is an HTTP API that is made of a few endpoints backed by powerful Machine Learning models.

Being a startup and thus having new companies join our customer base (bringing new needs and usage patterns) means that we frequently have to change and adapt the Ntropy API. It is often said that anything you release publicly needs to be maintained forever, so great flexibility to our customer needs comes with great responsibility to maintaining backwards compatibility.

Maintaining backwards compatibility is only one side of the coin. The other issue that comes with it is to make sure that the API is simple to understand and use and that new customers use only the latest version of the API. This is where the core topic of this blog post lies, so let’s first discuss how we describe the provided API and communicate to our customers.

There are many different ways to describe an HTTP API. Fortunately a number of companies of the tech industry&nbsp;<a href="https://www.openapis.org/about" target="_blank">have gathered together</a>&nbsp;to come up with a common standard format of HTTP API representation called&nbsp;<a href="https://spec.openapis.org/oas/latest.html" target="_blank">OpenAPI</a>&nbsp;(originated as&nbsp;<a href="https://swagger.io/about/" target="_blank">Swagger</a>&nbsp;at&nbsp;<a href="https://smartbear.com/" target="_blank">SmartBear Software</a>).

OpenAPI enables various possibilities and tools like client code auto-generation for different programming languages. One of those tools that we use here at Ntropy is the&nbsp;<a href="https://swagger.io/tools/swagger-ui/" target="_blank">SwaggerUI</a>&nbsp;which allows us to present the API in a nice, commonly familiar UI which even enables our existing or potential customers to interactively try it out right from the webpage.

In addition to allowing to specify the structure of the API’s input and output, OpenAPI also provides means to define some basic validation rules for the actual values as well using&nbsp;<a href="https://json-schema.org/" target="_blank">JSONSchema</a>. As the popularity of OpenAPI grew, more and more frameworks and libraries came out built around it, one of which is&nbsp;<a href="https://github.com/spec-first/connexion" target="_blank">Connexion</a>&nbsp;by&nbsp;<a href="https://www.zalando.com/" target="_blank">Zalando</a>&nbsp;following the&nbsp;<a href="https://opensource.zalando.com/restful-api-guidelines/#api-first" target="_blank">API-first design principle</a>&nbsp;used by Ntropy.

Using Connexion we first define the API using OpenAPI and then only implement the handlers for the API endpoints. With this setup Connexion does the heavy-lifting of type and value validation of the input and output as defined in OpenAPI allowing the handler implementations to be focused on the actual work and have cleaner code mostly containing the business logic.

In addition, Connexion includes an optional distribution of SwaggerUI out-of-the box (under the&nbsp;/ui&nbsp;path) that allows us to render the API definition visually as shown in the screenshot in the previous section.

So far so good, but as with everything there are trade-offs, and we need to consider backwards compatibility.

Imagine that we have some existing version of an API and need to ship new functionality that is not compatible with the current version. Not to disrupt the users of the current version we provide the new functionality as a new endpoint(s) of the API, standard business. Let’s look at the&nbsp;<a href="https://github.com/ntropy-network/openapi-hiding-example/blob/initial-version/openapi.yaml" target="_blank">OpenAPI definition</a>&nbsp;and the&nbsp;<a href="https://github.com/ntropy-network/openapi-hiding-example/blob/initial-version/api.py" target="_blank">handlers implementation</a>&nbsp;for the initial version:

Pretty straightforward. We have two endpoints defined in the OpenAPI and appropriate handlers in the Python module using Connexion. Now let’s have a look at the resulting SwaggerUI output:

So far so good. Now let’s&nbsp;<a href="https://github.com/ntropy-network/openapi-hiding-example/pull/1/files" target="_blank">add a new shiny endpoint</a>. As usual, we define it in the OpenAPI definition and then implement the handler.

Everything is good, except that we now have both old and new endpoints in the OpenAPI definition all of which show up in the SwaggerUI:

We would like to have only the new endpoint documented in the Swagger UI while still supporting the old endpoints. One way to do that is to simply remove them from the OpenAPI definition and implement them directly in Flask, but in that case we lose all of the convenience provided by Connexion.

Another option is to&nbsp;<a href="https://github.com/ntropy-network/openapi-hiding-example/pull/2/files" target="_blank">use the&nbsp;deprecated&nbsp;flag</a>, the official part of the OpenAPI specification, which, as the name implies, indicates that the endpoint is deprecated. While visually distinctive, the&nbsp;deprecated&nbsp;endpoints still appear in the SwaggerUI like this:

There is no way to hide endpoints which is natively supported&nbsp;<a href="https://github.com/spec-first/connexion/issues/481" target="_blank">by Connexion</a>&nbsp;<a href="https://github.com/OAI/OpenAPI-Specification/issues/433" target="_blank">or SwaggerUI</a>. So we’re on our own with this and need to implement something that will allow us to leave the old endpoints in the OpenAPI while hiding it in the SwaggerUI. To do that let’s first understand how SwaggerUI works.

SwaggerUI is a HTML template rendered and served by Connexion, which when loaded in the browser, requests&nbsp;/openapi.json&nbsp;(by default, configurable via&nbsp;<a href="https://swagger.io/docs/open-source-tools/swagger-ui/usage/configuration/" target="_blank">SwaggerUI settings</a>) containing the OpenAPI definition, which in turn is generated automatically by Connexion based on the original&nbsp;openapi.yaml&nbsp;that we define as the source of our API definition.

If we could serve our own&nbsp;/openapi.json&nbsp;file but use the rest of Connexion as is, we could change whatever parts of the YAML file we would want. Handily enough, Connexion allows us to disable serving the API definition using the&nbsp;options&nbsp;argument of the&nbsp;add_api&nbsp;function. That’s a good step to start with:

Now if we try to open the SwaggerUI, we will get a 500 error. Looking at the logs we see:‍

So it looks like when the definition serving is enabled, Connexion&nbsp;<a href="https://github.com/spec-first/connexion/blob/a1dddf62d4c6df13f6b67f35071e3e94193cf7d4/connexion/apis/flask_api.py#L43-L54" target="_blank">is defining a&nbsp;_openapi_json&nbsp;endpoint</a>&nbsp;which outputs the OpenAPI definition in JSON and is used by the SwaggerUI. Let’s try to define it ourselves and then we will have total control over what SwaggerUI shows:

It works! Now we can manipulate the output as we wish, it is only a matter of inspiration and creativity. The most straightforward way to go from here would be to delete the old endpoints from the&nbsp;openapi[“paths”]&nbsp;and the job is done.

But we’re human beings and tend to forget things. If we eventually remove some hidden endpoint from the original&nbsp;openapi.yaml&nbsp;and forget to remove the manual deletion of it from our custom&nbsp;_openapi_json&nbsp;implementation, that could result in some unpleasant 500 errors.

To prevent such issues let’s stick to having the original&nbsp;openapi.yaml&nbsp;as the source of truth and introduce a new&nbsp;x-hidden&nbsp;boolean flag, set to&nbsp;true&nbsp;for the old endpoints. Our&nbsp;_openapi_json&nbsp;implementation will iterate over all endpoints and delete all those from the output that have&nbsp;x-hidden&nbsp;enabled.

Exactly what we needed! As a final step let’s make sure that the old endpoints are still working:

:replace_this_with_any_custom_slack_emoji_you_use_to_represent_joy:

Checkout the diff for the&nbsp;x-hidden&nbsp;implementation&nbsp;<a href="https://github.com/ntropy-network/openapi-hiding-example/pull/3/files" target="_blank">here</a>&nbsp;and the full code in the&nbsp;<a href="https://github.com/ntropy-network/openapi-hiding-example" target="_blank">example GitHub repository</a>.

Btw, we’re&nbsp;<a href="https://ntropy.com/company/careers" target="_blank">hiring</a>.

War in Ukraine

Other articles

The Anatomy of our ML Production System: The Infrastructure (Part 1)

Evolving an API while keeping backwards compatibility

Join hundreds of companies taking control of their transactions