From Developing Microservice APIs with Python by Jose Haro Peralta

In this article we will explain what web APIs are. You will learn that a web API is a specific instance of the more general concept of Application Programming Interface (API). It is important to understand that an API is just a layer on top of an application, and that there are many different types of interfaces.


Take 40% off Developing Microservice APIs with Python by entering fccperalta into the discount code box at checkout at manning.com.


Let’s begin by defining what exactly an API is, and then we will move on to explaining the defining features of a well implemented web API.

What is an API?

We will begin this section by defining what an API is and how it is different from the application layer. We will use the example of the well-known CLI tool cURL to illustrate the concepts.

API stands for Application Programming Interface, and it defines the way we can programmatically interact with an application. It represents a contract. This means that the API tells you what you can do to interact with an application, and what you can expect from the application by executing those actions. An API is a layer on top of an application, and it must not interfere with the application logic. There are multiple types of application interfaces, such as command line interfaces (CLI), desktop UI interfaces, web UI interfaces, or web API interfaces. An application can have one or more of these interfaces, and therefore its logic should not be bound to a specific type of interface. In other words, you must avoid creating tight coupling between the application logic and the API layer (see Figure 1).


Figure 1. An application can have multiple interfaces, and therefore the application logic should be independent from the interface logic, and there should be no strong dependencies or tight coupling between the application layer and the interfaces layer. In this illustration, we show an application with three different types of interfaces: a web API interface; a command line interface; and a web UI interface.  


To illustrate this idea, think, for example, of the popular Client URL CLI (cURL). cURL is a command line interface to an application called libcurl. libcurl implements the capabilities that we need to be able to interact with URLs, while cURL exposes those capabilities through a command line interface (CLI). This CLI includes number of options that we can specify to trigger certain actions in the application. For example, we can use the -L option to instruct cURL to follow a redirect:

 
 $ curl -L http://www.google.com
  

Or we can use the -O option in order to download the contents of a URL endpoint to a file:

 
 $ curl -O http://www.gnu.org/software/gettext/manual/gettext.html
  

Adding the -C option also allows us to resume downloading content which, for some reason, was stopped. Similarly, we can use the -X, --request[1] option to specify a HTTP method, the -H option to add request headers, and the -d option to attach a payload in the request:

 
 $ curl -X POST -d '{"title": "foo", "body": "bar", "userId": 1}' -H 'Content-Type: application/json' 'https://jsonplaceholder.typicode.com/posts'
  

In all of these cases, we are using the API exposed by the libcurl application’s CLI (cURL). The API is well specified (to see it, run curl --help), and from it we know what to expect from the application when we specify different options. The libcurl application sits behind this interface, and nothing prevents us from accessing it directly through the source code[2], and even building additional types of interfaces for this application.

What is a web API?

Now that we understand what an API is, we will explain the defining features of a web API. We will explain how a web API layer works, what protocols and frameworks are available for its implementation, and how they fit within the architecture of a web application.

So, what exactly is a web API? Simply put, a web API is a web interface to an application. Web applications typically return an HTML document that can be rendered in the browser. However, that type of interface is not accessible programmatically, i.e. it does not expose commands that allow you to interact with it programmatically. With a web API, you expose an interface to the web application running in your server that can be used to interact with it programmatically.

Web APIs are implemented using a protocol or framework of choice. There are different options available for that, such as SOAP, REST, GraphQL, gRCP, Twirp, and others. Most of these API protocols run or can run on top of Hypertext Transfer Protocol (HTTP), which is the communication protocol that underpins the Internet. HTTP is designed to be able to transfer different kinds of media type over the Internet, such as text, images, video, JSON, etc. It uses the concept of Unique Resource Locator (aka URL) to find resources in the Internet, and provides utilities that can be leveraged by API technologies to enhance the interaction with the server, such as request methods (e.g. GET, POST, PUT).

The growth of web APIs has transformed the way we build applications. For one, the wide availability of APIs in the Internet means that, when building your own products, you do not have to implement all features or functionality by yourself. In many cases, you can just use one of the existing APIs offered by different providers or vendors, and integrate it with your application. This could be the case for authentication management, artificial intelligence applications such as voice recognition or natural language processing, to deliver emails, or to obtain the coordinates of a specific address. For this and many other use cases, there is a wide availability of offerings on the Internet that you can choose from.

It also means that, when building a web application, you have several choices of architecture and design. You can choose to build a traditional website with dynamically rendered views from the server, such as you would normally do with frameworks like Django. You can choose to build your application with a client-server architecture, whereby the client, typically an SPA (single page application) which interacts with the backend through an API. As part of the latter design, you can decide to open part of the backend API for public consumption (probably for a price), or you may decide to skip the frontend altogether and offer your service exclusively via web APIs.

Why does it make sense to deploy web APIs as microservices?

Now that we understand what the defining features of web APIs are, in this section we explain why it makes sense to deploy web APIs as microservices. To illustrate this idea, we resume the example of an application called CoffeeMesh (which we work with throughout the book), and we discuss how we can break it down into microservices with their own APIs. Along the way, we will explain the benefits that we gain from migrating the application to a microservices architecture.

You can deploy an API as part of a monolithic application, and for many use cases that can be perfectly fine. However, an API usually contains a collection of different endpoints, and often those endpoints represent different services or capabilities of the website. In that respect, it makes sense to break down the API into different microservices and deploy them separately.

Going back to our previous example, the CoffeeMesh Django project is already organized around applications which own different endpoints of the website. For example, the payments application is served under /payments, while the users application is served under /users, and the orders application is served under /orders. Each application encapsulates a specific business domain of the website, and owns specific segments of its data. Therefore, we could easily enhance or repurpose them to expose well defined API endpoints. We could then deploy all these applications together as part of a single monolith, and we would benefit from being able to share code and data across applications. However, as we saw earlier, this approach eventually leads to scenarios where it becomes difficult to manage the codebase, make changes to it, release new features, and scale the system.

Alternatively, we can break the applications apart and deploy them as independent services and avoid many of those problems. To be able to deploy each Django application as an independent service, we first need to remove the code dependencies among the applications. This is easier said than done, but for the purposes of the present discussion let’s assume we are able to do that and move forward. The process of breaking an application down into smaller components is called service decomposition.[3] If we migrate our CoffeeMesh Django project to a microservices architecture, the result might look like something like this (see Figure 2 for an illustration of this migration):

  • A products service which encapsulates logic and data about the products on offer and their stock availability. It exposes the /products URL path.
  • An orders service which encapsulates all logic necessary to place and manage orders. It exposes the /orders URL path.
  • A payments service to process payments. It exposes the /payments URL path.
  • A kitchen service which knows how to interface with the automated production system. It exposes the /kitchen URL path.
  • A delivery service which takes care of arranging deliveries. It exposes a /delivery URL path.
  • A users service to manage user accounts and authentication. It exposes a /user URL path.

Figure 2. In a monolithic application, all components ship together within the same codebase, while in a microservices architecture, components are deployed separately as independent applications and run in different processes.


This is not to say that, when migrating a traditional Django application to a microservices architecture, you can just simply take the applications in your Django project and deploy them as microservices. You have to consider whether each application owns a specific and well-defined business domain. It may be the case that some applications must be broken down into smaller microservices components. On the other hand, it may make more sense to merge some applications within the same service.

In the microservices scenario, communication and data sharing across services takes place through APIs. For example, to process a payment, the orders service has to interact with the service payments API. At this point, we are not specifying any details about how exactly these API calls should work. We are doing this intentionally, because such level of detail requires us to choose a specific technology for the implementation of the API, which is not the intention at this point. We are now just considering the pros and cons of microservices vs monoliths, and we should keep our considerations at a high level at this point.

In the microservices scenario, there are strict boundaries between the services. Each service owns a specific set of data, and other services cannot access such data except through the interface exposed by other services. This helps to prevent us from creating unnecessary coupling among the services at the level of the code. We are also able to scale each service independently, and to optimize the environment in which it runs. For example, it may be the case that the payments service performs tasks which are a lot more CPU intensive than the delivery service, which in turn may require more memory in order to load all the delivery details about each order, while the products service may require more optimized networking hardware in order to process quickly its numerous database queries and respond timely to each service which requires its data. It may also be the case that the kitchen service may need more resources during certain times of the day, while the rest of the time it can be idle. Thanks to the fact that each service runs in a different process and in a different environment, we are able to provision each of them with the most optimized configuration for their operations, and we are able to scale them (up and down) independently.

How Python makes it easier to develop microservice web APIs

In this section we will explain why it makes sense to implement API-driven microservices using Python. We will see that Python provides a rich ecosystem for the development of web APIs and for building microservices. We will use some of the tools discussed in this section over the course of this book, but we will not be able to provide examples of use for all of them. However, it is important that you are aware of the wide array of resources that we can choose from when implementing API-driven microservices in Python, and this section will give you a quick overview of the most relevant resources at the time of this writing.

Python is one of the most popular languages for developing web backend services. The Stack Overflow Developer Survey of 2019 ranks Django and Flask as two of the most popular web development frameworks (https://insights.stackoverflow.com/survey/2019). In fact, in terms of backend web development frameworks (that is, excluding frontend development frameworks such as Angular or React), Python is the only language represented more than once in the top 12 of the “Web Frameworks” ranking[4].

Python offers a rich ecosystem for web applications development, and in particular for APIs. If you are developing a Django website, you can use the Django REST framework, an excellent, battle-tested and very well documented framework for RESTful APIs (https://github.com/encode/django-rest-framework). Or you can use graphene-django if you intend to build a GraphQL API (https://github.com/graphql-python/graphene-django). If instead you are using Flask, you can choose between Flask-RESTful (https://github.com/flask-restful/flask-restful), Flask-RESTX (https://github.com/python-restx/flask-restx) or flask-smorest (https://github.com/marshmallow-code/flask-smorest) to build a RESTful API, or you can plug in graphene (https://github.com/graphql-python/graphene) directly into your Flask web application in order to enable GraphQL endpoints, or use Flask-GraphQL (https://github.com/graphql-python/flask-graphql) in order to facilitate the implementation. You can further use a library like marshmallow (https://github.com/marshmallow-code/marshmallow) in order to optimize your object serialization tasks. You can also use fastapi (https://github.com/tiangolo/fastapi) to build highly optimized APIs in Python, and to implement both REST and GraphQL endpoints.

The previous paragraph describes only some of the most popular choices available for web development with Python. There are many other popular libraries, such as Tornado (https://github.com/tornadoweb/tornado), Pyramid (https://github.com/Pylons/pyramid), Falcon (https://github.com/falconry/falcon), and others that we cannot cover in this book. Also, the ecosystem is growing by the day, with new choices becoming available that you should watch out for and consider. The point is that Python offers a very rich ecosystem for web application development, and therefore makes an excellent choice for the implementation of web APIs.

At the same time, Python offers a great ecosystem for building microservices. When it comes to microservices, there are multiple solutions you can use for deployments, including Docker, serverless, and managed hosting services such as Heroku or AWS Beanstalk, to name a few. Python enjoys good support in most of these platforms, and in many cases, the ecosystem provides custom tooling and possibilities for optimization.

For example, if you are going to deploy with Docker, you will be able to choose between minimal Linux distributions such as Alpine Linux, and purpose-built base images for Python. You can also consider deploying on AWS Lambda, the serverless computing platform provided by AWS, and you will be pleased to discover that, at the time of this writing, it currently supports Python versions 3.6, 3.7, and 3.8. If this is not sufficient for your needs, you can use Lambda Layers together with the Lambda Runtime API, which are additional services that AWS provides to help you to create support for additional runtimes, such as other versions of Python. When it comes to serverless deployments in AWS, you are also able to use tools such as Chalice (https://github.com/aws/chalice) and Zappa (https://github.com/Miserlou/Zappa), which are implemented in Python, thereby giving you the possibility to enhance their implementations and customize your builds. You can also deploy serverless applications written in Python to Azure using Azure Functions, and to Google Cloud using GCP Functions. If you’re considering deploying to Heroku[5], you will find that it has support for all the major Python frameworks, including Django, Flask, Pyramid, and so on.

The bottom line is that, when it comes to building and deploying microservice APIs, Python gives you a rich ecosystem, with a wide variety of frameworks that can help you boost your development productivity, as well as a great variety of tools and services to customize and optimize your deployments.

That’s all for this article. If you want to learn more about the book, you can check it out on our browser-based liveBook platform here.

 


[1] In command line applications, it is usual to offer a shorter and a longer version of the same option, separated by a comma. The shorter version is always preceded by one dash, and the longer version is preceded by two dashes. In this example, the notation -X, --request means that this option can be used as  -X or as --request.

[2] If you are curious, you can pull it from Github: https://github.com/curl/curl.

[3] Please bear in mind that the approach shown in this section is simplified. Simply deploying the applications of a Django project as independent services is not a robust approach for service decomposition. Effective service decomposition strategies will be covered in a later chapter of the book. The approach we follow in this article is an intended simplification to allow us to discuss the advantages and disadvantages of deploying web APIs as microservices.

[4] Combining Django and Flask, Python appears to be the second most popular choice for web backend development.

[5] Heroku is a cloud platform as a service (PaaS) provider. It allows you to upload your applications, and with minimal configuration get them running, while they take care of managing and provisioning computing resources. You can learn more about Heroku from their website: https://www.heroku.com/.