From Continuous Delivery with Kubernetes by Mauricio Salatino

This article discusses using Crossplane to provision real infrastructure in a declarative way.


Take 35% off Continuous Delivery for Kubernetes by entering fccsalatino into the discount code box at checkout at manning.com.


Defining Infrastructure in a declarative way using Crossplane

Using Helm to install application infrastructure components inside Kubernetes is far from ideal for large applications and user-facing environments, as the complexity of maintaining these components and their requirements, such as advanced storage configurations, might become too complex to handle for your teams.

Cloud Providers do a fantastic job at allowing us to provision infrastructure, but they rely on cloud provider-specific tools which are outside of the realms of Kubernetes.

In this article, we look at one such tool; a CNCF project called Crossplane (https://crossplane.io), which uses the Kubernetes APIs and extension points to enable users to provision real infrastructure in a declarative way, using the Kubernetes APIs. Crossplane relies on the Kubernetes APIs to support multiple Cloud Providers; this also means that it integrates nicely with all the existing Kubernetes tooling.

Crossplane Providers

Crossplane extends Kubernetes by installing a set of components called “Providers” which are in charge of understanding and interacting with cloud provider-specific services to provision these components for us.


Figure 1. Crossplane installed with GCP and AWS Providers


By installing Crossplane providers we extend the Kubernetes APIs functionality to provision external resources such as Databases, Message Brokers, Buckets and other Cloud Resources. Several Crossplane providers cover the major cloud providers such as GCP, AWS and Azure. You can find these Crossplane providers in the Crossplane Github’s organization: https://github.com/crossplane/

Once a Crossplane Provider is installed you can create provider-specific resources in a declarative way, which means that you can create a Kubernetes Resource, apply it with `kubectl apply -f`, or include the yaml file inside a Helm Chart or use a GitOps approach, where you store the yaml file in a repository and use an environment pipeline to provision these resources in an environment.

Provisioning cloud-specific resources relying on the Kubernetes APIs is a big step forward but Crossplane doesn’t stop there. If you go to look at the details of what it takes to provision a database in any major cloud provider you realize that provisioning the component is only one of the tasks involved in getting the component ready to be used. For connecting to these provisioned components, you need network and security configurations, as well as user credentials.

Crossplane Compositions

Crossplane aims to serve two different Personas: Infra Ops teams and App Ops teams. Although Infra Ops teams are Cloud Providers experts that understand how to provision cloud provider-specific components, App Ops teams know the application requirements and understand what is required from the Application Infrastructure perspective. The interesting thing about this approach is that when using Crossplane Infra Ops teams can define these complex configurations in the Cloud Provider and expose simplified interfaces for App Ops teams (closer to developers) who are interested in simple ways to provision Application Infrastructure components. For achieving these abstractions and simplified interfaces Crossplane introduced the concept of “CompositeManagedResources”.


Figure 2. Cloud Resources Composition and abstraction by Crossplane Composites


The previous diagram shows, how you can use Crossplane Composite Managed Resources to define these abstractions for different Cloud Providers. In this case, the App Ops team is interested in provisioning a new MySQL Instance. The Infra Ops team, which probably knows better the underlying details of each cloud provider, can define the DBSubnetGroup for AWS and the MySQLFirewallRule for Azure, hiding away all these details from the AppOps team.

Another important feature of Crossplane’s that you can package these compositions into an OCI image (a container) and share it as any other docker image. Once you install these Crossplane packages with configurations then App Ops team can create these simplified resources without the need of understanding or knowing which provider is available to them. This clear separation of concerns (requesting application infrastructure and defining all the configuration required to provision these components) makes Crossplane strong.

One more thing worth noticing is that Crossplane also creates a secret with the user credentials needed to connect with the provisioned resources. This deep integration with Kubernetes, allows applications that need to connect with the newly provisioned components to know where to fetch the credentials needed to access.

Crossplane Components and Requirements

To work with Crossplane Providers and CompositeManagedResources we need to understand how Crossplane components work together to provision and manage these components inside different Cloud Providers.

This section covers what Crossplane needs in order to work and how Crossplane components manage our CompositeManagedResources.

First of all, it’s important to understand that you need to install Crossplane in a Kubernetes cluster. This can be the Cluster where your applications are running or a separate cluster where Crossplane runs. This cluster has some Crossplane components that understand our CompositeManagedResources and have enough permissions on the cloud platform to provision resources on our behalf.  You might be wondering if you can install Crossplane into a KIND Cluster, and the answer is yes, but Crossplane shines if you have access to a Cloud Provider. For that reason, we use Google Cloud Platform in the next sections.


Figure 3. Crossplane in Google Cloud Platform


The previous figure shows Crossplane installed inside a Kubernetes Cluster, with the Crossplane GCP provider installed and configured to use a Google Cloud Platform account with enough rights to provision PostgreSQL and Redis instances.

Each of the Crossplane Providers available require a specific security configuration to work and an account inside the Cloud Provider where we can create resources.

Once a Crossplane Provider is installed and configured, in this case the GCP provider we can start creating resources which are managed by this provider. You can find the resources offered by each provider in the following documentation site: https://doc.crds.dev/github.com/crossplane/provider-gcp


Figure 4. Crossplane GCP supported resources


As you can see in the previous figure, the GCP Provider version 0.17.1 supports 20 different CRDs (Custom Resource Definitions) for creating resources in Google Cloud Platform. Crossplane defines each of these resources as Managed Resources. Each of these individual managed resources need to be enabled for the Crossplane Provider to have the right access to list, create and modify these resources.

Crossplane Behaviours

In contrast with installing components with Helm in our Kubernetes Clusters, we use Crossplane to interact with the cloud provider specific APIs to provision resources inside the Cloud infrastructure. This should simplify the maintenance tasks and costs related with these resources. Another important thing that the Crossplane provider (GCP provider in this case) monitors the created Managed Resources for us. These Managed Resources offer some advantages compared with the installed resources using Helm. Managed Resources have well-defined behaviors; here’s a summary of what to expect from a Crossplane Managed Resource:

  • Visible as any other Kubernetes Resource: Crossplane Managed Resources are only Kubernetes resources, which means that we can use any Kubernetes tool to monitor and query the state of these resources.
  • Continuous Reconciliation: when a managed resource is created the Provider continuously monitors the resource to make sure that it exists and that it’s working, and report back the status to the Kubernetes resource. The parameters defined inside the managed resource are considered the desired state (source of truth) and providers work to apply these configurations into the Cloud Provider resources. Once again, we can use standard Kubernetes tools to monitor change in state and trigger remediation flows.
  • Immutable Properties: providers are in charge of reporting back if a user manually changed properties in the cloud provider. The idea here’s to avoid configuration drifts from what was defined to what is running in the cloud provider. If true, the state is reported back to the managed resource. Crossplane won’t delete the cloud provider resource, but it notifies back to allow actions to be taken. Other tools like terraform (https://www.terraform.io) automatically delete the remote resources in order to recreate them.
  • Late initialization: some properties in the Managed Resources can be optional, meaning that each provider selects the default values for these properties. When this happens Crossplane creates the resource with the default values and then sets the selected values into the Managed Resource. This simplifies the amount of configuration needed to create resources and reuses the sensible defaults defined by cloud providers usually in their user interfaces.
  • Deletion: when deleting a Managed Resource, the action is immediately triggered in the cloud provider, but the Managed Resource is kept until the resource is fully removed from the Cloud Provider. Errors that might happen during deletion on the cloud provider are added to the Managed resource status field.
  • Importing existing resources: Crossplane doesn’t necessarily needs to create the resources to manage them. You can create Managed Resources that start monitoring components which were created previously before Crossplane was installed. You can achieve this by using a specific Crossplane annotation on the Managed Resource: crossplane.io/external-name

To summarize the interactions between Crossplane, the Crossplane GCP Provider and our Managed Resources, let’s look at the following diagram:


Figure 5. Lifecycle of Managed Resources with Crossplane


The following points indicates the sequence observed in Figure 5:

  1. First, we need to create a resource, we can use any tool to create Kubernetes resources, `kubectl` here’s an example.
  2. If the resource that we created is a Crossplane Managed Resource, let’s imagine a CloudSQLInstance resource, the specific Crossplane Provider picks it up and manages it.
  3. The first step to execute when managing a resource is checking if it exists in the infrastructure (which is in the configured GCP account). If it doesn’t exist the Provider sends a request for the resource to be created in the infrastructure. Depending on the properties set on the resource, for example which kind of SQL database is required, the appropriate SQL database is provisioned; imagine for the sake of the example that we chose a PostgreSQL database.
  4. The Cloud Provider after receiving the request, if the resources are enabled, proceeds to create a new PostgreSQL instance with the configured parameters in the Managed Resource.
  5. The status of the PostgreSQL is reported back to the Managed Resource, which means that we can use `kubectl` or any other tool to monitor the status of the provisioned resources. Crossplane providers keep these in sync.
  6. When the database is up and running the Crossplane Provider creates a secret to store the credentials and properties that our applications need to connect to the newly created instance
  7. Crossplane regularly checks the status of the PostgreSQL instance and updates the managed resource.

Crossplane Configuration Packages

If we create a CloudSQLInstances resource from the GCP provider we still make a strong reference to the provider. These GCP resources are detailed, as for example, it allows us to set up properties such as DiskEncryptionConfiguration, GceZone, OnPremisesConfiguration among others (https://doc.crds.dev/github.com/crossplane/provider-gcp/database.gcp.crossplane.io/CloudSQLInstance/[email protected] ) that we might want to standardize across teams or have a policy defined on how to set them.

In order to abstract away these providers specific and low-level details on the resources, Crossplane allows us to define Composite Managed Resources. These compositions allow us to compose several provider specific resources, configure them together and expose a simple interface for the App Ops teams.

You can create your packages with your own Managed Resources based on these compositions. These packages are easy to create as soon as you know who will be consuming them and what kind of resources they want to create. These packages also allow you to create the Managed Resources that can have different cloud providers implementations but expose the same simplified type for the users.

At the end of the day a package is a set of configuration files that can be packaged as an OCI container and published to Docker Hub or any other container registry.

That’s all for this article. If you want to learn more about the book, check it out on Manning’s liveBook platform here.