Hands-on Kubernetes Operator Development: Kubebuilder

Hello Kubernetes users! If you're among those unfortunate people, who have to struggle with Kubernetes on a daily basis, you may find yourself needing to write an operator at some point.

That's why we wanted to start a series of blog posts on how to write a kubernetes operator from scratch, so you can follow along. Here is the outline:

• Introduction & Environment Bootstrap (this post)
• Implementing Main Reconcile Logic
• Implementing Resource Cleanup
• Implementing Webhooks
• Testing Your Operator

What is a Kubernetes Operator?

In simple terms, a Kubernetes Operator is an application-specific controller that extends the Kubernetes API to create, configure, and manage instances of complex applications. It incorporates the operational knowledge of the application, automating common tasks and providing a consistent and standardized way of deploying and running applications.

Introducing the Sample Use case

In this series, we'll be developing kubebuilder operator example, called Tenant operator. The role of this operator is to manage multi-tenant environments within a Kubernetes cluster. If you're running a multi-tenant Kubernetes cluster, you know how challenging it can be to isolate and manage resources for each tenant.

Our operator aims to simplify this process. Here's a quick overview of what our operator is going to do:

• Namespace Management: For each Tenant object created, our operator will create corresponding namespaces in the cluster. This allows us to have a separate namespace(s) for each tenant. Additionally we want to annotate each namespace with tenant's email for easier communication.
• Role Management: The operator will manage role bindings and permissions in each namespace to ensure each tenant has the necessary permissions in their respective namespaces.
• Lifecycle Management: Our operator will handle lifecycle events, such as cleanup tasks when a Tenant object is deleted.
• Conflict Resolution: We need to ensure that two tenants can't claim the same namespace

This is a sample Custom Resource Definition (CRD) specification we'll be working towards:

apiVersion: multitenancy.codereliant.io/v1
kind: Tenant
metadata:
  name: tenant-sample
spec:
  adminEmail: admin@yourdomain.com
  adminGroups:
    - tenant-sample-admins
  userGroups:
    - tenant-sample-users
    - another-group-users
  namespaces:
    - tenant-sample-ns1
    - tenant-sample-ns2
    - tenant-sample-ns3

After applying this Tenant resource, the associated operator's reconcile loop will execute the following steps:

1. It creates three namespaces as mentioned in the namespaces list under the spec section.
2. Within each namespace, it generates RoleBindings for the tenant's admin groups (adminGroups) and user groups (userGroups).
3. It annotates each namespace with adminEmail

Now that we've outlined our use case, let's jump into bootstrapping our development environment.

Setting Up Your Environment

Before we start, make sure you have the following prerequisites installed on your local machine:

• Go (version 1.20 or later)
• Docker
• Kubernetes Cluster (we will use Kind in this example)
• kubectl

Install Kubebuilder

Kubebuilder is an excellent tool for developing Kubernetes Operators. It provides scaffolding for creating and managing Operators, and it hides much of the complexity of interacting with the Kubernetes API. It's essentially a toolkit that helps you write Controllers and define CRDs, which are the main components of an Operator.

$ curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"
$ chmod +x kubebuilder && sudo mv kubebuilder /usr/local/bin/

Create a Project

Start by creating a new directory to host our operator code:

$ mkdir ~/projects/sample-tenant-operator
$ cd ~/projects/sample-tenant-operator
$ kubebuilder init --domain codereliant.io --repo codereliant.io/tenant

This command initializes a new project, setting up all the necessary base files and configuring the domain for the CRDs.

Create an API

With the project initialized, the next step is to create an API for our operator:

$ kubebuilder create api --group multitenancy --version v1 --kind Tenant
Create Resource [y/n]
y
Create Controller [y/n]
y

This command scaffolds the necessary files for our Tenant API under the api and conrollers directories.

After bootstrapping, you will notice a set of generated directories and files. Each has its specific purpose:

• Dockerfile: A Dockerfile used for building a containerized version of the operator.
• Makefile: A Makefile containing targets for building, testing, and deploying the operator.
• PROJECT: A YAML file containing project metadata.
• api/: A directory that will contain the API definitions for our CRDs.
• internal/controllers/: A directory that will contain the controller implementations.
• config/: A directory that contains different configuration on how your operator would be deployed and interact with k8s cluster
• cmd/main.go: The entrypoint of our operator.

Design the API

Our first task is to design the API for the Tenant resource. This API defines the structure of a Tenant resource, the kind of data it holds, and how Kubernetes should manage it. The API definition is created in a file named tenant_types.go inside the api/v1/ directory.

Here's a simple example of how our TenantSpec structure could look:

// TenantSpec defines the desired state of Tenant
type TenantSpec struct {
	// Namespaces are the namespaces that belong to this tenant
	Namespaces []string `json:"namespaces,omitempty"`

	// AdminEmail is the email of the administrator
	AdminEmail string `json:"adminEmail,omitempty"`

	// AdminGroups are the admin groups for this tenant
	AdminGroups []string `json:"adminGroups,omitempty"`

	// UserGroups are the user groups for this tenant
	UserGroups []string `json:"userGroups,omitempty"`
}

This structure closely mirrors our Tenant CRD specification from earlier.

Alongside TenantSpec, you'll also see TenantStatus. It's another struct that represents the status of a Tenant resource at any given time.

// TenantStatus defines the observed state of Tenant
type TenantStatus struct {
    // NamespaceCount holds the number of namespaces that belong to this tenant
    NamespaceCount int `json:"namespaceCount"`

    // AdminEmail holds the admin email
    AdminEmail string `json:"adminEmail"`
}

Kubebuilder Markers for Code Generation

Kubebuilder makes use of a tool called controller-gen for generating utility code and Kubernetes YAML. This code and config generation is controlled by the presence of special “marker comments” in Go code.

In our case we'd need to add some new kubebuilder markers in the tenant_types.go, so the generated code/yaml suits our needs

// +kubebuilder:resource:scope=Cluster
// +kubebuilder:printcolumn:name="Email",type="string",JSONPath=".status.adminEmail",description="AdminEmail"
// +kubebuilder:printcolumn:name="NamespaceCount",type="integer",JSONPath=".status.namespaceCount",description="NamespaceCount"
// Tenant is the Schema for the tenants API
type Tenant struct {
...

• // +kubebuilder:resource:scope=Cluster: This marker specifies that the scope of the Tenant CRD is cluster-wide. This means a Tenant resource is globally unique and can be accessed across the whole cluster.
• // +kubebuilder:printcolumn:name="Email",type="string",JSONPath=".status.adminEmail",description="AdminEmail": This marker customizes the kubectl output for our CRD. It adds a column "Email" that maps to the .status.adminEmail JSONPath of the Tenant object.
• // +kubebuilder:printcolumn:name="NamespaceCount",type="integer",JSONPath=".status.namespaceCount",description="NamespaceCount": Similarly, this marker also customizes the kubectl output by adding a "NamespaceCount" column that maps to the .status.namespaceCount JSONPath of the Tenant object.

Find out more about these markers here.

Test it out

As mentioned above you can use Kind to get a local cluster for testing and once you have it installed you can test your generated code.

The make manifests command generates CRD manifests, RBAC manifests, and webhook manifests if required. Here, we're using it to create CRD manifests for our Tenant operator. After running the command, you'll see a new file in the config/crd/bases directory - multitenancy.codereliant.io_tenants.yaml. This is the CRD manifest for our Tenant resource.

$ make manifests
sample-tenant-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases

Next, we can apply this manifest to our Kubernetes cluster:

$ make install 
customresourcedefinition.apiextensions.k8s.io/tenants.multitenancy.codereliant.io created

To validate the setup, let's create a sample Tenant resource. Update config/samples/multitenancy_v1_tenant.yaml to the spec we mentioned above and apply it

$ kubectl apply -f config/samples/multitenancy_v1_tenant.yaml
tenant.multitenancy.codereliant.io/tenant-sample created

$ kubectl get tenants
NAME            EMAIL   NAMESPACECOUNT
tenant-sample

Wrapping Up

That concludes the first part of our series of creating a Kubernetes operator. In this post, we have introduced the concept of a Kubernetes Operator, outlined the requirements for our Tenant operator, and bootstrapped our project using Kubebuilder.

In the next part of our series, we will dive into implementing the controller logic for the Tenant operator. This includes responding to lifecycle events of Tenant resources, managing namespaces, roles, and role bindings, as well as handling any potential conflicts in namespace usage.

The code discussed in this blog series can be found in this github repo.