Hands-on Kubernetes Operator Development: Webhooks

Photo by Morgane Perraud / Unsplash

• Introduction & Environment Bootstrap

• Implementing Main Reconcile Logic

• Implementing Resource Cleanup

• Implementing Webhooks (this post)

• Testing Your Operator

---

In previous posts, we covered bootstrapping an operator, implementing the reconciliation loop, and handling cleanup logic using finalizers. Now let's look at enhancing the robustness and reliability of our Tenant Operator using admission webhooks.

Webhooks allow our operator to intercept and validate requests before the Kubernetes API server persists changes. This gives us the power to enforce business logic, prevent conflicts, and ensure consistency.

Understanding Webhooks

Admission webhooks are HTTP callbacks that receive admission requests and can mutate or validate resources during create, update, and delete operations. They allow injecting custom logic into the Kubernetes API server request flow without modifying the core Kubernetes code.

Some key things to know about admission webhooks:

• They intercept API requests before persistence of the object in etcd, allowing mutating or validating the resources

• Two types - mutating webhooks can modify objects sent to the API server, validating webhooks can reject requests.

• Run as standalone services, not part of API server. API server POSTs admission requests to webhook over HTTP.

• Can be used to enforce custom business logic, security policies, schema validation, defaults, etc

To summarize, admission webhooks allow inserting validation, mutation and business logic checks into the Kubernetes API request flow without touching the core k8s code. This allows extending the platform's functionality and robustness.

Challenge: Namespace conflicts

In a multi-tenant environment, it is critical to keep tenants isolated from each other. For our Tenant Operator, we want to prevent a namespace from being claimed/created by more than one Tenant resource. This will avoid potential conflicts where two Tenants try to deploy resources to the same namespace.

There are multiple ways to address this, but for the sake of this tutorial we will create a validating webhook that intercepts Tenant creation and update requests.

Implementing a Validation Webhook

The webhook will intercept all create and update requests for the Tenant resource.

When such a request comes in, the webhook handler will:

1. Decode the Tenant object from the request

2. Check if the namespace specified in the Tenant already exists

3. If the namespace exists, deny the request

4. Otherwise, allow the request to proceed

First, let’s scaffold the webhooks for our CRD using kubebuilder

$ kubebuilder create webhook --group multitenancy --version v1 --kind Tenant --programmatic-validation
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
api/v1/tenant_webhook.go
api/v1/webhook_suite_test.go

In api/v1/tenant_webhook.go we define TenantValidator and pass a Client, so we can query k8s for existing resources:

// +kubebuilder:object:generate=false

type TenantValidator struct {
	client.Client
}

Set it up:

func (r *Tenant) SetupWebhookWithManager(mgr ctrl.Manager) error {
	validator := &TenantValidator{
		mgr.GetClient(),
	}

	return ctrl.NewWebhookManagedBy(mgr).
		For(r).
		WithValidator(validator).
		Complete()
}

Now to implement the validator interface we need three methods - ValidateCreate, ValidateUpdate and ValidateDelete for each type of operation with the object.

Let's use ValidateCreate as an example

// +kubebuilder:webhook:path=/validate-multitenancy-codereliant-io-v1-tenant,mutating=false,failurePolicy=fail,sideEffects=None,groups=multitenancy.codereliant.io,resources=tenants,verbs=create;update,versions=v1,name=vtenant.kb.io,admissionReviewVersions=v1

func (v *TenantValidator) ValidateCreate(ctx context.Context, obj runtime.Object) (admission.Warnings, error) {
    // read the object, make sure it's Tenant type
	tenant, ok := obj.(*Tenant)
	if !ok {
		return nil, fmt.Errorf("unexpected object type, expected Tenant")
	}

    // get the list of existing namespaces
	var namespaces corev1.NamespaceList
	if err := v.List(ctx, &namespaces); err != nil {
		return nil, fmt.Errorf("failed to list namespaces: %v", err)
	}

	for _, ns := range tenant.Spec.Namespaces {
		// Check if namespace already exists
		if namespaceExists(namespaces, ns) {
			return nil, fmt.Errorf("namespace %s already exists", ns)
		}
	}

	return nil, nil
}

// Check if a namespace exists in a list
func namespaceExists(namespaces corev1.NamespaceList, ns string) bool {
	for _, namespace := range namespaces.Items {
		if namespace.Name == ns {
			return true
		}
	}
	return false
}

// Check if a namespace is contained in a list
func contains(namespaces []string, ns string) bool {
	for _, n := range namespaces {
		if n == ns {
			return true
		}
	}
	return false
}

This code snippet validates a create operation for our Tenant object. It verifies the object type, lists the existing namespaces, and checks for the existence of namespaces specified in the tenant.Spec.Namespaces field, ensuring that they don't already exist.

Let's test it!

Previously we tested the controller with simple make run on our local environment, but it's a bit more tricky when dealing with webhooks. In order to properly test it you would need:

• install cert-manager in your k8s environment (e.g. via cmctl x install)

• build a docker image of your controller

• upload docker image either to docker registry or load it directly to your k8s environment

• update kustomization configuration to enable webhooks and cert-manager

• run `make deploy`

Once it's done kubebuilder would take care of generating, provisioning and deploying the right configuration of your operator. Here is how my command looks like with Kind k8s cluster

$ make manifests && \
make install && \
make docker-build && \ 
kind load docker-image controller:latest && \
make deploy && \
stern sample-tenant-operator-controller -n sample-tenant-operator-system

And voila! You should have your operator running within the k8s environment with webhooks configured to prevent the namespace conflicts:

$ kubectl get pods -n sample-tenant-operator-system
NAME                                                         READY   STATUS    RESTARTS   AGE
sample-tenant-operator-controller-manager-5cc6d77cd5-8phpn   2/2     Running   0          84s

To verify that the webhook works, create the Tenant resource and use a namespace that already exists in the cluster:

apiVersion: multitenancy.codereliant.io/v1
kind: Tenant
metadata:
  name: tenant-sample2
spec:
  adminEmail: admin@yourdomain.com
  adminGroups:
    - tenant-sample-admins
  userGroups:
    - tenant-sample-users
    - another-group-users
  namespaces:
    - tenant-sample-ns1

Applying it should confirm that the operation can't be performed due to the webhook protection:

$ kubectl apply -f config/samples/multitenancy_v1_tenant2.yaml
Error from server (Forbidden): error when creating "config/samples/multitenancy_v1_tenant2.yaml": admission webhook "vtenant.kb.io" denied the request: namespace tenant-sample-ns1 already exists

Wrapping up

In this post we demonstrated how to leverage admission webhooks in operators to enforce business logic and improve it's robustness.

In the upcoming, final chapter, we will explore in detail how to test your operator and webhook, stat tuned.

The code discussed in this blog series can be found in this github repo.