Hands-on Kubernetes Operator Development: Finalizers

Photo by Sam Pak / Unsplash

• Introduction & Environment Bootstrap

• Implementing Main Reconcile Logic

• Implementing Resource Cleanup (this post)

• Implementing Webhooks

• Testing Your Operator

---

In the first two blog posts on this topic we learned how to bootstrap your operator and how to implement the reconciliation loop.

In the third episode we will discuss an important aspect of Kubernetes resource lifecycle management - garbage collection. As we continue building our Tenant Operator, it is critical to ensure that resources allocated to a tenant are properly cleaned up when a tenant is deleted. This not only optimizes the resource usage but also avoids potential conflicts and issues that can arise from stale or orphaned resources.

Understanding Finalizers in Kubernetes

Finalizers are an essential part of Kubernetes' resource management system. They provide a mechanism to prevent specific resources from being deleted before we've performed necessary cleanup operations.

Each Kubernetes object's metadata includes a field called finalizers, which is an array of strings. These strings are arbitrary, but by convention, they take the form of domain/name. The absence of finalizers or an empty list in a resource means it can be deleted immediately by the Kubernetes API server. However, if the finalizers list contains one or more elements, the Kubernetes API server will not delete the resource, giving our operator the chance to perform any necessary cleanup operations. Here is example how the finalizer might look like:

apiVersion: multitenancy.codereliant.io/v1
kind: Tenant
metadata:
  finalizers:
  - tenant.codereliant.io/finalizer

You can read more about the finalizers in the Kubernetes reference docs.

Implementing Finalizers in Tenant Operator

In the context of our Tenant Operator, we want to ensure that when a tenant is deleted, all associated namespaces and their resources are also cleaned up. This cleanup step is critical to avoid accumulating orphaned resources. We can achieve this using a finalizer.

Let's start by defining a finalizer for our Tenant Operator. This could be done in our Tenant controller file:

const (
	finalizerName            = "tenant.codereliant.io/finalizer"
)

Next, we will add the finalizer to the Tenant resource when it is created and remove it once the cleanup is done. We can accomplish this in our Reconcile function:

func (r *TenantReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
  ...
  // Check if the Tenant is being deleted
  if tenant.DeletionTimestamp != nil {

    // Check if the finalizer is present
	if controllerutil.ContainsFinalizer(tenant, finalizerName) {
      
      // Cleanup Resources
      log.Info("Finalizer found, cleaning up resources")
      if err := r.DeleteExternalResources(ctx, tenant); err != nil {
        // retry if failed
        log.Error(err, "Failed to cleanup resources")
        return ctrl.Result{}, err
      }

      // Remove the finalizer from the Tenant object once the cleanup succeded
      // This will free up tenant resource to be deleted
	  controllerutil.RemoveFinalizer(tenant, finalizerName)
	  if err := r.Update(ctx, tenant); err != nil {
	    log.Error(err, "unable to update Tenant")
		return ctrl.Result{}, err
	  }
    }
  } else {
    ...
    // If tenant is not being deleted
    // Add the finalizer to the Tenant object if not already present
	if !controllerutil.ContainsFinalizer(tenant, finalizerName) {
		tenant.ObjectMeta.Finalizers = append(tenant.ObjectMeta.Finalizers, finalizerName)
		if err := r.Update(ctx, tenant); err != nil {
			log.Error(err, "unable to update Tenant")
			return ctrl.Result{}, err
		}
	}
  }
  ...
}

Here, when a deletion timestamp is present, it indicates the Tenant resource is being deleted. We first check if our finalizer is present. If it is, we call our DeleteExternalResources function, which will delete all resources associated with the Tenant. After successful cleanup, we remove our finalizer from the list and update the Tenant resource.

Resource cleanup

The DeleteExternalResources function is responsible for deleting all resources that were created for a specific Tenant. In our case, we want to delete all namespaces that were created for the tenant and the rest of it (like roleBindings) would be automatically deleted.

Here is an example of how we could implement this function:

func (r *TenantReconciler) DeleteExternalResources(ctx context.Context, tenant *multitenancyv1.Tenant) error {
	// Delete any external resources created for this tenant
	log := log.FromContext(ctx)
	for _, ns := range tenant.Spec.Namespaces {
		// Delete Namespace
		namespace := &corev1.Namespace{
			ObjectMeta: metav1.ObjectMeta{
				Name: ns,
			},
		}
		if err := r.Delete(ctx, namespace); client.IgnoreNotFound(err) != nil {
			log.Error(err, "unable to delete Namespace", "namespace", ns)
			return err
		}
		log.Info("Namespace deleted", "namespace", ns)
	}
	log.Info("All resources deleted for tenant", "tenant", tenant.Name)
	return nil
}

The function is simple - we loop through all the namespaces listed in the Tenant's spec and delete them one by one.

Testing

Let's quickly test the code:

# start the operator
$ make run

# delete the namespace
$ kubectl delete tenant tenant-sample 

# checking the logs
...
INFO	Reconciling tenant
INFO	Finalizer found, cleaning up resources
INFO	Namespace deleted
...
INFO	All resources deleted for tenant
INFO	Resource cleanup succeeded

What's Next?

Our Tenant Operator now not only handles the creation of namespaces and role bindings based on the Tenant resource but also ensures that they are cleaned up when the Tenant is no longer needed.

In the next part of this series, we will look into to making our operator more robust by utilizing webhooks.

The code discussed in this blog series can be found in this github repo.