Kubernetes Operator Framework: Building Custom Controllers

Kubernetes Operators are software extensions that use custom resources to manage applications and their components. They are part of the Kubernetes control plane and act as controllers to reconcile the actual state of the cluster with the desired state. This blog will delve into the technical aspects of building custom controllers using the Kubernetes Operator Framework.

Understanding Kubernetes Operators

Kubernetes Operators are introduced to extend the functionality of Kubernetes APIs. They manage application logic and are designed to automate Day-1 tasks (installation, configuration) and Day-2 tasks (reconfiguration, upgrade, backup, failover, recovery) for applications running within a Kubernetes cluster.

Operator Pattern

The Operator pattern involves creating custom resources and controllers to manage these resources. For example, an Operator can manage a custom resource named SampleDB by ensuring a Pod is running with the controller part of the Operator, setting up PersistentVolumeClaims for durable storage, and managing regular backups.

Custom Resources and Definitions

Operators use Custom Resources (CR) and Custom Resource Definitions (CRD) to define the desired configuration and state of a specific application. The Operator's role is to reconcile the actual state of the application with the state desired by the CRD using a control loop.

Example Operator

Here is an example of how an Operator might look in more detail:

Custom Resource: A custom resource named SampleDB.
Deployment: A Deployment that ensures a Pod is running with the controller part of the Operator.
Container Image: A container image of the Operator code.
Controller Code: Code that queries the control plane to find out what SampleDB resources are configured.
Reconciliation: The core of the Operator is code that tells the API server how to make reality match the configured resources.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: sampledbs.example.com
spec:
  group: example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                size:
                  type: integer
                  minimum: 1
                  maximum: 10
            status:
              type: object
              properties:
                phase:
                  type: string
                  enum:
                    - Pending
                    - Running
                    - Failed
  names:
    plural: sampledbs
    singular: sampledb
    kind: SampleDB
    shortNames:
      - sdb

Building Custom Controllers

To build a custom controller, you need to follow these steps:

Define the Custom Resource Definition (CRD): This defines the structure of the custom resource.
Write the Controller Code: This code will reconcile the actual state with the desired state.
Deploy the Controller: Deploy the controller to the Kubernetes cluster.

Step 1: Define the CRD

The CRD defines the structure of the custom resource. Here is an example of a CRD for a SampleDB resource:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: sampledbs.example.com
spec:
  group: example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                size:
                  type: integer
                  minimum: 1
                  maximum: 10
            status:
              type: object
              properties:
                phase:
                  type: string
                  enum:
                    - Pending
                    - Running
                    - Failed
  names:
    plural: sampledbs
    singular: sampledb
    kind: SampleDB
    shortNames:
      - sdb

Step 2: Write the Controller Code

The controller code is responsible for reconciling the actual state with the desired state. Here is an example of a simple controller written in Go using the controller-runtime library:

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/operator-framework/operator-sdk/pkg/sdk"
    "github.com/operator-framework/operator-sdk/pkg/sdk/example/memcached-operator/api/v1alpha1"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/apimachinery/pkg/types"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/controller"
    "sigs.k8s.io/controller-runtime/pkg/handler"
    "sigs.k8s.io/controller-runtime/pkg/manager"
    "sigs.k8s.io/controller-runtime/pkg/reconcile"
    "sigs.k8s.io/controller-runtime/pkg/source"
)

func add(mgr manager.Manager, r reconcile.Reconciler) error {
    c, err := controller.New("sampledb-controller", mgr, controller.Options{Reconciler: r})
    if err != nil {
        return err
    }

    err = c.Watch(&source.Kind{Type: &v1alpha1.SampleDB{}}, &handler.EnqueueRequestForObject{})
    if err != nil {
        return err
    }

    return nil
}

func main() {
    scheme := runtime.NewScheme()
    err := v1alpha1.AddToScheme(scheme)
    if err != nil {
        log.Fatalf("failed to add to scheme: %v", err)
    }

    mgr, err := sdk.NewManager("sampledb-operator", sdk.ManagerOptions{
        Scheme: scheme,
    })
    if err != nil {
        log.Fatalf("failed to create manager: %v", err)
    }

    reconciler := &ReconcileSampleDB{
        client: mgr.GetClient(),
        scheme: mgr.GetScheme(),
    }

    err = add(mgr, reconciler)
    if err != nil {
        log.Fatalf("failed to add controller: %v", err)
    }

    log.Println("starting manager")
    if err := mgr.Start(context.TODO()); err != nil {
        log.Fatalf("failed to start manager: %v", err)
    }
}

type ReconcileSampleDB struct {
    client client.Client
    scheme *runtime.Scheme
}

func (r *ReconcileSampleDB) Reconcile(ctx context.Context, req reconcile.Request) (reconcile.Result, error) {
    log.Printf("reconciling SampleDB %s/%s\n", req.Namespace, req.Name)

    sampleDB := &v1alpha1.SampleDB{}
    err := r.client.Get(ctx, req.NamespacedName, sampleDB)
    if err != nil {
        return reconcile.Result{}, err
    }

    size := sampleDB.Spec.Size
    if size < 1 || size > 10 {
        return reconcile.Result{}, fmt.Errorf("invalid size: %d", size)
    }

    // Create or update resources based on the desired state
    // ...

    return reconcile.Result{}, nil
}

Step 3: Deploy the Controller

To deploy the controller, you need to build the Docker image and deploy it to the Kubernetes cluster. Here are the steps:

Build the Docker Image:

docker build -t <your-registry>/sampledb-operator:v1 .

Push the Image:

docker push <your-registry>/sampledb-operator:v1

Deploy the Controller:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sampledb-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: sampledb-operator
  template:
    metadata:
      labels:
        name: sampledb-operator
    spec:
      containers:
      - name: sampledb-operator
        image: <your-registry>/sampledb-operator:v1
        command:
        - sampledb-operator

kubectl apply -f deployment.yaml

Conclusion

Building custom controllers using the Kubernetes Operator Framework involves defining custom resources, writing the controller code, and deploying the controller to the Kubernetes cluster. This approach extends the functionality of Kubernetes APIs and automates complex tasks for managing applications. By following these steps, you can create robust and scalable solutions for Platform Engineering.