My experience building my first Kubernetes operator

This is the story of my experience with writing a sample Kubernetes operator

Prerequisites

Kubebuilder - this is the go-to tool for building Kubernetes operators. Running on Go, it gives you the full glimpse of how an operator works. First we need to install it:

brew install kubebuilder

Go - required version >= 1.20.0

Initialize a project

kubebuilder init --domain "atanass.dev" --owner "atanass"

Yeah, but nah... Everything would have been so cool if it worked out of the box. Instead I get this:

Error: failed to initialize project: unable to inject the configuration to "base.go.kubebuilder.io/v4": error finding current repository: could not determine repository path from module data, package data, or by initializing a module: go: cannot determine module path for source directory /Users/atanas.dichev/personal/kubernetes-operator (outside GOPATH, module path must be specified)

Luckily, the error comes with handy instructions:

Example usage:
    'go mod init example.com/m' to initialize a v0 or v1 module
    'go mod init example.com/m/v2' to initialize a v2 module

Ok, this makes sense - I need to init a go module first, before bootstrapping a kubebuilder project.

➜  /Users/atanas.dichev/personal/kubernetes-operator go mod init atanass.dev/kubernetes-operator
go: creating new go.mod: module atanass.dev/kubernetes-operator

Ok, let's try again:

kubebuilder init --domain=atanass.dev --owner "atanass"

Great success:

        
INFO Writing kustomize manifests for you to edit...
INFO Writing scaffold for you to edit...
INFO Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.21.0
INFO Update dependencies:
$ go mod tidy
Next: define a resource with:
$ kubebuilder create api
        
      

Now I see a whole lot of files and directories:

        
➜  /Users/atanas.dichev/personal/kubernetes-operator ls -la
total 136
drwxr-xr-x@ 17 atanas.dichev  staff    544 Aug 21 17:16 .
drwxr-xr-x@  9 atanas.dichev  staff    288 Aug 21 17:13 ..
drwx------@  4 atanas.dichev  staff    128 Aug 21 17:16 .devcontainer
-rw-------@  1 atanas.dichev  staff    120 Aug 21 17:16 .dockerignore
drwx------@  3 atanas.dichev  staff     96 Aug 21 17:16 .github
-rw-------@  1 atanas.dichev  staff    411 Aug 21 17:16 .gitignore
-rw-------@  1 atanas.dichev  staff    826 Aug 21 17:16 .golangci.yml
-rw-------@  1 atanas.dichev  staff   1256 Aug 21 17:16 Dockerfile
-rw-------@  1 atanas.dichev  staff  10277 Aug 21 17:16 Makefile
-rw-------@  1 atanas.dichev  staff    371 Aug 21 17:16 PROJECT
-rw-------@  1 atanas.dichev  staff   3832 Aug 21 17:16 README.md
drwx------@  3 atanas.dichev  staff     96 Aug 21 17:16 cmd
drwx------@  7 atanas.dichev  staff    224 Aug 21 17:16 config
-rw-r--r--@  1 atanas.dichev  staff   4463 Aug 21 17:16 go.mod
-rw-r--r--@  1 atanas.dichev  staff  23000 Aug 21 17:16 go.sum
drwx------@  3 atanas.dichev  staff     96 Aug 21 17:16 hack
drwx------@  4 atanas.dichev  staff    128 Aug 21 17:16 test
        
      

This is the backbone of an operator project. So it contains nothing specific. Our next step is to create the so called "api". Because of lack of imagination I will name the group of my operator "atanass":

kubebuilder create api --group atanass --version v1 --kind AppConfig

After getting a couple of prompts:

        
INFO Create Resource [y/n]
y
INFO Create Controller [y/n]
y
        
      

I now have an AppConfig type and operator in the project. The files of interest are:

• api/v1/appconfig_types.go - This holds the Custom Resource Definition schema
• internal/controller/appconfig_controller.go - contains the reconciliation logic

Our operator implementation will go into these 2 files.

Implementation

The functionality of the AppConfig operator is going to be listening for specific objects (AppConfig) and creating a ConfigMap, containing the data in those object - pretty simple.

Types

In the api/v1/appconfig_types.go file we'll see an AppConfigSpec struct type which defines the schema of the object. Let's change it to contain the following structure:

type AppConfigSpec struct {
	Key   *string `json:"key,omitempty"`
	Value *string `json:"value,omitempty"`
}

The type AppConfigStatus struct { ... } we leave empty for the sake of the POC.

Next, we execute make generate and make manifests to respectively generate the Go code for Kubernetes operators and create the CRD manifest of the AppConfig object in config/crd/bases/

Controllers

Our next step is to actually write what needs to be executed by the operator when a AppConfig object gets passed to the API server.

In the controllers/appconfig_controller.go file we'll find a Reconcile method which does the whole thing:

func (r *AppConfigReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) - lets do a drill down into what this method's signature is

• func - declare a function in Go (thank you Captain "Obvious")
• (r *AppConfigReconciler) - a pointer receiver of type AppConfigReconciler called r. The reconciler will work on this type.
• ctx context.Context - This is a context object, used for things like timeouts and cancellation. It helps manage the lifetime of the request.
• req ctrl.Request - This is information about the specific Kubernetes object that needs to be reconciled (like its name and namespace).
• (ctrl.Result, error) - The function returns two things:
  • ctrl.Result - tells the controller what to do next (for example, requeue the request).
  • error is used to report if something went wrong.

Lets start filling up the content. Here's the complete example method:

        
func (r *AppConfigReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	logger := logf.FromContext(ctx)

	var appconfig atanassv1.AppConfig
	if err := r.Get(ctx, req.NamespacedName, &appconfig); err != nil {
		if errors.IsNotFound(err) {
			// resource deleted
			return ctrl.Result{}, nil
		}
		return ctrl.Result{}, err
	}

	cmName := appconfig.Name + "-config"

	var cm corev1.ConfigMap
	err := r.Get(ctx, client.ObjectKey{Name: cmName, Namespace: req.Namespace}, &cm)
	if err != nil && errors.IsNotFound(err) {
		// Create new ConfigMap
		cm = corev1.ConfigMap{
			ObjectMeta: metav1.ObjectMeta{
				Name:      cmName,
				Namespace: req.Namespace,
			},
			Data: map[string]string{
				*appconfig.Spec.Key: *appconfig.Spec.Value,
			},
		}
		if err := r.Create(ctx, &cm); err != nil {
			return ctrl.Result{}, err
		}
		logger.Info("Created ConfigMap", "name", cmName)
	} else if err == nil {
		// Update ConfigMap if needed
		if cm.Data[*appconfig.Spec.Key] != *appconfig.Spec.Value {
			cm.Data = map[string]string{*appconfig.Spec.Key: *appconfig.Spec.Value}
			if err := r.Update(ctx, &cm); err != nil {
				return ctrl.Result{}, err
			}
			logger.Info("Updated ConfigMap", "name", cmName)
		}
	} else {
		return ctrl.Result{}, err
	}

	return ctrl.Result{}, nil
}
        
      

Build

We're ready to build it: make generate

Deployment

Ok, so far so good. We've built the operator code. Now it's time to deploy it and test it. But where do I do that 🤔?

Minikube seems like a sensible sandbox!

But we need to have a local minikube cluster first! How do we do that?

minikube start 🤯 (This of course assumes you've got it installed)

For the sake of avoiding complex set up with container registries and access, lets just build the image inside the cluster:

1. eval $(minikube docker-env) This will alter your local docker client to use minikube's API to send the Docker context to, ultimately building the image inside the cluster
2. make docker-build - build the image
3. make deploy - deploy the operator workload to the cluster

Boom! The operator is up and running... or not

kubernetes-operator-system   kubernetes-operator-controller-manager-8564c6c98c-wj7lq   0/1     ErrImagePull   0          9s

Describing the pod we can see that it failed to run because of:

Failed to pull image "controller:latest": Error response from daemon: pull access denied for controller, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Here comes a small caveat. What make deploy does in the operator's project is applying a manifest config/manager/manager.yml. Inspecting it we can see that the image used for the operator deployment is controller:latest. But why can't the cluster spin up a pod with an image that is locally available?

The answer is imagePullPolicy. It is not explicitly set in the manager.yml manifest, which automatically sets it to its default value of Always. This means that the cluster will always try to pull it from a central Docker registry, which obviously our image is not uploaded to. To fix this, we need to set the image pull policy to IfNotPresent.

Redeploy with make deploy and now we're talking:

kubernetes-operator-system   kubernetes-operator-controller-manager-67bdffc7f7-g6q6k   1/1     Running   0          21s

Test

In the config/samples/ there should be an already prepared AppConfig resource manifest to test with. Simply decorate it with the required spec:

        
apiVersion: atanass.atanass.dev/v1
kind: AppConfig
metadata:
  labels:
    app.kubernetes.io/name: kubernetes-operator
    app.kubernetes.io/managed-by: kustomize
  name: appconfig-sample
spec:
  key: some-awesome-key
  value: even-more-awesome-value
        
      

Applying this manifest should produce a new ConfigMap resource in the default namespace. Let's go:

kubectl apply -f config/samples/atanass_v1_appconfig.yaml

appconfig.atanass.atanass.dev/appconfig-sample created

Moment of truth:

kubectl get configmap
NAME               DATA   AGE
kube-root-ca.crt   1      54m

Wait, what?! Let's see what's going on in the operator's pod logs: kubectl logs -n kubernetes-operator-system kubernetes-operator-controller-manager-67bdffc7f7-g6q6k

The error states that:

2025-08-25T13:11:02Z	ERROR	controller-runtime.cache.UnhandledError	Failed to watch	{"reflector": "pkg/mod/k8s.io/client-go@v0.33.0/tools/cache/reflector.go:285", "type": "*v1.ConfigMap", "error": "failed to list *v1.ConfigMap: configmaps is forbidden: User \"system:serviceaccount:kubernetes-operator-system:kubernetes-operator-controller-manager\" cannot list resource \"configmaps\" in API group \"\" at the cluster scope"}
        

The service account which the operator pod is associated with does not have permissions to list configmaps. You gotta love Kubernetes.

In the config/rbac/role.yml file resides the ClusterRole bound to the Service Account of the pod. The first thing you would do is update the rules by adding all the verbs to the configmaps resources in group "", right? At least that was what I did at first, but when I ran make deploy they disappeared from the manifest. It turned out that those permissions are dynamically configured by controller-gen based on the comment markers in the Go code of the controller. They look like this: // +kubebuilder:rbac:

Long story short you need to add the following comment above the Reconcile method:

// +kubebuilder:rbac:groups="",resources=configmaps,verbs=get;list;watch;create;update;patch;delete
      

Now that we've redeployed the operator, when applying the sample we get a configmap. Victory!

        
kapply config/samples/atanass_v1_appconfig.yaml
appconfig.atanass.atanass.dev/appconfig-sample created
kubectl get configmaps,appconfigs
NAME                                DATA   AGE
configmap/appconfig-sample-config   1      102s
configmap/kube-root-ca.crt          1      3h34m

NAME                                             AGE
appconfig.atanass.atanass.dev/appconfig-sample   102s
        
      

The operator's pod logs confirm the creation of a ConfigMap:

2025-08-25T15:49:27Z	INFO	Created ConfigMap	{"controller": "appconfig", "controllerGroup": "atanass.atanass.dev", "controllerKind": "AppConfig", "AppConfig": {"name":"appconfig-sample","namespace":"default"}, "namespace": "default", "name": "appconfig-sample", "reconcileID": "d244f280-1a8b-48ef-bbf4-dcdf76bbdd6d", "name": "appconfig-sample-config"}

Now, let's test an update to the AppConfig resource - I will change the value of the key:

spec:
  key: some-awesome-key
  value: an-updated-version

Logs:

        
2025-08-25T16:01:40Z	INFO	Updated ConfigMap	{"controller": "appconfig", "controllerGroup": "atanass.atanass.dev", "controllerKind": "AppConfig", "AppConfig": {"name":"appconfig-sample","namespace":"default"}, "namespace": "default", "name": "appconfig-sample", "reconcileID": "004555da-be3a-4d27-9375-c8baf016480a", "name": "appconfig-sample-config"}
        
      

Result:

        
kubectl get appconfig appconfig-sample -o yaml
apiVersion: atanass.atanass.dev/v1
kind: AppConfig
metadata:
  creationTimestamp: "2025-08-25T16:01:05Z"
  generation: 2
  labels:
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/name: kubernetes-operator
  name: appconfig-sample
  namespace: default
  resourceVersion: "17762"
  uid: c138eea3-0850-41a2-a2b5-d9e93453c521
spec:
  key: some-awesome-key
  value: even-more-awesome-value
        
      

Great, updates are working. Now, let's test deleting the AppConfig resource.

Copy
kubectl delete -f config/samples/atanass_v1_appconfig.yaml && \
kubectl get appconfig

appconfig.atanass.atanass.dev "appconfig-sample" deleted
No resources found in default namespace.
      

BUT

        kubectl get configmap
NAME                      DATA   AGE
appconfig-sample-config   1      13m
kube-root-ca.crt          1      3h46m
        
      

Shouldn't the ConfigMap get deleted? No, because our code is not complete as it does not include logic what to do if a resource is not found:

        
if errors.IsNotFound(err) {
	// resource deleted
	cmName := req.Name + "-config"
	cm := &corev1.ConfigMap{}
	cmKey := client.ObjectKey{Name: cmName, Namespace: req.Namespace}
	logger.Info("AppConfig resource not found. Deleting ConfigMap", "name", cmName)
	if err := r.Get(ctx, cmKey, cm); err == nil {
		_ = r.Delete(ctx, cm)
	}
	return ctrl.Result{}, nil
}
        
      

So now the complete Reconcile method looks like this:

        
func (r *AppConfigReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	logger := logf.FromContext(ctx)

	var appconfig atanassv1.AppConfig

	if err := r.Get(ctx, req.NamespacedName, &appconfig); err != nil {
		logger.Info("Error getting AppConfig", "error", err)
		if errors.IsNotFound(err) {
			// resource deleted
			cmName := req.Name + "-config"
			cm := &corev1.ConfigMap{}
			cmKey := client.ObjectKey{Name: cmName, Namespace: req.Namespace}
			logger.Info("AppConfig resource not found. Deleting ConfigMap", "name", cmName)
			if err := r.Get(ctx, cmKey, cm); err == nil {
				_ = r.Delete(ctx, cm)
			}
			return ctrl.Result{}, nil
		}
		return ctrl.Result{}, err
	}

	cmName := appconfig.Name + "-config"

	var cm corev1.ConfigMap
	err := r.Get(ctx, client.ObjectKey{Name: cmName, Namespace: req.Namespace}, &cm)
	if err != nil && errors.IsNotFound(err) {
		// Create new ConfigMap
		cm = corev1.ConfigMap{
			ObjectMeta: metav1.ObjectMeta{
				Name:      cmName,
				Namespace: req.Namespace,
			},
			Data: map[string]string{
				*appconfig.Spec.Key: *appconfig.Spec.Value,
			},
		}
		if err := r.Create(ctx, &cm); err != nil {
			return ctrl.Result{}, err
		}
		logger.Info("Created ConfigMap", "name", cmName)
	} else if err == nil {
		// Update ConfigMap if needed
		if cm.Data[*appconfig.Spec.Key] != *appconfig.Spec.Value {
			cm.Data = map[string]string{*appconfig.Spec.Key: *appconfig.Spec.Value}
			if err := r.Update(ctx, &cm); err != nil {
				return ctrl.Result{}, err
			}
			logger.Info("Updated ConfigMap", "name", cmName)
		}
	} else {
		return ctrl.Result{}, err
	}

	return ctrl.Result{}, nil
}