Deploying Kubernetes addons with Makefiles

This article demonstrate how applications together with their configuration can be organized in a repository and deployed using Make. In this case, all the applications that form the required eco-system to run an environment (nginx ingress controller, prometheus, rbac, pod security policies, etc.). In Kubernetes such application are called addons and are usually managed by a cluster administrator.

For the one that are impatient, the source code of such structure is available on Github.

But first, what is Make?

What is Make?

Make is an utility to maintain groups of programs. In a normal setup, Make is used to automatically determine which pieces of a large program need to be recompiled but it is not only tied to building applications. There is a bunch of other build utilies (Ant, Rake, Bazel, etc.) but Make is the most widespread amongst them.

How can it be used to deploy Kubernetes addons?

Make can be used as command line utility to execute tasks in a specific order therefore it can also be used to orchestrate the deployment of applications.

When working in a cloud-native environment, the installation and maintenance of an application varies depending on the technology. Most of the time, the configuration already exist out there (Helm charts, Kustomize packages, Terraform, AWS Cloudformation, etc.) and only a small piece of this application needs to be changed to have it adjusted to work in your own environment.

Additionally, application configuration can differ from one environment to another (security policies, role base access control, etc.) which can be difficult to manage and prone to error. Re-using configuration and only changing a piece of it for the targetted environment is essential. Keep it DRY.

How do we organize all of this?

The method described below is fairly simple and lets you decide what application and which version can be deployed to a given environment. Configuration can be shared accross environment. The steps to execute the deployment of an application are independant from each other which provide flexibility.

Each application configuration is stored in its own directory.

├── external-dns/
│   ├── Makefile
│   ├── README.md
│   ├── values.common.yaml
│   ├── values.development.yaml
│   └── values.minikube.yaml
│   └── values.production.yaml
|
├── limit-ranges/
├── namespaces/
├── nginx-ingress/
├── psp/
├── rbac/
|
├── env.development.mk                  <  Environment files
├── env.minikube.mk
├── env.production.mk
|
├── Makefile
├── README.md
└── common.mk                           < Common make functions/variables

It uses inheritance (aka overlays) where you have a common configuration (the base) and an environment configuration which override any settings from the common configuration.

The following Makefile demonstrate how the configuration is being applied to a Kubernetes cluster using kubectl.

include ../env.$(ENVIRONMENT).mk
include ../common.mk

.PHONY: deploy
deploy:
  kubectl apply -f ./$(ENVIRONMENT).yaml

Note the first includes instruction which loads configuration for the environment.

The environment configuration could look like this:

# Basic example of what the env.development.mk could look like
KUBE_CONTEXT = aks-development

# Azure Keyvault name where secrets are stored (ie: cloudflare password)
AZURE_KEY_VAULT_NAME = my-infra

# Applications to deploy. Each application is a directory in the repository.
# (order is important)
APPS = \
  namespaces \
  limit-ranges \
  nginx-ingress \
  external-dns \
  psp \
  rbac

# Ref: https://github.com/helm/charts/tree/master/stable/nginx-ingress
NGINX_INGRESS_CHART_VERSION = 1.24.3

# Ref: https://github.com/helm/charts/blob/master/stable/external-dns
EXTERNAL_DNS_CHART_VERSION = 2.6.4

The following is an example of directory structure for a Helm chart deployment:

└── external-dns
    ├── Makefile
    ├── README.md
    ├── values.common.yaml
    ├── values.development.yaml
    └── values.minikube.yaml
    └── values.production.yaml

And below is a directory structure for deploying Kubernetes manifests. It can also be used to deploy overlays using Kustomize.

└── rbac
    ├── common
    │   └── cicd-user.yaml
    ├── development
    │   ├── cicd-user.yaml
    │   ├── user-1.yaml
    │   └── user-2.yaml
    ├── minikube
    │   ├── cicd-user.yaml
    │   ├── user-1.yaml
    │   └── user-2.yaml
    ├── production
    │   └── cicd-user.yaml
    ├── Makefile
    └── README.md

How to deploy using this framework?

From this, you can provision a full environment with one command (to be executed from the root directory):

$ ENVIRONMENT=development make deploy-all

You can also deploy one application at a time:

$ ENVIRONMENT=development make deploy-nginx-ingress

The next step would be to have these tasks executed as part of your CI/CD process. For example the development environment can be applied when changes are made to the master branch of this configuration repository and the production deployment can be triggered when a tag is created via the release page in Github.

Some might find it funky. It is definitely not perfect but gets things done in a really simple way. Perfect for prototyping!

Hope that helps, make sure to check the full example on Github:

https://github.com/fikaworks/deploy-kubernetes-addons-makefile-example