Using Kubectl's new Kustomize support for per-environment deployment of cert-manager resources


Kustomize is an increasingly popular tool for generating Kubernetes manifests, and is now included with Kubectl 1.14. Rather than using templates Kustomize works by applying modifications to already valid manifests. Using this pattern, it provides various features including resource namespacing, modification of metadata, and generation of Kubernetes Secrets.

To start using Kustomize you need one or more Kubernetes manifests and a kustomization.yaml file. The kustomization.yaml file is itself a manifest, which specifies a list of resources, patches to apply, and various other options. The docs give this example file.

Manifests that give complete Kubertnetes resources are listed in the kustomization.yaml as resources. Manifests can also be used as patches, where they are strategically merged with existing resources with matching metadata.

To modify a kustomization further, or create variants of it, Kustomize uses overlays. Overlays also contain a kustomization.yaml file, and can include manifests, either as new additional resources, or to patch existing resources. The overlay’s kustomization.yaml must specify one or more bases. The bases are directories containing kustomization.yaml files, and optionally manifests. The overlay can then build on these bases, and is also dependent on them. Overlays can also specify other overlays as bases.


To demonstrate how Kustomize can be used we will deploy and certify a simple web application (Helloweb in multiple different environments using cert-manager to provision Certificates.

All of the Kustomize files can be found in a GitHub repo so you can try the demo out yourself and dig into what Kustomize can do. Please report any issues you find on the repo! The demo repo contains two directories, base/ and overlays/, as well as a

Before running the demo make sure you have Kubectl 1.14 installed.

Alternatively you can use an older Kubectl version and install Kustomize separately.

Additionally you will need access to a Kubernetes cluster with an ingress controller to deploy the app. As well as a domain (or subdomain) that you can point at your Helloweb deployment to use cert-manager.


The initial Helloweb manifests can be found in the base/ directory. They will work as they are, however a simple kustomization.yaml is used to add a label and Namespace to the resources:

kind: Kustomization

  app: helloweb

namespace: helloweb

- namespace.yaml
- deployment.yaml
- service.yaml
- ingress.yaml

Using the standalone Kustomize command the modified manifests can be generated and printed to the terminal with:

kustomize build base

With Kubectl 1.14 the equivalent command would be:

kubectl kustomize base

The manifests can be applied, either using separate kubectl and kustomize commands:

kustomize build base | kubectl apply -f -

Or using the new Kubectl 1.14 integrated Kustomize functionality:

kubectl apply -k base

Thought the rest of this guide the new method will be used, though either should work and have the same effect.

This will deploy the minimal resources to get the app up and running in a Namespace called helloweb. Wait for the Ingress to be allocated a public IP address and then enter this into a browser to confirm the app is working. It may take some time, even after the IP address is allocated, for you to be able to load the app. When it’s working it should display the message Hello, world!, with the page using plain HTTP.

Point a domain at the Ingress’ IP address. Wait for the DNS record to propegate and confirm that you can still access the page at the domain you have used.



The first overlay builds on the resources created by the base. It adds a cert-manager Issuer and Certificate, as well as modifying the Ingress to point it at the TLS Secret that cert-manager will create. This shows how Kustomize can be used to enhance sets of manifests. It also makes it easy to keep the base manifests separate from the ones responsible for adding cert-manager.

The cert-manager overlays kustomization.yaml looks like this:

kind: Kustomization

- ../../base

  app: helloweb

namespace: helloweb

- issuer.yaml
- certificate.yaml

- ingress.yaml

- cert-manager-crds.json

- cert-manager-configuration.yaml

This starts by specifying the location of the base directory which contains the basic Helloweb deployment.

Note that the base specification is the only one that can be ‘above’ the current directory. Kustomize allows for sub directories and does not enforce any specific structure, but it does not allow resources to be used from directories ‘up’ from it. This is enforced for for security reasons, for example to prevent a kustomization.yaml from pulling private information from elsewhere on a users filesystem.

The labels and Namespace are the same as in the base/kustomization.yaml, these are applied to the cert-manager resources. The overlay doesn’t inherit the labels and namespace defined in the base layer, but it is possible to override the base layer from within the overlay layer.

The resources specification is only needed for new manifests, everything specified in the base resources is also used in the overlay. This adds a cert-manager Issuer which uses Let’s Encrypt Staging, and a Certificate which uses this Issuer.

The patchesStrategicMerge allows partial YAML files to be provided, which are then patched on top of resources of the same group, version, kind and name. In this example it is used to modify the Ingress, adding the TLS specification with the Secret that will be created by cert-manager.

apiVersion: extensions/v1beta1
kind: Ingress
  name: ingress
  - hosts:
    # Change this to your own domain
    secretName: secret-demo

The top section, giving the apiVersion, kind and name is needed to match the patch to the resource it should be applied to. New items can then be specified in an additive way; so the spec: definition will match the existing definition of the same name, but the tls: key is different so this will be added without overwriting anything.

The kustomization.yaml also features a configurations specification which points to a YAML file telling Kustomize how to recognise when other resources are named in cert-manager custom resources. For example, a Certificate definition references an Issuer. If Kustomize adds a name prefix to all resources, then it also needs to add that prefix to references to those resources in the manifests.

Note that Kustomize does not install cert-manager itself, so before applying the cert-manager manifests make sure your cluster has cert-manager installed.

Modify the overlay’s manifests to use the domain you set up earlier, and enter you email address into the Issuer. Apply the cert-manager overlay like so:

kubectl apply -k overlays/cert-manager

Once the resources are deployed cert-manager should fetch a Certificate from Let’s Encrypt staging. When visiting the page it will now use HTTPS. It may give a warning about the Certificate not being trusted, this is just because it is from Let’s Encrypt’s staging environment, so the warning can be ignored.


The development overlay is designed to show how Kustomize can help with multi environment deployments. It uses the cert-manager overlay as a base, which in turn builds upon the Helloweb app.

- ../cert-manager

  app: helloweb-development

namespace: helloweb-development

nameSuffix: -development

- ingress.yaml
- issuer.yaml
- certificate.yaml

- selfsigned-issuer.yaml
- selfsigned-certificate.yaml

It overrides the Namespace and app label, setting them to helloweb-development. It also adds a nameSuffix of development to all resources. When deployed, these resources will be created from scratch in their own Namespace, so this can be done alongside a deployment of the existing resources.

Certificates must often be issued differently in different environments, so Kustomize is a great way to manage these variations. The development overlay add a self signed Issuer and Certificate, and changes the Issuer and Certificate used by the Helloweb app to use these.

kind: Issuer
  name: issuer
    secretName: ca-secret-development
  acme: null

This patch to the Issuer tells it to use the CA created by the self signed Issuer and Certificate that have been added. Importantly it also set the acme: key to null to remove the values inherited from the base, so that the Issuer can use the ca: definition instead.

In general Kustomize is only additive, and by design does not allow manifests to be removed, however the patching can be used to set parts of a manifest to null if they are no longer required.

Once again, modify the overlay’s manifests to use your own domain. It’s suggested that a subdomain like dev. is used. The overlay can then be deployed like before:

kubectl apply -k overlays/development

Because of the addition of a name suffix new resources will be created in a new namespace. Observe the new Ingress resource to find it’s external IP address and update the chosen domain’s record as before. When the Certificate is issued you can access the page using HTTPS. It will also give a warning, but inspecting the Certificate will show it was signed by the self signed CA.


Staging is another overlay based on the cert-manager overlay and designed to exist alongside the development overlay.

- ../cert-manager

  app: helloweb-staging

namespace: helloweb-staging

nameSuffix: -staging

- ingress.yaml
- issuer.yaml
- certificate.yaml

- name: ca-secret
    - secret/tls.crt
    - secret/tls.key
  type: ""

It is similar to develop, overriding the app label, Namespace and adding a different suffix. However, it changes the Issuer and Certificate to use a CA that is locally generated. This makes use of Kustomize’s Secret generator. A TLS key and Certificate can be created locally and used in the kustomization.yaml.

This overlay can be deployed and set up in the same way as the development overlay.


Production is the final overlay, it also uses the cert-manager overlay as a base. Again, it can be deployed like the previous two overlays with it’s own app label, Namespace and name suffix. It’s effect is more minimal, just changing the Let’s Encrypt ACME provider to the production version, rather than staging, to get a fully valid Certificate.

This overlay can also be deployed and set up like the previous ones. However as it requests a production valid Certificate there should be no warning when accessing it over HTTPS.


Finally, the last overlay is multi-environment. It shows the multi base feature of Kustomize. It does not perform any modifications, but by giving the development, staging and production overlays as bases it allows them to all be generated and applied at once.

kind: Kustomization

- ../development
- ../staging
- ../production


Kustomize is a very flexible and helpful tool. It’s layered approach to managing manifests makes is very well suited to managing your own variations of other peoples work, and to managing variations across environments, without requiring duplication.

As this demo shows, Kustomize makes it easy to start with a minimal app deployment, as is often provided with an app, then modify and build on this to suit your own requirements. It can also be used to separate parts of a deployment so they can be more easily worked on by different teams. All of these modifications are expressed explicitly, making it clear what the changes are, and allowing them to be put under version control.

More Reading