How a simple admission webhook lead to a cluster outage

Jetstack often works with customers to provision multi-tenant platforms on Kubernetes. Sometimes special requirements arise that we cannot control with stock Kubernetes configuration. In order to implement such requirements, we’ve recently started making use of the Open Policy Agent project as an admission controller to enforce custom policies.

This post is a write up of an incident caused by misconfiguration of this integration.

Introducing our best-practice GKE Terraform module

Jetstack works with many customers using Google Cloud’s Kubernetes Engine (GKE). We work closely with teams to configure their clusters to conform with best practices. While GKE’s robust default settings provide an excellent abstraction of the lower level details of control plane configuration, there are still many more considerations when automating the build of a production grade cluster. Automating Cluster Deployments To ensure deployments of clusters are as reliable as possible it’s best to automate as much as possible.