Configuring MySQL SSL/TLS authentication with cert-manager

We recently worked on a customer project, where they wanted to secure the connection between their Java Spring Boot application and their MySQL Database, all this running on Google Kubernetes Engine (GKE). We suggested they use cert-manager, our preferred certificate management tool on Kubernetes.

It was the first time that we would use cert-manager to secure MySQL for a customer project and during the deployment of it we encountered some challenges.

In this post, we will show you how to secure the connection between a Java Spring Boot application and MySQL in a Kubernetes environment with cert-manager.

Deploy cert-manager

What if someone intercepted sensitive data that is stored in your database? That is why it is important to add TLS (it is common to use the term SSL but today usually refers to TLS) encryption between your app and the MySQL server. To do so, we need to configure MySQL to use encrypted connections using TLS certificates provisioned by cert-manager.

Install all cert-manager’s resources and CustomResourceDefinitions. We do it using regular manifests for Kubernetes, but you can find more ways of installation here.

# Kubernetes 1.15+
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.0/cert-manager.yaml

You can verify it is installed correctly by checking the cert-manager pods running:

kubectl get pods --namespace cert-manager

Install Issuers

Now that we have the certificate management tool deployed correctly, let’s get the Issuers which will represent certificate authorities (CAs) that are able to generate signed certificates. In this case we will setup a CA and SelfSigned issuer.

An Issuer is a namespaced resource, so you will need to create the issuers in the same namespace as your application and database resources. You can create a ClusterIssuer instead if you want to be able to request certificates from any namespace.

SelfSigned Issuer

This is useful to generate a root CA for use with the CA Issuer.

Create this manifest locally and apply it to your cluster.

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}
$ kubectl apply -f selfsigned-issuer.yaml
issuer.cert-manager.io "selfsigned-issuer" created

CA Issuer

We’re generating an internal certificate authority (CA) that will be used to sign incoming certificate requests for the MySQL instance. Both MySQL server and the client will rely on the CA to to make sure a veritable connection.

Certificate CA

In order to create the CA issuer, you must first create a self signed CA, which will be issued by issuer selfSigned.

Note that isCA is set to true in the body of the spec.

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: ca-certificate
spec:
  secretName: ca-cert
  duration: 2880h # 120d
  renewBefore: 360h # 15d
  commonName: MySQL admin
  isCA: true
  keySize: 2048
  usages:
    - digital signature
    - key encipherment
  issuerRef:
    name: selfsigned-issuer
    kind: Issuer
    group: cert-manager.io

Copy the manifest above and apply it:

$ kubectl apply -f ca-certificate.yaml

Note: The Secret resource will contain the certificate and signing key, this will be created when the CA certificate is deployed, and the CA issuer references that secret. So that it will trust resulting signed certificates.

Next step is to deploy the CA issuer.

apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: ca-issuer
spec:
  ca:
    secretName: ca-cert
$ kubectl apply -f ca-issuer.yaml
issuer.cert-manager.io "ca-issuer" created

You can then check that the issuers have been successfully configured by checking the status, example:

$ kubectl get issuers
NAME          READY   STATUS                AGE
ca-issuer     True    Signing CA verified   60s

MySQL TLS and client certificates authentication

Once the Issuers are deployed, you are ready to request your certificates. It’s important to set the correct usages, otherwise the certificate will be created incorrectly (we had this issue, which is why we’re writing this post so you can avoid the same mistake).

Most of the certificates require by default the usages digital signature and key encipherment. We will add server auth to the server certificate and client auth to the client to ensure that the certificates can be used for client/server authentication.

The signed certificates will be stored in a secret resource in the same namespace as the certificates.

MySQL Server Certificate

Let’s start by requesting the MySQL server certificate and private key, copying the following manifest and applying to your cluster with the command kubectl apply -f mysql-server.yaml

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: mysql-server
spec:
  secretName: mysql-server-cert
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  isCA: false
  keySize: 2048
  keyAlgorithm: rsa
  keyEncoding: pkcs1
  usages:
    - digital signature
    - key encipherment
    - server auth
  commonName: MySQL server
  issuerRef:
    name: ca-issuer
    kind: Issuer
    group: cert-manager.io

I guess you are wondering how MySQL will be able to use the certificate. Let’s suppose your MySQL server is running as a pod in your Kubernetes environment. Create a ConfigMap with your MySQL Server configuration file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-config
data:
  mysql.cnf: |-
    [mysqld]
    ssl-ca=ca.crt
    ssl-cert=tls.crt
    ssl-key=tls.key
    require_secure_transport=ON   ## This line is the only setting required to enforce secure connections

Add this ConfigMap data and the Secret which contains the cert and key of the certificate created previously to a volume inside your MySQL pod or deployment. Now TLS is enabled on your MySQL server.

Access your MySQL server and check the values of the TLS related variables have been populated with the names of the certificates that we generated:

SHOW VARIABLES LIKE '%ssl%';
# Output
+---------------+-----------------+
| Variable_name | Value           |
+---------------+-----------------+
| have_openssl  | YES             |
| have_ssl      | YES             |
| ssl_ca        | ca.crt          |
| ssl_capath    |                 |
| ssl_cert      | tls.crt         |
| ssl_cipher    |                 |
| ssl_crl       |                 |
| ssl_crlpath   |                 |
| ssl_key       | tls.key         |
+---------------+-----------------+
9 rows in set (0.00 sec)

MySQL Client Certificate

Currently, we have MySQL server’s certificate signed by certificate authority (CA) and a key pair. Having these is enough to provide encryption for incoming connections.

However this is not enough for us, as we also need to authenticate the clients connecting to our MySQL server. To do so, we will request a client certificate and key, so that both parties can provide proof that their certificates were signed by a mutually trusted certificate authority.

MySQL TLS Connection Using JDBC

MySQL Connector/J is the Java library that manages the connection with the MySQL database. It can use TLS certificates to encrypt all that data between the JDBC driver and the MySQL server.

MySQL Connect/J has different ways of setting up an TLS connection. We are setting up two-way TLS authentications, in this form, both the client and the server has to establish trust between themselves using a trusted certificate.

Java application needs two Java keystore files to communicate over TLS. The truststore one file which contains CA certificate, and another called keystore which contains the keys and certificate for the client.

Java’s keytool is used to import CA certificate into Java truststore file, and import the client key and certificate into a Java keystore.

The latest version of cert-manager can do this for you, as you can see on the manifest, we are adding the keystore option on the Certificate spec. The keystore will be added in the Secret resource.

First, create a secret containing the keystore password, which will be used by cert-manager to generate the truststore.

kubectl create secret generic jks-password-secret --from-literal=password-key=[your-password]

Let’s create that client certificate and private key, same as you did with the server certificate, but using this manifest:

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: client-mysql
spec:
  secretName: mysql-client-cert
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  keySize: 2048
  keyAlgorithm: rsa
  keyEncoding: pkcs1
  usages:
    - digital signature
    - key encipherment
    - client auth
  commonName: MySQL client
  keystores:
    jks:
      create: true
      passwordSecretRef:
        key: password-key
        name: jks-password-secret  
  issuerRef:
    name: ca-issuer
    kind: Issuer
    group: cert-manager.io

Mount the secret as volume on your app deployment, so that the keystore and truststore will be available for the application. You can get more details about secrets and how to create them from the Kubernetes documentation.

Edit your app deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  ...
spec:
  replicas: 1
  selector:
    ...
  template:
    ...
    spec:
      containers:
        - name: app
          image: your-app-image
          volumeMounts:
            - mountPath: "/certs"
              name: truststore
              readOnly: true
            - mountPath: "certs2"
              name: keystore
              readOnly: true
      ...
      volumes:
        - name: truststore
          secret:
            secretName: mysql-client-cert
            items:
            - key: truststore.jks
              path: truststore
        - name: keystore
          secret:
            secretName: mysql-client-cert
            items:
            - key: keystore.jks
              path: keystore

Finally, update your application’s connection string, adding the useSSL, trustCertificateKeyStoreUrl, trustCertificateKeyStorePassword, clientCertificateKeyStoreUrl and clientCertificateKeyStorePassword parameters.

The connection string should look something like this:

jdbc:mysql://[url]:[port]/[dbname]?useSSL=true&clientCertificateKeyStoreUrl=file:/certs/keystore&clientCertificateKeyStorePassword=[your-password]&trustCertificateKeyStoreUrl=file:/certs/truststore&trustCertificateKeyStorePassword=[your-password]

The trustCertificateKeyStorePassword and clientCertificateKeyStorePassword keys set the password value that you set when you created the secret (we called it as jks-password-secret in this post) which is used by cert-manager to generate the java keystores.

Hopefully this post will be useful for your Java Spring Boot implementation, but most importantly for the security of your database.