Add additional trusted CA certificates to a mutable Kubernetes Harness delegate

Harness Experts
1

This article is similar to Adding 3rd party CA certificate(s) to Kubernetes and/or OpenShift Harness delegates but focuses on adding a configmap to the delegate namespace and using that configmap in the delegate’s environment variables, including INIT_SCRIPT.

Create CA certificates configmap

Although there are many paths to add a list of CAs to Harness’s trusted CAs, the easiest and most stable so far uses a configmap containing all individual CA certificates to trust in PEM format.

If you have this configmap already, and can 100% GUARANTEE the following, you can skip this step and go to Trust CA certificates within the Harness platform.

  • Contains ONLY keys ending in .crt
  • Every key’s data represents an individual CA certificate
  • Every CA certificate is in PEM format
  • Have YAML available to create in a new namespace

If you are unsure of any of the items above, follow the below steps to guarantee a properly formatted configmap Harness can use for adding trusted CAs and CA chains.

Requirements for CA configmap creation

  • CLI tools
    • openssl
    • kubectl
  • CA certificates to trust
    • Individual certificates preferred
      • ONLY file formats of .cer, .crt, and .pem are covered by this guide
    • Bundle of CAs in PEM format also covered by this guide

Step 1: Move all CA certs into a single folder

Begin with an empty folder, and move all CA certificates you want Harness to trust into this folder.

Usually these files end in .cer or .crt (and sometimes .pem).

For example:

mkdir harness-ca-certs
cp cert-a.crt cert-b.cer cert-c.pem ./harness-ca-certs/
cd harness-ca-certs

If you have a CA bundle in PEM format

Sometimes you don’t have individual CA certificate files, and only have a bundle of CAs (usually in PEM format).

To split this bundle apart to use the CAs in this bundle alongside of other CAs, use the following steps:

  1. Confirm the bundle is in PEM format

    PEM files must contain -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- markers surrounding base-64 encoded certificate contents.
    This script makes a guess around if a file is PEM-encoded or not based on if the PEM “begin certificate” marker is found.
    You can also less ca-bundle.pem to inspect the contents and confirm the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- markers are present.

    #!/bin/bash
# NOTE: change ca_file_target to point to your CA bundle
ca_file_target="ca-bundle.pem"
# NOTE: change output_dir to point to the folder where all individual CAs will
# be placed for harness usage
output_dir="harness-ca-certs"

function guess_is_pem() {
  while read -r file_line ; do
    if [ -z "${file_line}" ]; then continue ; fi
    if [ "${file_line}" == "-----BEGIN CERTIFICATE-----" ]; then
      echo "Guess: is PEM format"
      return
    fi
  done < "${1}"

  echo "Guess: NOT PEM format"
}

guess_is_pem "${ca_file_target}"

  2. Split PEM bundle into individual certificates

    #!/bin/bash
ca_file_target="ca-bundle.pem"

function split_pem_ca_bundle() {
  idx=0
  while read -r file_line ; do
    if [ -z "${file_line}" ]; then continue ; fi
    
    echo "${file_line}" > "ca-${idx}.crt"
    if [ "${file_line}" == "-----END CERTIFICATE-----" ]; then
      idx=$((idx + 1))
    fi
  done < "${1}"
}

split_pem_ca_bundle "${ca_file_target}"

Step 2: Convert certificates to PEM format

It’s easiest to work with CA certificates in PEM format.
When in your certificate directory (e.g. harness-ca-certs), ensure all provided certificates are in PEM format by running the following.

#!/bin/bash

# ASSUMING starting in folder harness-ca-certs

out_dir="./crt-out/"
mkdir "${out_dir}"

cert_files="$(find . -maxdepth 1 -type f \( -name "*.crt" -o -name "*.der" -o -name "*.pem" \))"

while read -r cert_file ; do
  cert_file_basename="$(basename "${cert_file}")"
  cert_extension="${cert_file_basename##*.}"
  cert_id="$(basename "${cert_file_basename}" ".${cert_extension}")"
  echo "${cert_id}"
  
  out_name="${cert_id}.crt"
  idx=0
  until [ ! -f "${out_dir}${out_name}" ]; do
    out_name="${cert_id}_${idx}.crt"
    idx=$((idx + 1))
  done
  
  openssl x509 -outform pem -in "${cert_file}" \
    -out "${out_dir}${out_name}"
done <<< "${cert_files}"

Step 3: Create CA cert configmap

Using the converted certificates in the output folder (crt-out above), create a configmap to store all individual CA certificates in.
Note that because these certificates are public certificates, there is no need to treat these as private - no need to create a secret, for example.

#!/bin/bash

# ASSUMING starting in folder harness-ca-certs

# inspect the current working directory on function call and create a configmap
# containing all .crt files found in the folder, targeting the namespace passed
# in as the first positional argument
#
# e.g. ca_certs_configmap_yaml "harness-delegate-ng"
function ca_certs_configmap_yaml() {
  local target_ns="$1"
  
  all_certs_args=()
  for cert in *.crt ; do
    if [ "${cert}" == "*.crt" ]; then echo "no certs found; exiting" ; fi
    
    all_certs_args+=("--from-file=${cert}")
  done

  kubectl create configmap -n "${target_ns}" ca-certs \
    "${all_certs_args[@]}" --dry-run=client -o yaml \
    > "configmap_${target_ns}_ca-certs.yaml"
}

# ---
# change to directory including all PEM-converted individual CA certificates
out_dir="./crt-out/"
pushd "${out_dir}" || {
  echo "could not change to out dir ${out_dir}; exiting"
  exit 1
}

# ---
# create configmap YAML and apply it
delegate_ns="harness-delegate-ng"
ca_certs_configmap_yaml "${delegate_ns}"
kubectl apply -f "configmap_${delegate_ns}_ca-certs.yaml"

popd || {
  echo "could not change to base dir; exiting"
  exit 1
}

The rest of this guide will assume the configmap name as above.

Trust CA certificates within the Harness platform

Modify the delegate YAML to mount the configmap to the pod and configure an init script (using the INIT_SCRIPT environment variable) to add the mounted certificates to the Ubunutu and java trust stores.

NOTE: the following change could be needed in the below partial YAML.

  • jre_path (currently jdk-11.0.14+9-jre in the init script) should be changed to the jdk / jre version used in your delegate

PARTIAL yaml representing changes made to the delegate statefulset:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    harness.io/name: test-delegate
  name: test-delegate
  namespace: harness-delegate-ng
spec:
  # ...
  template:
    # ...
    spec:
      containers:
      - image: harness/delegate:latest
        # ...
        env:
        # ...
        - name: INIT_SCRIPT
          value: |
            jre_path="/opt/harness-delegate/jdk-11.0.14+9-jre"
            for f in /tmp/ca-certs/*.crt ; do
              no_prefix="${f#/tmp/ca-certs/}"
              id="${no_prefix%.crt}"
              echo "adding cert $id to trust store"
            
              # create bundle of all CAs
              cat "${f}" >> /tmp/ca-certs/cacerts.pem
              echo "" >> /tmp/ca-certs/cacerts.pem
            
              # copy target cert to Ubuntu CA certs location
              cp "${f}" /usr/local/share/ca-certificates/
            
              # add target cert to Java trust store
              "${jre_path}/bin/keytool" -import -trustcacerts -keystore "${jre_path}/lib/security/cacerts" -storepass changeit -alias "${id}" -file "${f}" -noprompt
            done
            update-ca-certificates
        - name: ADDITIONAL_CERTS_PATH
          value: "/tmp/ca-certs/cacerts.pem"
        - name: CI_MOUNT_VOLUMES
          value: "/etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-bundle.crt,/tmp/ca-certs/cacerts.pem:/kaniko/ssl/certs/additional-ca-cert-bundle.crt"
          # ...
        volumeMounts:
          - name: ca-certs
            mountPath: /tmp/ca-certs
      # ...
      volumes:
        - name: ca-certs
          configMap:
            name: ca-certs
3 Likes
2

What a great article. Welcome to the community!