Adding 3rd party CA certificate(s) to Kubernetes and/or OpenShift Harness delegates

Harness Experts
1

Howdy folks!

I have noticed a lot of organizations are increasingly using their own internal certificate authorities to issue certificates across their organization internally. Of course, the majority, if not all of these CA certificates, will not be included in the list of trusted root certificate authorities distributed with most software that utilizes SSL/TLS for encryption.

For example, the harness delegate uses the trust store(cacerts) that is distributed by default with the Java runtime environment, and if your organization has configured the harness manager to serve up an SSL certificate signed by your internal CA (or you have an SSL proxy using the same), you may run into the following issue when it tries to initially connect:

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

With a shell script delegate, as the delegate installation is more persistent, this is relatively easy to work around by manually importing the certificate into the truststore. However, the way the delegate is designed to run on Kubernetes/OpenShift, we run into two problems:

  1. We cannot use the delegate profile as this operation needs to be performed before the delegate attempts to connect to the manager, and the profile is not executed until after this connection is established.
  2. If we add this to the delegate keystore manually, after restarting the delegate the changes are lost and the delegate is still unable to connect.

Solution:

There is of course a myriad of ways to accomplish this(Creating a secret with the certificate and mounting this, or using it as an environment variable, init containers, and more). However the focus here is going to go with the simplest method that requires the least amount of changes to the delegate yaml, and is easy to manage in the event you need to update the certificate across multiple delegates quickly. This method involves having a location to host the certificate(s) that you would like to inject into the cacerts trust store, and a simple modification to the harness-delegate.yaml:

  1. Place your PEM certificate at a location that can be accessed via http/https by the delegate at startup time, in my example it’s:

http://192.168.10.253/cert.pem

  1. In the container spec of the delegate StatefulSet, I added the following lifecycle postStart command:

    lifecycle:
  postStart:
    exec:
      command:
        - /bin/sh
        - -c
        - >
          curl http://192.168.10.253/cert.pem -o /tmp/cert.pem ;
          jdk8u242-b08-jre/bin/keytool -importcert -file /tmp/cert.pem -keystore jdk8u242-b08-jre/lib/security/cacerts -alias customcert1 -storepass changeit

For context example:

image
image1448×346 7.8 KB

This will effectively fetch the PEM certificate, and insert it into the delegate JRE cacerts truststore each time the delegate starts, and hopefully make management of internally signed certificates for delegates a cinch!

3 Likes