Run STO SonarQube scan trusting private CAs

Harness Experts
1

Run STO SonarQube scan trusting private certificates

Running STO SonarQube plugin against Sonar server using private CA certs (Kubernetes infrastructure)

SonarQube servers often use private CA chains to provide SSL / TLS trust. Harness’s SonarQube STO integration can be configured to explicitly trust this CA chain in order to maintain secure network connectivity.

This guide covers the full set of configurations needed to enable this capability. This includes 3 main layers:

  1. Add additional CA certificates to delegate
  2. Configure delegate to mount certificate bundles to all CI pods
  3. In a CI pipeline, move the additional certificates into the correct path for the SonarQube STO plugin to trust them.

IMPORTANT: This guide assumes using a Helm delegate and using Kubernetes CI infrastructure.

Core principals may apply to other delegate installation methods or CI infrastructure options, but scripts will not work for these additional configurations.

Add additional CA certificates to delegate + mount CAs to all CI pods

Prerequisites

The machine running the commands / scripts must have the following installed:

  • helm (required for helm-based delegate)
  • kubectl (tested with)

Ensure ca-certs configmap exists in proper format

A ca-certs configmap is required to be present in the Harness namespace. This configmap expects:

  • individual CA certificates in PEM format
  • only .crt file names

CA bundles are NOT supported in this ca-certs configmap. See https://discuss.harness.io/t/add-additional-trusted-ca-certificates-to-a-mutable-kubernetes-harness-delegate/12757#create-ca-certificates-configmap-1 for more details.

Example kubectl command for creating ca-certs configmap

An example: given 3 CA certificates in the current directory …

ubuntu@ip-172-31-34-8:~$ ls *.crt
harness-ca-inter-a.crt  harness-ca-inter-b.crt  harness-ca-root.crt

The following kubectl command will create a properly formatted ca-certs configmap YAML:

kubectl create configmap ca-certs --dry-run=client -o yaml \
  --from-file=harness-ca-root.crt \
  --from-file=harness-ca-inter-a.crt \
  --from-file=harness-ca-inter-b.crt \
  > configmap_ca-certs.yaml

This YAML can be applied in the correct namespace of the target harness delegate, e.g. …

kubectl apply -n harness-delegate-ng -f configmap_ca-certs.yaml

Configure delegate to trust additional certificates + mount CAs to CI pods

When installing a Helm-based delegate, a post-render script can be configured to modify the installed YAML of the delegate to trust the provided CA certificates.

LIMITATIONS: The below script will OVERRIDE any INIT_SCRIPT already present in the delegate Helm config.

If additional INIT_SCRIPT logic is required, insert into patch-delegate-configmap.yaml instead of configuring via Helm values.

The following components are involved:

  • patch-delegate-deployment.yaml
    • mounts the created ca-certs configmap into the delegate as files for runtime use
  • patch-delegate-configmap.yaml
    • INIT_SCRIPT
      • reads each .crt file from the mounted ca-certs configmap, and…
        • adds it to an additional CAs bundle
        • imports the CA into the delegate’s CA cert trust store so the delegate will trust the additional CAs
      • updates the delegate’s OS trust store using the addtional CA bundle for native CLI binaries to trust the additional CAs as well
    • ADDITIONAL_CERTS_PATH points the delegate to the additional certs bundle for other Harness components to use for additional CA trust
    • CI_MOUNT_VOLUMES
      • /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:/etc/ssl/certs/ca-bundle.crt
        • mounts the full set of OS trusted CA certificates (including additional CAs) to where Harness CI’s git clone step expects the full bundle of CA certs to exist
      • /tmp/harness/ca-certs/cacerts.pem:/kaniko/ssl/certs/additional-ca-cert-bundle.crt
        • mounts the bundle of only additionally trusted CA certificates to where Harness’s kaniko-based docker build step expects the additional CA bundle to exist

Create post-render script

Run the below script to create the post-render script:

(Thanks Ryan Nelson for the switch away from kustomize and to kubectl kustomize instead!)

# !!! CHANGE DELEGATE NAME to reflect installed delegate name
delegate_name="harness-delegate"

cat <<EOF > post-render.sh
#!/bin/bash

cat <&0 > all.yaml

kubectl kustomize && rm all.yaml
EOF

cat <<EOF > kustomization.yaml
resources:
  - all.yaml
patches:
  - path: patch-delegate-configmap.yaml
    target:
      kind: ConfigMap
      name: "${delegate_name}"
  - path: patch-delegate-deployment.yaml
    target:
      kind: Deployment
      name: "${delegate_name}"
EOF

cat <<EOF > patch-delegate-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: _
data:
  INIT_SCRIPT: |
    jre_path="/opt/java/openjdk"
    mkdir -p /tmp/harness/ca-certs/
    
    for f in /tmp/ca-certs/*.crt ; do
      no_prefix="\${f#/tmp/ca-certs/}"
      id="\${no_prefix%.crt}"
      echo "adding cert \${id} to trust store"
    
      # create bundle of all CAs
      echo "\${id}" >> /tmp/harness/ca-certs/cacerts.pem
      cat "\${f}" >> /tmp/harness/ca-certs/cacerts.pem
      echo "" >> /tmp/harness/ca-certs/cacerts.pem
    
      # copy target cert to UBI CA certs location
      cp "\${f}" /etc/pki/ca-trust/source/anchors
    
      # add target cert to Java trust store
      "\${jre_path}/bin/keytool" -import -trustcacerts -keystore "\${jre_path}/lib/security/cacerts" -storepass changeit -alias "\${id}" -file "\${f}" -noprompt
    done
    update-ca-trust
  ADDITIONAL_CERTS_PATH: "/tmp/harness/ca-certs/cacerts.pem"
  CI_MOUNT_VOLUMES: "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:/etc/ssl/certs/ca-bundle.crt,/tmp/harness/ca-certs/cacerts.pem:/kaniko/ssl/certs/additional-ca-cert-bundle.crt"
EOF

cat <<EOF > patch-delegate-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: _
spec:
  template:
    spec:
      containers:
        - name: delegate
          volumeMounts:
            - name: ca-certs
              mountPath: /tmp/ca-certs
      volumes:
        - name: ca-certs
          configMap:
            name: ca-certs
EOF

chmod +x post-render.sh

Use post-render script in Helm install

The post-render script can be used in a helm install as seen below:

delegate_name="del-sonar-cas"
helm upgrade -i "${delegate_name}" \
  --namespace harness-delegate-ng --create-namespace \
  harness-delegate/harness-delegate-ng \
  -f values-harness-delegate.yaml \
  --post-renderer ./post-render.sh

In CI stage: move CA certs to location expected by SonarQube STO plugin

Use the following template to split the additional CA bundle back into individual certs and add each individual cert to the location expected by the STO plugin.

IMPORTANT: The CI stage Overview > Shared Paths must include an entry for /shared/customer_artifacts to ensure processed CA certs will appear in future STO steps.

template:
  name: Process STO CA certs
  type: Step
  spec:
    type: Run
    spec:
      connectorRef: account.harnessImage
      image: ubuntu:22.04
      shell: Bash
      command: |
        mkdir -p /shared/customer_artifacts/certificates/
        
        idx=0
        while read -r file_line ; do
          if [ -z "${file_line}" ]; then continue ; fi
          
          echo "${file_line}" > "/shared/customer_artifacts/certificates/ca-${idx}.crt"
          if [ "${file_line}" == "-----END CERTIFICATE-----" ]; then
            idx="$((idx + 1))"
          fi
        done < "/kaniko/ssl/certs/additional-ca-cert-bundle.crt"
    description: |-
      Move CA certs into correct location for STO plugins

      Original author: nikkelma
  identifier: Process_Sonar_CA_certs
  versionLabel: v1.0
  tags: {}

Full recreation

For completeness, a full recreation of this setup can be achieved via the below steps.

Setup

A machine to run SonarQube and the Harness delegate is required, as well as a publicly exposed IP.

Install k3s + helm + kustomize 

curl -sfL https://get.k3s.io -o install.sh
chmod +x install.sh
INSTALL_K3S_CHANNEL="v1.25" ./install.sh

sudo cp /etc/rancher/k3s/k3s.yaml "${HOME}/kube_config.yaml"
sudo chown "${USER}" "${HOME}/kube_config.yaml"
KUBECONFIG="${HOME}/kube_config.yaml"
export KUBECONFIG

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

curl -LO https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.0.1/kustomize_v5.0.1_linux_amd64.tar.gz
tar -xf kustomize_v5.0.1_linux_amd64.tar.gz
sudo install -o root -g root -m 0755 kustomize /usr/local/bin/kustomize

Install cert-manager

helm repo add jetstack https://charts.jetstack.io
helm repo update

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.crds.yaml

helm upgrade --install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.11.0

Configure PKI

# !!! EDIT target_ip to reflect IP exposed by machine.
target_ip="12.34.56.78"

cat <<EOF > cluster-issuer_ss-root.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ss-root
spec:
  selfSigned: {}
EOF

cat <<EOF > certificate_ca-root.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ca-root
  namespace: cert-manager
spec:
  isCA: true
  subject:
    organizations:
      - harness
  commonName: harness-root
  secretName: root-tls
  privateKey:
    algorithm: RSA
    size: 2048
  issuerRef:
    name: ss-root
    kind: ClusterIssuer
    group: cert-manager.io
EOF

cat <<EOF > cluster-issuer_ca-root.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ca-root
spec:
  ca:
    secretName: root-tls
EOF

cat <<EOF > certificate_ca-inter-a.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ca-inter-a
  namespace: cert-manager
spec:
  isCA: true
  subject:
    organizations:
      - harness
  commonName: harness-inter-a
  secretName: inter-a-tls
  privateKey:
    algorithm: RSA
    size: 2048
  issuerRef:
    name: ca-root
    kind: ClusterIssuer
    group: cert-manager.io
EOF

cat <<EOF > certificate_ca-inter-b.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ca-inter-b
  namespace: cert-manager
spec:
  isCA: true
  subject:
    organizations:
      - harness
  commonName: harness-inter-b
  secretName: inter-b-tls
  privateKey:
    algorithm: RSA
    size: 2048
  issuerRef:
    name: ca-root
    kind: ClusterIssuer
    group: cert-manager.io
EOF

cat <<EOF > cluster-issuer_ca-inter-a.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ca-inter-a
spec:
  ca:
    secretName: inter-a-tls
EOF

cat <<EOF > cluster-issuer_ca-inter-b.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ca-inter-b
spec:
  ca:
    secretName: inter-b-tls
EOF

# apply PKI manifests into cert-manager
kubectl apply -f cluster-issuer_ss-root.yaml
kubectl apply -f certificate_ca-root.yaml
kubectl apply -f cluster-issuer_ca-root.yaml
kubectl apply -f certificate_ca-inter-a.yaml
kubectl apply -f certificate_ca-inter-b.yaml
kubectl apply -f cluster-issuer_ca-inter-a.yaml
kubectl apply -f cluster-issuer_ca-inter-b.yaml
kubectl apply -f certificate_wildcard.yaml

# after certs are generated, fetch contents
kubectl get secret -n cert-manager root-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > harness-ca-root.crt
kubectl get secret -n cert-manager inter-a-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > harness-ca-inter-a.crt
kubectl get secret -n cert-manager inter-b-tls -o jsonpath='{.data.tls\.crt}' | base64 -d > harness-ca-inter-b.crt

kubectl create configmap ca-certs --dry-run=client -o yaml \
  --from-file=harness-ca-root.crt \
  --from-file=harness-ca-inter-a.crt \
  --from-file=harness-ca-inter-b.crt \
  > configmap_ca-certs.yaml

Install SonarQube

helm repo add sonarqube https://sonarsource.github.io/helm-chart-sonarqube
helm repo update

kubectl create ns sonarqube

kubectl create secret generic ca-certs -n sonarqube \
  --from-file=harness-ca-root.crt \
  --from-file=harness-ca-inter-a.crt \
  --from-file=harness-ca-inter-b.crt

cat > values-sonarqube.yaml <<EOF
caCerts:
  enabled: true
  secret: ca-certs
EOF

helm upgrade --install \
  sonarqube sonarqube/sonarqube \
  -n sonarqube --create-namespace \
  --version "10.0.0+521" \
  -f values-sonarqube.yaml

# !!! target_ip USED BELOW for DNS name

cat <<EOF > certificate_sonarqube.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: chain-b-sonarqube
  namespace: sonarqube
spec:
  secretName: chain-b-sonarqube-tls
  duration: 2160h # 90d
  renewBefore: 360h # 15d
  subject:
    organizations:
      - harness
  isCA: false
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
  usages:
    - server auth
    - client auth
  dnsNames:
    - "sonarqube.${target_ip}.sslip.io"
  issuerRef:
    name: ca-inter-b
    kind: ClusterIssuer
    group: cert-manager.io
EOF

# !!! target_ip USED BELOW for DNS name

cat <<EOF > ingress_sonarqube.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sonarqube
  namespace: sonarqube
spec:
  ingressClassName: traefik
  rules:
    - host: sonarqube.${target_ip}.sslip.io
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: sonarqube-sonarqube
                port:
                  number: 9000
  tls:
    - hosts:
        - sonarqube.${target_ip}.sslip.io
      secretName: chain-b-sonarqube-tls
EOF

kubectl apply -f certificate_sonarqube.yaml
kubectl apply -f ingress_sonarqube.yaml

Configure sonar project

  • Log in to SonarQube server for first time

    • default username / password is admin / admin

  • Create manual project

    • display name: OWASP WebGoat
    • key: owasp-webgoat

  • create token in project

    • example result: sqp_example4082490396561e4ea03949b86b006e34f

Install harness delegate

helm repo add harness-delegate https://app.harness.io/storage/harness-download/delegate-helm-chart/
helm repo update harness-delegate

# !!! fill in variables according to Harness Helm delegate UI settings
delegate_name=""
account_id=""
delegate_token=""
manager_endpoint=""
delegate_image=""

cat <<EOF > values-harness-delegate.yaml
delegateName: ${delegate_name}
accountId: ${account_id}
delegateToken: ${account_id}
managerEndpoint: ${manager_endpoint}
delegateDockerImage: ${delegate_image}
replicas: 1
upgrader:
  enabled: false
EOF

cat <<EOF > post-render.sh
#!/bin/bash

cat <&0 > all.yaml

kubectl kustomize && rm all.yaml
EOF

cat <<EOF > kustomization.yaml
resources:
  - all.yaml
patches:
  - path: patch-delegate-configmap.yaml
    target:
      kind: ConfigMap
      name: "${delegate_name}"
  - path: patch-delegate-deployment.yaml
    target:
      kind: Deployment
      name: "${delegate_name}"
EOF

cat <<EOF > patch-delegate-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: _
data:
  INIT_SCRIPT: |
    jre_path="/opt/java/openjdk"
    mkdir -p /tmp/harness/ca-certs/
    
    for f in /tmp/ca-certs/*.crt ; do
      no_prefix="\${f#/tmp/ca-certs/}"
      id="\${no_prefix%.crt}"
      echo "adding cert \${id} to trust store"
    
      # create bundle of all CAs
      echo "\${id}" >> /tmp/harness/ca-certs/cacerts.pem
      cat "\${f}" >> /tmp/harness/ca-certs/cacerts.pem
      echo "" >> /tmp/harness/ca-certs/cacerts.pem
    
      # copy target cert to UBI CA certs location
      cp "\${f}" /etc/pki/ca-trust/source/anchors
    
      # add target cert to Java trust store
      "\${jre_path}/bin/keytool" -import -trustcacerts -keystore "\${jre_path}/lib/security/cacerts" -storepass changeit -alias "\${id}" -file "\${f}" -noprompt
    done
    update-ca-trust
  ADDITIONAL_CERTS_PATH: "/tmp/harness/ca-certs/cacerts.pem"
  CI_MOUNT_VOLUMES: "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:/etc/ssl/certs/ca-bundle.crt,/tmp/harness/ca-certs/cacerts.pem:/kaniko/ssl/certs/additional-ca-cert-bundle.crt"
EOF

cat <<EOF > patch-delegate-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: _
spec:
  template:
    spec:
      containers:
        - name: delegate
          volumeMounts:
            - name: ca-certs
              mountPath: /tmp/ca-certs
      volumes:
        - name: ca-certs
          configMap:
            name: ca-certs
EOF

chmod +x post-render.sh

kubectl create ns harness-delegate-ng
kubectl apply -n harness-delegate-ng -f configmap_ca-certs.yaml

helm upgrade -i "${delegate_name}" \
  --namespace harness-delegate-ng --create-namespace \
  harness-delegate/harness-delegate-ng \
  -f values-harness-delegate.yaml \
  --post-renderer ./post-render.sh

Create Harness configs and pipeline

  • create k8s connector using delegate credentials
  • create global github connector using url https://github.com
  • create harness-ci namespace in k3s cluster (kubectl create ns harness-ci)
  • create secret sonar_webgoat_token at account level containing sonarqube token
  • create Process STO CA certs template at account level
template:
  name: Process STO CA certs
  type: Step
  spec:
    type: Run
    spec:
      connectorRef: account.harnessImage
      image: ubuntu:22.04
      shell: Bash
      command: |
        mkdir -p /shared/customer_artifacts/certificates/

        idx=0
        while read -r file_line ; do
        if [ -z "${file_line}" ]; then continue ; fi

        echo "${file_line}" > "/shared/customer_artifacts/certificates/ca-${idx}.crt"
        if [ "${file_line}" == "-----END CERTIFICATE-----" ]; then
            idx=$((idx + 1))
        fi
        done < "/kaniko/ssl/certs/additional-ca-cert-bundle.crt"
    description: |-
      Move CA certs into correct location for STO plugins

      Original author: Matt Nikkel
  identifier: Process_STO_CA_certs
  versionLabel: v1.0
  tags: {}
  • create pipeline to execute sonar scan
pipeline:
  name: Sonar Scan Custom CAs
  identifier: Sonar_Scan_Custom_CAs
  projectIdentifier: Main
  orgIdentifier: default
  tags: {}
  properties:
    ci:
      codebase:
        connectorRef: <+input>
        repoName: WebGoat/WebGoat
        build: <+input>
  stages:
    - stage:
        name: clone and scan
        identifier: clone_and_scan
        type: CI
        spec:
          cloneCodebase: true
          infrastructure:
            type: KubernetesDirect
            spec:
              connectorRef: <+input>
              namespace: harness-ci
              automountServiceAccountToken: true
              nodeSelector: {}
              os: Linux
          execution:
            steps:
              - step:
                  type: Run
                  name: Build
                  identifier: Build
                  spec:
                    connectorRef: account.harnessImage
                    image: maven:3.9.1-eclipse-temurin-17-alpine
                    shell: Bash
                    command: mvn clean install -Dmaven.test.skip
              - step:
                  name: Move CA certs
                  identifier: Move_CA_certs
                  template:
                    templateRef: account.Process_STO_CA_certs
                    versionLabel: v1.0
              - step:
                  type: Sonarqube
                  name: SonarQube
                  identifier: SonarQube
                  spec:
                    mode: orchestration
                    config: default
                    target:
                      name: github.com/WebGoat/WebGoat
                      type: repository
                      variant: main
                    advanced:
                      log:
                        level: info
                    resources:
                      limits:
                        memory: 2Gi
                        cpu: "1"
                    auth:
                      access_token: <+secrets.getValue("account.sonar_webgoat_token")>
                      domain: <+input>
                      ssl: true
                    tool:
                      java:
                        binaries: /harness/target
                      project_key: owasp-webgoat
          sharedPaths:
            - /shared/customer_artifacts