Overriding at the Infra Definition Level

rohan · May 3, 2020, 6:35pm

Developers want the capability of overriding specific sets of variables at the infrastructure definition level within Harness. A common use case we have seen is that developers have one common chart per environment however within their dev environment, there could be numerous namespaces or infrastructure definitions that have different variables that need to be passed to the Values.yaml that dictates the chart.

This technique is not limited to Kubernetes, developers can leverage a similar technique for Traditional SSH deployments, ECS Deployments, etc.

Before You Begin

Your target Environment must have multiple infrastructure definitions
You must have a service that needs to have overrides at the infrastructure definition level
The application must contain a service, workflow, and environment

Review: Variable Overrides

Please review the Kubernetes variables overrides documentation in Harness: Override Harness Kubernetes Service Settings - Harness.io Docs

This doc will provide further context into how Harness does variable overrides and what is the hierarchy for these overrides.

Overview:

Sometimes our services have specific variables overrides at the Infrastructure level. This means, in a specific namespace, the service could have different configurations or secrets that it needs to use. Given a specific infrastructure mapping, the specs of the service could be very different in comparison to other infrastructure. This allows users to tie configuration to specific pieces of infrastructure which is more granular than the environment override capabilities we have in Harness today.

Step 1: Configure the Service

In this example, we will be configuring a Harness Kubernetes Service that deploys to an environment with multiple infrastructure definitions. A common use case we see is that customers want to override at the Infrastructure definition level because configurations can differ per environment, per infrastructure definition. In case of a conflict in variables, a customer would prefer to have the Infrastructure Definition value override the value defined in the Environment. The base chart will most likely remain the same in this case. Fields that may differ in an environment:

suffix for ingress domains
differs per cluster or region (for example, DNS server IPs, S3 bucket names)

Below is are the sample service Manifests

Deployment.yaml

apiVersion: v1

kind: Namespace

metadata:

name: {{.Values.namespace}}

apiVersion: v1

kind: ConfigMap

metadata:

name: {{.Values.name}}-{{.Values.track}}

labels:

app: {{.Values.name}}

track: {{.Values.track}}

data:

    APP_ENV1: {{.Values.appEnv1}}

    APP_ENV2: {{.Values.appEnv2}}

apiVersion: apps/v1beta1

kind: Deployment

metadata:

name: {{.Values.name}}-{{.Values.track}}

labels:

app: {{.Values.name}}

track: {{.Values.track}}

version: {{.Values.version}}

spec:

replicas: {{.Values.replicas}}

selector:

matchLabels:

app: {{.Values.name}}

track: {{.Values.track}}

template:

metadata:

labels:

app: {{.Values.name}}

track: {{.Values.track}}

version: {{.Values.version}}

spec:

containers:

- name: {{.Values.name}}

image: {{.Values.image}}

imagePullPolicy: Always

resources:

requests:

cpu: 100m

memory: 50Mi

ports:

- name: http

containerPort: 8080

envFrom:

- configMapRef:

name: {{.Values.name}}-{{.Values.track}}

Service Yaml

    apiVersion: v1

    kind: Service

    metadata:

    name: {{.Values.name}}

    labels:

    app: {{.Values.name}}

    spec:

    type: ClusterIP

    ports:

    - name: http

    port: 9080

    protocol: TCP

    targetPort: http

    selector:

    app: {{.Values.name}}

In the service, please add configuration variables to the service. Call them s3bucketName and dnsServer.

Set the configuration variable “appEnv1” to “aaa”

Set the configuration variable “appEnv2” to “bbb”

In your values.yaml, reference those Harness variables like so:

appEnv1: ${serviceVariable.appEnv1}

appEnv2: ${serviceVariable.appEnv2}

Values Yaml

namespace: ${infra.kubernetes.namespace}

apiUrl: http://localhost:8080

replicas: 1

name: infra-override

image: ${artifact.metadata.image}

version: ${artifact.metadata.tag}

track: primary

endpoint: rpc

appEnv1: ${serviceVariable.appEnv1}

appEnv2: ${serviceVariable.appEnv2}

Step 2: Add the Environment Overrides

Overview:

We need to add Environment overrides so we can have them be overridden at the infrastructure definition level. Based on the name of the infrastructure definition or a specific value, we can override the variables we wish to at the Infrastructure Definition Level. Environment Variables are the first step to overriding, the second is enabling specific values to be passed when Infra Mapping is selected. The Environment Variables are the access point for the override variables to be assigned.

In the environment section, make sure you have multiple infra definition mapping. Once the mappings are configured, add Service Configuration Override Variables to the Environment.

You will need to provide an override variable per environment, the best way to distinguish these variables is having the infrastructure def name associated with the variable name so that you know which override is being applied per environment.

Another thing that needs to be configured is the values.yaml file override.

Step 3: Configure the Workflow

Overview:

The user will need to configure a shell script to handle the assignment of these variables in the workflow. This shell script will be crucial because it will assign the Infrastructure variables as the environment variables configured in the previous step.

In the workflow, write a script assigning variables based on the infra name.

Here is a sample shell script that does that:

echo
echo Using infrastructure definition [${infra.name}]
echo

appEnv1=${serviceVariable.appEnv1}
appEnv2=${serviceVariable.appEnv2}

if [[ "${infra.name}" == infra2 ]]; then

appEnv2=${serviceVariable.infra2_appEnv2}

elif [[ "${infra.name}" == infra3 ]]; then

appEnv1=${serviceVariable.infra3_appEnv1}
appEnv2=${serviceVariable.infra3_appEnv2}

fi

echo Setting appEnv1 to [$appEnv1]
echo Setting appEnv2 to [$appEnv2]
echo

Export the variables into the context, which will be leveraged in the override configured earlier. The ${override.appEnv1} is referencing a value based on this shell script.

By publishing the variable under the name “override”, we can reference it in the Values.yaml configuration override.

Add the shell script to the Deploy steps before the Rollout Deployment:

Deploy the workflow and, based on the infrastructure definition, we can see certain variables being overwritten.

For InfraDef1 the values were assigned based on the service configuration variables we provided in the Environment. InfraDef1 did not override the environment level values.

Run this deployment again in InfraDef2,

Now the environment level value is taken for appEnv1 but appEnv2 is overridden with the value specific to InfraDef2.

Now deploy the third infrastructure definition. This time both variables are overridden with values specific to InfraDef3.

Next Steps

Check out the community article on publishing variable outputs: Publish Variables