Hello everyone 
Introduction
We have no doubt that Harness is the best CD platform in the universe and makes our deployment life easier for Kubernetes environments. But we also need to be prepared for the pranks that the Kubernetes play on us.
Let’s see how Harness can behave when one of these Kubernetes tricks impacts our deployment. Here’s the error we see in our deployment logs, which we’ll cover in this article
ERROR 2021-09-17 17:04:03 Error from server (Invalid): error when applying patch:
ERROR 2021-09-17 17:04:03 {"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"app.kubernetes.io/managed-by\":\"Helm\",\"kubernetes.io/change-cause\":\"kubectl apply --kubeconfig=config --filename=manifests.yaml --record=true\"},\"labels\":{\"app-id\":\"secondary\"},\"name\":\"new-app-harness-example-deployment\",\"namespace\":\"pedro-staging\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"new-app-harness-example\"}},\"template\":{\"metadata\":{\"labels\":{\"app\":\"new-app-harness-example\",\"harness.io/release-name\":\"release-f3fe376d-0d43-3346-a22b-5be4ba45247c\"}},\"spec\":{\"containers\":[{\"env\":[{\"name\":\"SECRET_USERNAME\",\"valueFrom\":{\"secretKeyRef\":{\"key\":\"username\",\"name\":\"mysecret1-2\"}}},{\"name\":\"SECRET_PASSWORD\",\"valueFrom\":{\"secretKeyRef\":{\"key\":\"password\",\"name\":\"mysecret1-2\"}}},{\"name\":\"SECRET_PASSWORD_2\",\"value\":\"0.0.7\"}],\"image\":\"redis\",\"name\":\"new-app-harness-example\"}]}}}}\n"}},"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"new-app-harness-example"}],"containers":[{"$setElementOrder/env":[{"name":"SECRET_USERNAME"},{"name":"SECRET_PASSWORD"},{"name":"SECRET_PASSWORD_2"}],"env":[{"name":"SECRET_USERNAME","valueFrom":{"secretKeyRef":{"name":"mysecret1-2"}}},{"name":"SECRET_PASSWORD","valueFrom":{"secretKeyRef":{"key":"password","name":"mysecret1-2"}}}],"name":"new-app-harness-example"}]}}}}
ERROR 2021-09-17 17:04:03 to:
ERROR 2021-09-17 17:04:03 Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
ERROR 2021-09-17 17:04:03 Name: "new-app-harness-example-deployment", Namespace: "pedro-staging"
ERROR 2021-09-17 17:04:03 Object: &{map["kind":"Deployment" "apiVersion":"apps/v1" "metadata":map["managedFields":[map["operation":"Update" "apiVersion":"apps/v1" "time":"2021-09-17T19:49:18Z" "fieldsType":"FieldsV1" "fieldsV1":map["f:metadata":map["f:annotations":map["f:kubectl.kubernetes.io/last-applied-configuration":map[] "f:kubernetes.io/change-cause":map[] ".":map[] "f:app.kubernetes.io/managed-by":map[]] "f:labels":map[".":map[] "f:app-id":map[]]] "f:spec":map["f:revisionHistoryLimit":map[] "f:selector":map[] "f:strategy":map["f:rollingUpdate":map[".":map[] "f:maxSurge":map[] "f:maxUnavailable":map[]] "f:type":map[]] "f:template":map["f:metadata":map["f:labels":map[".":map[] "f:app":map[] "f:harness.io/release-name":map[]]] "f:spec":map["f:containers":map["k:{\"name\":\"new-app-harness-example\"}":map[".":map[] "f:env":map[".":map[] "k:{\"name\":\"SECRET_PASSWORD\"}":map[".":map[] "f:name":map[]] "k:{\"name\":\"SECRET_PASSWORD_2\"}":map[".":map[] "f:name":map[] "f:value":map[]] "k:{\"name\":\"SECRET_USERNAME\"}":map["f:name":map[] "f:valueFrom":map["f:secretKeyRef":map[".":map[] "f:key":map[] "f:name":map[]] ".":map[]] ".":map[]]] "f:image":map[] "f:imagePullPolicy":map[] "f:name":map[] "f:resources":map[] "f:terminationMessagePath":map[] "f:terminationMessagePolicy":map[]]] "f:dnsPolicy":map[] "f:restartPolicy":map[] "f:schedulerName":map[] "f:securityContext":map[] "f:terminationGracePeriodSeconds":map[]]] "f:progressDeadlineSeconds":map[] "f:replicas":map[]]] "manager":"kubectl"] map["fieldsV1":map["f:spec":map["f:template":map["f:spec":map["f:containers":map["k:{\"name\":\"new-app-harness-example\"}":map["f:env":map["k:{\"name\":\"SECRET_PASSWORD\"}":map["f:value":map[]]]]]]]]] "manager":"kubectl-edit" "operation":"Update" "apiVersion":"apps/v1" "time":"2021-09-17T20:03:46Z" "fieldsType":"FieldsV1"] map["fieldsV1":map["f:metadata":map["f:annotations":map["f:deployment.kubernetes.io/revision":map[]]] "f:status":map["f:conditions":map[".":map[] "k:{\"type\":\"Available\"}":map["f:lastUpdateTime":map[] "f:message":map[] "f:reason":map[] "f:status":map[] "f:type":map[] ".":map[] "f:lastTransitionTime":map[]] "k:{\"type\":\"Progressing\"}":map["f:lastUpdateTime":map[] "f:message":map[] "f:reason":map[] "f:status":map[] "f:type":map[] ".":map[] "f:lastTransitionTime":map[]]] "f:observedGeneration":map[] "f:readyReplicas":map[] "f:replicas":map[] "f:updatedReplicas":map[] "f:availableReplicas":map[]]] "manager":"kube-controller-manager" "operation":"Update" "apiVersion":"apps/v1" "time":"2021-09-17T20:03:47Z" "fieldsType":"FieldsV1"]] "namespace":"pedro-staging" "resourceVersion":"29039258" "generation":'\x02' "creationTimestamp":"2021-09-17T19:49:18Z" "labels":map["app-id":"secondary"] "annotations":map["kubernetes.io/change-cause":"kubectl apply --kubeconfig=config --filename=manifests.yaml --record=true" "app.kubernetes.io/managed-by":"Helm" "deployment.kubernetes.io/revision":"2" "kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"app.kubernetes.io/managed-by\":\"Helm\",\"kubernetes.io/change-cause\":\"kubectl apply --kubeconfig=config --filename=manifests.yaml --record=true\"},\"labels\":{\"app-id\":\"secondary\"},\"name\":\"new-app-harness-example-deployment\",\"namespace\":\"pedro-staging\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"new-app-harness-example\"}},\"template\":{\"metadata\":{\"labels\":{\"app\":\"new-app-harness-example\",\"harness.io/release-name\":\"release-f3fe376d-0d43-3346-a22b-5be4ba45247c\"}},\"spec\":{\"containers\":[{\"env\":[{\"name\":\"SECRET_USERNAME\",\"valueFrom\":{\"secretKeyRef\":{\"key\":\"username\",\"name\":\"mysecret1-1\"}}},{\"name\":\"SECRET_PASSWORD\",\"valueFrom\":{\"secretKeyRef\":{\"key\":\"password\",\"name\":\"mysecret1-1\"}}},{\"name\":\"SECRET_PASSWORD_2\",\"value\":\"0.0.7\"}],\"image\":\"redis\",\"name\":\"new-app-harness-example\"}]}}}}\n"] "name":"new-app-harness-example-deployment" "uid":"94143aa2-d668-4055-a3df-cf3db70fa13c"] "spec":map["revisionHistoryLimit":'\n' "progressDeadlineSeconds":'\u0258' "replicas":'\x01' "selector":map["matchLabels":map["app":"new-app-harness-example"]] "template":map["spec":map["containers":[map["terminationMessagePath":"/dev/termination-log" "terminationMessagePolicy":"File" "imagePullPolicy":"Always" "name":"new-app-harness-example" "image":"redis" "env":[map["name":"SECRET_USERNAME" "valueFrom":map["secretKeyRef":map["name":"mysecret1-1" "key":"username"]]] map["name":"SECRET_PASSWORD" "value":"pass"] map["name":"SECRET_PASSWORD_2" "value":"0.0.7"]] "resources":map[]]] "restartPolicy":"Always" "terminationGracePeriodSeconds":'\x1e' "dnsPolicy":"ClusterFirst" "securityContext":map[] "schedulerName":"default-scheduler"] "metadata":map["creationTimestamp":<nil> "labels":map["app":"new-app-harness-example" "harness.io/release-name":"release-f3fe376d-0d43-3346-a22b-5be4ba45247c"]]] "strategy":map["rollingUpdate":map["maxUnavailable":"25%" "maxSurge":"25%"] "type":"RollingUpdate"]] "status":map["updatedReplicas":'\x01' "readyReplicas":'\x01' "availableReplicas":'\x01' "conditions":[map["lastUpdateTime":"2021-09-17T19:49:19Z" "lastTransitionTime":"2021-09-17T19:49:19Z" "reason":"MinimumReplicasAvailable" "message":"Deployment has minimum availability." "type":"Available" "status":"True"] map["lastTransitionTime":"2021-09-17T19:49:18Z" "reason":"NewReplicaSetAvailable" "message":"ReplicaSet \"new-app-harness-example-deployment-67c8fbfcb9\" has successfully progressed." "type":"Progressing" "status":"True" "lastUpdateTime":"2021-09-17T20:03:48Z"]] "observedGeneration":'\x02' "replicas":'\x01']]}
ERROR 2021-09-17 17:04:03 for: "manifests.yaml": Deployment.apps "new-app-harness-example-deployment" is invalid: spec.template.spec.containers[0].env[1].valueFrom: Invalid value: "": may not be specified when `value` is not empty
Tutorial
Requirements
For this example, we will only need a basic Rolling deployment Workflow and some Deployment in our manifest that have environment variables populated by reference (to a secret, config map, among others), in this example I will populate them from a secret. For a better visibility, I’ve created a repository on GitHub that basically contains the resources existing in my Service Manifest
First Step
Let’s create our Service! For our newly created service we will use as a manifest all the resources that are in the repository I shared above:
Second Step
Let’s create an environment where our service will be deployed, there’s no secret here, basically, you define your target deployment infrastructure using a Harness Environment. Environments represent your deployment infrastructures, such as Dev, QA, Stage, Production, etc.
Environment Overview
Third Step
Finally, let’s now create our Rolling Workflow that will deploy our amazing service.
Workflow Overview
Now that everything is ready, we just need to hit the Deploy button for our Workflow for the first time 
Successfully deployed 
…
But after our magnificent deployment process, for some reason one of the developers identified a problem with the secret value used in our defined variables for our pods and manually edited the Deployments directly on the cluster using:
kubectl edit Deployment <deployment-name>
changing the secret referent to plain text in all our Deployments. Basically, updating from this:
containers:
- name: new-app-harness-example
image: redis
env:
- name: SECRET_USERNAME
valueFrom:
secretKeyRef:
name: mysecret1
key: username
- name: SECRET_PASSWORD
valueFrom:
secretKeyRef:
name: mysecret1
key: password
to this:
containers:
- name: new-app-harness-example
image: redis
env:
- name: SECRET_USERNAME
valueFrom:
secretKeyRef:
name: mysecret1
key: username
- name: SECRET_PASSWORD
value: 'pass'
After that, the secret value was fixed and we trigger our workflow again to update it properly and our deployments will again use the reference to our secret instead of a plain text value. easy-peasy…
Ohh no 
, my Workflow is no longer working, I’m getting this weird error message…
Final Step
Now we see this gross error for any new deployment, so to solve it we need to again edit the resources that we changed manually and remove the modified entries in the process, and after that perform the Deploy of our Workflow again so that the resources are recreated correctly:
So let’s edit the deployment that is having issues manually in the cluster, removing all the manually modified entries with:
kubectl edit Deployment <deployment-name>
For example, updating from this:
containers:
- name: new-app-harness-example
image: redis
env:
- name: SECRET_USERNAME
valueFrom:
secretKeyRef:
name: mysecret1
key: username
- name: SECRET_PASSWORD
value: 'pass'
to this:
containers:
- name: new-app-harness-example
image: redis
env:
- name: SECRET_USERNAME
valueFrom:
secretKeyRef:
name: mysecret1
key: username
Now that we have removed the entry that was changed and was causing the kubectl apply process to fail, let’s run our workflow again so that the variable excluded in the process is restored to our Deployment, but correctly.
All green again
Outcome
The error happens whenever a resource is edited directly and manually in the cluster after its default creation, and a new update on the manually edited resource is done through the standard process, kubectl apply for example So any manual changes made to Kubernetes resources deployed by the Harness process can present similar issues when a new deployment occurs, to avoid this, the ideal is not to perform any manual adjustments to the resources that are deployed through the Workflow process, but apply any change following the processes of release.
Now you don’t have to worry anymore about this trick that Kubernetes can present and continue exploring more possibilities with Harness and Kubernetes