This post will be addressing a rather annoying issue with Kubernetes: Immutable Selectors
The premise of the issue is commonly found when a Kubernetes Workload, specifically a Deployment, exists in a namespace and a user wants to start using a Canary workflow to deploy into that namespace. When Harness executes a deployment, there are certain selectors that Harness has to add for proper tracking. If the workload has always been deployed in Canary style by Harness, then there are no issues. But, if the workload already exists in the namespace, Kubernetes will tell you that there is an error because the Label Selector field is immutable.
Good rolling deployment
Bad canary deployment
Notice the error in the bad canary workflow states that the Label Selector field is immutable. This would be an issue whether deploying with Harness or going to the cluster and trying to edit the deployment with a new selector (i.e. kubectl edit deployment or using a kubectl patch command).
This can cause some issues with trying to do a canary deployment because it seems like everything should just work as expected. And the typical way to fix this issue is by deleting the deployment in Kubernetes first, then deploying the canary workflow after the deletion is complete. But this can cause hesitation, especially since canary deployments are typically done in Production, and there is a desire to not have a live application stop service for a period of time to get this deployment out in a canary style.
To help solve this issue in an automated way, this post will walk through a workflow design for the migration process.
Complete execution
At a high level, this workflow and pipeline will accomplish the following outcomes:
- Deploy a temporary workload with the same network connections as the main workload. This will allow for consistent uptime as the migration happens.
- Delete the main workload. Since the fields are immutable, the only way to get the canary deployment out successfully is to delete it.
- Deploy out the Canary workflow. This will establish the deployment with the correct label selectors to continue with canary deployments in the future.
- Delete the temporary workload. A final cleanup step to remove the workloads that are not needed going forward.
Setup Requirements
To accomplish a successful migration deployment the Service in Harness needs to have the workload name templatized:
This will allow you to override the name for the migration deployment, but also for ongoing deployments (to add more flexibility in the core manifest usage). In this example I am using a Service Variable to provide the override.
Workflow Requirements
There are some built-in variables that Harness automatically creates that cannot be unset in a workflow. As such, this migration will require two main workflows that are connected in a pipeline.
Temp Workload
The first workflow will consist of two main steps. The first one is to deploy out the temporary workload and the second is to delete the main workload after the temporary one has been created (you can add a verification step to validate that traffic is received on the temporary workload, if needed):
When you create the workflow, there is a section for Service Variable Overrides. If you followed the example in this post you will want to specify the Service Variable that you created and add a -temp to the end of the variable:
In the Wrap Up phase of the workflow, add a Delete step that will delete the old workload:
(NOTE: I deployed the original rolling deployment with Harness, which is why I added the deployment to the end of the Resources section. If you did not do that, make sure you add the correct name of the deployment, or the appropriate variable to the end)
Canary Workflow
The next workflow to create is the Canary workflow, which will start with the canary deployment, and then finish with the main deployment and the clean-up.
Canary Deployment
Full Deployment
In the Full Deployment phase, the first step is the basic Rolling Deployment. The Wrap Up phase is where the clean-up work will happen. The first thing to do is add a Delete step to clean-up the canary workload that was deployed by using the ${k8s.canaryWorkload} for the Resources section:
The next Delete step is to clean-up the temporary workload we previously deployed:
And to get really fancy with this clean-up phase, you can execute the two Delete steps in parallel:
Migrate Pipeline
The last thing to do is string the two workflows together using a Pipeline:
Expansion Ideas
In the case of one Service being migrated, everything in the workflows can be rather static. However, the most likely scenario is that there are other Services that need to migrate to a Canary workflow. To accomplish this you can Templatize a Workflow to share the Workflows and Pipelines across multiple Services.
Hopefully this helps! Let me know of any questions you might have.
Don’t forget to Like, Comment, and Share!