K8s Blue-Green Traffic splitting without istio

rohan · May 4, 2020, 8:42pm

Traffic Splitting without Istio

Hi fellow canaries, my fellow Blackbelt @Venkat has written an excellent article on Traffic Splitting without Istio. I will be sharing it with the team here. Many of our users have asked for this:

We do have this in our docs, however, also sharing it here as another source of information.

Implementing Traffic Splitting in Harness without Istio!

Using Harness, application deployment is automated that enables teams to release more frequently with the fast rollback in case of any failure. Typically, applications are promoted to production after going through various stages of testing - QA, UAT, SIT etc. To minimize disruptions, there are different deployment strategies such as Blue-Green or Canary deployment which allows both old and new versions of the applications to co-exist before completely switching over to the latest version in production.

Key Concepts/Features:

Annotation to specify primary service: Harness - Self-Service CI/CD Tool for DevOps Teams & Engineers
Annotation to specify stage service: harness.io/stage-service
Using harness.io/skip-file-for-deploy for ingress resources
Using Approval step allows for manual review and proceed
Native capabilities of Ingress controllers to support traffic splitting.

Pre-requisites:

Kubernetes cluster with ingress controller deployed that supports traffic splitting.

The technique described here uses existing Harness features which can be adapted to both blue-green and canary deployment. For applications that are implemented using service mesh - such as ISTIO, Harness has native integration with Istio for Traffic splitting. For applications that are not implemented using service mesh, Harness makes it possible to implement traffic splitting using the native features of ingress controllers - such as Nginx.

Let us start with a Kubernetes Blue-Green deployment using Harness. With Blue-Green deployment, two different versions of the same application are deployed to a Kubernetes cluster. Two Kubernetes services - primary and stage are defined with harness specific annotation

harness.io/primary-service: true

harness.io/stage-service: true

When the workflow is deployed, Harness modifies the selector at runtime to add harness.io/color with values “blue” and “green”, to redirect traffic always to the “primary service” i.e current version. After route update where the primary and stage is swapped, the primary service routes requests to the new version based on the selector - harness.io/color. In this model, only one version of the application is servicing the requests at any time.

While the blue-green deployment model is a nice improvement for customers over rolling updates because of efficiency in rollback with almost no downtime, there is still a possibility of unexpected failures. To minimize the risk in such a situation, applications can be deployed such that both versions - old and new are active at the same time handling requests. The advantage of this approach is the ability to test new versions safely and securely in production environments alongside the old version by limiting the exposure to a small percentage of requests. This also provides the flexibility to gradually increase traffic to new versions or stop the requests to the new version in the event of any errors at higher loads.

Harness blue-green workflow can be easily modified to support traffic splitting with minimal changes. In this example, we are using the weight-based splitting of traffic using Nginx ingress controller. Nginx controller also supports other ways to split traffic such as header or path but that is not the focus of this article. The standard blue-green workflow is modified to add approval steps to progressively increasing traffic to the new versions using ingress resources with annotation to specify the “weight” i.e percentage of the traffic to be handled by the “stage” service i.e new version.

Here is a modified workflow with additional steps to demonstrate traffic splitting:

After Step 1 (Deploy), all traffic is still handled by the old version (primary). In Step 2, 25% of traffic is sent to the new version (stage) after approval. After reviewing the test results, traffic increased to 50%. Step 3 - performs the swap resulting in a new version becoming the primary version. The new version takes on additional load to handle 75% followed by 100% of traffic.

Here is the sample approval step:

To implement traffic split, add three ingress resources to the harness service defined as below (named using the weight i.e percentage of the split):

1. Ingress-traffic-split0.yaml
2. Ingress-traffic-split25.yaml
3. Ingress-traffic-split50.yaml

Use the following YAML file as a template for creating the above three files, by replacing the weight factor to 0, 25, 50.

# harness.io/skip-file-for-deploy
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: {{.Values.name}}-stage
  
  labels:
    app: {{.Values.name}}
  annotations:
    kubernetes.io/ingress.class: "nginx"

    # Enable canary and send 25% of traffic to version 2
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "25"
spec:
  rules:
  - host: {{.Values.name}}.com
    http:
      paths:
      - backend:
          serviceName: {{.Values.name}}-stage
          servicePort: 80

Technical details:
Use # harness.io/skip-file-for-deploy with kubectl apply step to change percentage. This is not needed for #1 - Ingress-traffic-split0.yaml.
Nginx specific annotations to specify the percentage of traffic to route to stage service.
annotations:
kubernetes.io/ingress.class: "nginx”

Enable canary and send 25% of traffic to version 2

nginx.ingress.kubernetes.io/canary: “true”
nginx.ingress.kubernetes.io/canary-weight: “25”

The backend/serviceName should be referring to the “stage” service

            `serviceName: {{.Values.name}}-stage`

The ingress resource to split traffic are referred in kubectl apply step as below:

Please refer to the screenshot below for a working version of a workflow with traffic splitting

The output of “runtest1”, a shell script sending 100 requests to the endpoint using curl:

References

Reference Images

Docs: https://docs.harness.io/l/en/article/tkubd954r6-traffic-splitting-without-istio