Mutable Delegate(Existing Delegate) - If the user reuses the StatefulSet Name since a new delegate rolled out in a different cluster with a different delegate name
Expected Behaviour: In the case of Mutable delegates (Current Delegates) Harness delegate logs show that the delegate is up and running but Harness UI will show disconnected, analyzing the logs of the delegate revel that the new delegate leverage id of the existing delegate and show that it’s up and running. This behavior is observed due to various actions that can happen
- A new delegate is registered
- A delegate is upgraded
- The delegate pod can get restarted
- A number of replicas can come for the same delegate group
Whenever the delegate is getting upgraded or Pod restart Harness reuse the same delegate, in order do so Harness rely on
- Delegate name
- Delegate hostname - How Does Harness Manager Identify Delegates?
The current Mutable delegates use a StatefulSet, which means the hostname remains the same, it also infers that the same delegate YAML is getting applied in two different clusters. The harness manager will receive the same delegate entry from two different pods which is running different clusters. Harness make use of StatefulSet name to determine hostname both will match causing confusion. Harness doesn’t use the IP address to determine unique pods in the different cluster but this will cause confusion if one or more delegate pods getting restarted or scheduled in the different cluster as it results in different IP address. Always give a unique name for StatefulSet Name or Use Immutable Delegate
Immutable Delegate:
In the case of immutable delegates Harness make use of deployment instead of StatefulSet, what it means is that pod names and hostnames will always be unique across many clusters. So if someone uses the YAML in two different clusters Harness identifies and reflects more delegate instances in UI for the same delegate name.