Trying to play with harness, downloaded the Delegate and setup on AWS EKS using K8s yaml. It seemingly worked:
kubectl get pods -n harness-delegate
NAME READY STATUS RESTARTS AGE
harness-sample-k8s-delegate-skjrdc-0 0/1 Pending 0 56s
But in the harness console I do not see the delegate, So I am stuck.
What do I do fix this
You’ll need to make sure that the Delegate has 443 outbound to app.harness.io (usually through a security group)
Sam,
Hope you are well!
Hmm, can try describe what is going on with the pod.
kubectl describe pod
Or if the deployment is stuck:
kubectl get deployments
kubectl describe deployments
What do those say?
-Ravi
kubectl get pods -n harness-delegate
NAME READY STATUS RESTARTS AGE
harness-sample-k8s-delegate-skjrdc-0 0/1 Pending 0 6h17m
kubectl describe deployments
(returns nothing)
kubectl describe pod harness-sample-k8s-delegate-skjrdc-0
Error from server (NotFound): pods “harness-sample-k8s-delegate-skjrdc-0” not found
@sneakyaneurysm
How do I do this?
My SG on the cluster have outbound All/All/ 0.0.0.0
I had picked the the defaults on the harness node group:
API server endpoint access
Private access Disabled
Public access Enabled
Hmm weird. Can you deploy anything else on your EKS cluster? Are there other workloads running on said EKS cluster?
Usually something that is pending for so long is usually a resource issue e.g Kubernetes can’t place the work. By default the Harness Delegate needs 8gb of memory. Not sure of your machine types that are running at EKS Worker Nodes if they have enough space.
Are you using this for an example? If so I would suggest deleting the deployment/pod and giving that another whirl.
Can check out these Minikube steps for pruning the memory required [not recommended for production workloads]: “Harness K8s Steps”
-Ravi
I am using t3.medium - 4 GB. So I will have to upgrade to t3.xlarge - 16GB
I dont have anything else running on the cluster. It is just example/testing
Let me try that
Open the YAML for the delegate and change the memory to 4 Gi. Since you’re testing, as long as you don’t do anything super heavy, it should be fine.
How do I reapply?
I changed memory in yaml to 4Gi, also upgraded istances to 16GB
kubectl apply -f harness-delegate.yaml
namespace/harness-delegate unchanged
If I try delete
kubectl delete pods harness-sample-k8s-delegate-skjrdc-0
Error from server (NotFound): pods “harness-sample-k8s-delegate-skjrdc-0” not found
kubectl get pods
No resources found in default namespace.
Make sure you add the -n and specify the namespace in the command (should be harness-delegate).
Also, you might want to join the community slack channel, if you have slack.
kubectl apply -f harness-delegate.yaml -n harness-delegate
namespace/harness-delegate unchanged
clusterrolebinding.rbac.authorization.k8s.io/harness-delegate-cluster-admin unchanged
secret/harness-sample-k8s-delegate-proxy unchanged
statefulset.apps/harness-sample-k8s-delegate-skjrdc configured
kubectl get pods
No resources found in default namespace.
So odd…
How did you create your EKS Cluster? Using a CLI like EKSTL? What version of K8’s?
Try deploying the K8’s web ui:
https://docs.aws.amazon.com/eks/latest/userguide/dashboard-tutorial.html
If you can’t deploy that, your cluster most likely needs to be re-created.
In training class I have students all share an EKS cluster so multiple delegates running. You can re-download the delegate YAML and give another name and re-deploy since your EKS cluster is for learning,
-Ravi
try running kubectl get pods -n harness-delegate instead. If you aren’t changing the context before you run kubectl get pods then you will be running the command under the default namespace.
I recreated the node group with 8GB - large (not just restart the instance with new type), and set the YAML to 4Gi.
But still same situation.
kubectl get pods -n harness-delegate
NAME READY STATUS RESTARTS AGE
harness-sample-k8s-delegate-skjrdc-0 0/1 Pending 0 24h
Somehow when I apply it does not Replace the delegate - the age still shows 24h and still shows pending
K8s version 1.14 (EKS default), used the dashboard to create the cluster (not CLI).
In AWS/EKS First you create the cluster (VPC/Subnets). Next step create and attach the nodes/node group to the Cluster. You size the node group - number of instances and size/type of instance.
Sam,
How did you create your EKS Cluster? EKS for me is a funny one, if you don’t get it right the first go I never had any luck re-sizing infra, etc. The closest success I got is with EKSTL.
Can try force deleting the pod:
Re-download the Delegate YAML and rename.
-Ravi
I could deploy the the dashboard in my cluster
kubectl delete pods harness-sample-k8s-delegate-skjrdc-0 --grace-period=0 --force
(using the old pod name)
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
Error from server (NotFound): pods “harness-sample-k8s-delegate-skjrdc-0” not found
So I have done everything except recreating the cluster itself (I did re-create the node group)
???
kubectl get pods -n harness-delegate
NAME READY STATUS RESTARTS AGE
harness-delegate1-skjrdc-0 0/1 Pending 0 24m
So this returned something, but still pending. What does this mean - which one is running and which one is not?
kubectl -n harness-delegate delete pods harness-sample-k8s-delegate-skjrdc-0 --grace-period=0 --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod “harness-sample-k8s-delegate-skjrdc-0” force deleted
kubectl get pods -n harness-delegate
NAME READY STATUS RESTARTS AGE
harness-delegate1-skjrdc-0 0/1 Pending 0 30m
harness-sample-k8s-delegate-skjrdc-0 1/1 Running 0 70s
So 1 still pending after 30m ??
kubectl -n harness-delegate get deployments
No resources found in harness-delegate namespace.
So maybe I will start form scratch - destroy and re-create the cluster itself.
Note: There is some limit on this site - you can only reply so many times a day, so i can only edit a post 
Update
I recreated the cluster and the nodes with 16GB. It finally worked 
kubectl -n harness-delegate get pods
NAME READY STATUS RESTARTS AGE
harness-delegate1-skjrdc-0 1/1 Running 0 95s
Also I can see the delegate in the console
I feel like there is something up with your EKS Cluster. Depending how you created your EKS cluster, Kubernetes land is one of many interpretations. Even the AWS provided Cloud Formation Stack when kicking off an EKS cluster or modifying an EKS from the AWS Web Console
UI has a ton of nuances. Even EKS limits you to K8s 1.14, not even a 1.14.x release for bug fixes as Kubernetes 1.16.x is out.
Have you tried installing on another right-sized EKS cluster? How did you go about re-sizing your worker nodes EC2 instance sizes inside your EKS cluster?
I created a Sam-B-Test Cluster using Weave’s/Amazon’s CLI, EKSTL [https://eksctl.io/]
Installed the Delegate
Leads me to believe something is hosed in your K8s cluster.
Let us know if you can try somewhere else. e.g Minikube or another EKS cluster.
-Ravi
