1.0.0-rc.2: k8s 1.9.3: job pod reports missing scheme

Drone Support
1

Hiyoo

I have created my cluster role binding, service account, service, ingress, deployment, and secrets. I do not see any errors in the server logs. But I see an error in the logs of my drone-job, even though the job seems to have exited 0.

job status

$ kubectl -n drone get jobs
NAME              DESIRED   SUCCESSFUL   AGE
drone-job-wfsqr   1         1            19m

job log

$ kubectl -n drone logs drone-job-wfsqr-892r6 
{"arch":"amd64","level":"warning","machine":"ip-10-30-36-21.ec2.internal","msg":"runner: cannot get stage details","os":"linux","stage-id":1,"time":"2018-12-08T03:23:06Z"}
{"error":"parse ://rpc/v1/details: missing protocol scheme","level":"warning","msg":"program terminated","time":"2018-12-08T03:23:06Z"}

job env

      DRONE_STAGE_ID:     1
      DRONE_STAGE_ID:     1
      DRONE_LOGS_DEBUG:   true
      DRONE_LOGS_TRACE:   false
      DRONE_LOGS_COLOR:   false
      DRONE_LOGS_PRETTY:  false
      DRONE_LOGS_TEXT:    false
      DRONE_STAGE_ID:     1
      DRONE_RPC_PROTO:    
      DRONE_RPC_HOST:     
      DRONE_RPC_SERVER:   
      DRONE_RPC_SECRET:   MATCHES_SERVER
      DRONE_RPC_DEBUG:    false
      KUBERNETES_NODE:     (v1:spec.nodeName)
      DRONE_RUNNER_NAME:   (v1:spec.nodeName)

server env

      DRONE_KUBERNETES_ENABLED:    true
      DRONE_RPC_SECRET:            <set to the key 'secret' in secret 'drone-drone'>  Optional: false
      DRONE_KUBERNETES_NAMESPACE:  drone
      DRONE_TLS_AUTOCERT:          false
      DRONE_SERVER_HOST:           drone.example.com
      DRONE_SERVER_PROTO:          https
      DRONE_SECRET_ENDPOINT:       http://drone-drone-secrets:3000
      DRONE_SECRET_SECRET:         <set to the key 'secretService' in secret 'drone-drone'>  Optional: false
      DRONE_DATABASE_DATASOURCE:   /var/lib/drone/drone.sqlite
      DRONE_DATABASE_DRIVER:       sqlite3
      DRONE_GITHUB_CLIENT_ID:      ...
      DRONE_GITHUB_CLIENT_SECRET:  ...
      DRONE_LOGS_DEBUG:            true

server log

{"fields.time":"2018-12-08T03:22:41Z","latency":68693,"level":"debug","method":"POST","msg":"","remote":"10.32.5.205:38222","request":"/hook","request-id":"8c3c056856cf407446141004dd615698","time":"2018-12-08T03:22:41Z"}
{"fields.time":"2018-12-08T03:22:41Z","latency":429336,"level":"debug","method":"GET","msg":"","remote":"10.32.5.205:38230","request":"/api/badges/ORG/REPO/status.svg","request-id":"825ed7fe15920e2c56067e35b35e8542","time":"2018-12-08T03:22:41Z"}
{"fields.time":"2018-12-08T03:22:45Z","latency":11483,"level":"debug","method":"GET","msg":"","remote":"10.30.36.21:39980","request":"/","request-id":"1E3RbF9J0NAh9L3AffNfOpaamuz","time":"2018-12-08T03:22:45Z"}
{"fields.time":"2018-12-08T03:22:55Z","latency":12185,"level":"debug","method":"GET","msg":"","remote":"10.30.36.21:40134","request":"/","request-id":"1E3RcTNs22vzpmYFzDWHvd99A8F","time":"2018-12-08T03:22:55Z"}
{"commit":"0772b0052d4988e010cc4163327277b56c6c3c57","event":"push","level":"debug","msg":"webhook parsed","name":"REPO","namespace":"ORG","time":"2018-12-08T03:23:00Z"}
{"commit":"0772b0052d4988e010cc4163327277b56c6c3c57","event":"push","level":"debug","msg":"trigger: received","ref":"refs/heads/pc-add-drone","repo":"ORG/REPO","time":"2018-12-08T03:23:00Z"}
{"build-id":1,"level":"debug","msg":"kubernetes: creating job","repo-id":24,"stage-id":1,"stage-name":"default","stage-number":1,"time":"2018-12-08T03:23:00Z"}
{"build-id":1,"level":"debug","msg":"kubernetes: successfully created job","repo-id":24,"stage-id":1,"stage-name":"default","stage-number":1,"time":"2018-12-08T03:23:00Z"}
{"fields.time":"2018-12-08T03:23:00Z","latency":201639303,"level":"debug","method":"POST","msg":"","remote":"10.32.5.205:38494","request":"/hook","request-id":"751203cd1ead33d5559ba7ae3c07515b","time":"2018-12-08T03:23:00Z"}
{"fields.time":"2018-12-08T03:23:05Z","latency":10798,"level":"debug","method":"GET","msg":"","remote":"10.30.36.21:40308","request":"/","request-id":"1E3RdpslkpuCHnqFzrihpEPKRMn","time":"2018-12-08T03:23:05Z"}
{"fields.time":"2018-12-08T03:23:10Z","latency":2878802,"level":"debug","method":"GET","msg":"","remote":"10.32.5.205:38674","request":"/api/user/repos?latest=true","request-id":"612b8b3d983c3a1a008fd91b30003761","time":"2018-12-08T03:23:10Z"}
{"admin":true,"level":"debug","msg":"api: access granted","name":"REPO","namespace":"ORG","read":true,"request-id":"557698d29ede3ee91f56afcffd9593cc","time":"2018-12-08T03:23:13Z","user.login":"flah00","visibility":"private","write":true}
{"fields.time":"2018-12-08T03:23:13Z","latency":475608,"level":"debug","method":"GET","msg":"","remote":"10.32.5.205:38724","request":"/api/repos/ORG/REPO","request-id":"557698d29ede3ee91f56afcffd9593cc","time":"2018-12-08T03:23:13Z"}
{"admin":true,"level":"debug","msg":"api: access granted","name":"REPO","namespace":"ORG","read":true,"request-id":"1358df1f466a80cf8be685fa09a3f821","time":"2018-12-08T03:23:13Z","user.login":"flah00","visibility":"private","write":true}
{"fields.time":"2018-12-08T03:23:13Z","latency":1005774,"level":"debug","method":"GET","msg":"","remote":"10.32.5.205:38726","request":"/api/repos/ORG/REPO/builds","request-id":"1358df1f466a80cf8be685fa09a3f821","time":"2018-12-08T03:23:13Z"}
{"fields.time":"2018-12-08T03:23:15Z","latency":11913,"level":"debug","method":"GET","msg":"","remote":"10.30.36.21:40452","request":"/","request-id":"1E3Rf5Q0ysrUbhP59SofaVLBEpU","time":"2018-12-08T03:23:15Z"}
{"admin":true,"level":"debug","msg":"api: access granted","name":"REPO","namespace":"ORG","read":true,"request-id":"c5651a74a22828c6b130386f4e84418b","time":"2018-12-08T03:23:16Z","user.login":"flah00","visibility":"private","write":true}
{"fields.time":"2018-12-08T03:23:16Z","latency":421442,"level":"debug","method":"GET","msg":"","remote":"10.32.5.205:38770","request":"/api/repos/ORG/REPO","request-id":"c5651a74a22828c6b130386f4e84418b","time":"2018-12-08T03:23:16Z"}
{"admin":true,"level":"debug","msg":"api: access granted","name":"REPO","namespace":"ORG","read":true,"request-id":"f5bb0352dd3d2a33980b4d6a71857f47","time":"2018-12-08T03:23:16Z","user.login":"flah00","visibility":"private","write":true}
{"fields.time":"2018-12-08T03:23:16Z","latency":1212170,"level":"debug","method":"GET","msg":"","remote":"10.32.5.205:38772","request":"/api/repos/ORG/REPO/builds/1","request-id":"f5bb0352dd3d2a33980b4d6a71857f47","time":"2018-12-08T03:23:16Z"}
{"fields.time":"2018-12-08T03:23:25Z","latency":11417,"level":"debug","method":"GET","msg":"","remote":"10.30.36.21:40604","request":"/","request-id":"1E3RgI0HEYzCE7Fiim1ExydziS9","time":"2018-12-08T03:23:25Z"}
{"fields.time":"2018-12-08T03:23:35Z","latency":10823,"level":"debug","method":"GET","msg":"","remote":"10.30.36.21:40750","request":"/","request-id":"1E3RhaCw6JAoIQqjT5he8RxONK2","time":"2018-12-08T03:23:35Z"}
{"fields.time":"2018-12-08T03:23:45Z","latency":11509,"level":"debug","method":"GET","msg":"","remote":"10.30.36.21:40886","request":"/","request-id":"1E3Rim2HRQUTsJABqw50v6vQRhv","time":"2018-12-08T03:23:45Z"}
{"fields.time":"2018-12-08T03:23:55Z","latency":12637,"level":"debug","method":"GET","msg":"","remote":"10.30.36.21:41040","request":"/","request-id":"1E3Rk3bCwJd9MvV740JXw9c9neO","time":"2018-12-08T03:23:55Z"}
{"fields.time":"2018-12-08T03:24:05Z","latency":10558,"level":"debug","method":"GET","msg":"","remote":"10.30.36.21:41192","request":"/","request-id":"1E3RlKk5ZOE3XHoAh4NZKZRpI6I","time":"2018-12-08T03:24:05Z"}
{"fields.time":"2018-12-08T03:24:15Z","latency":10318,"level":"debug","method":"GET","msg":"","remote":"10.30.36.21:41328","request":"/","request-id":"1E3RmcqPwL6salVFm8wn81Gm1Fa","time":"2018-12-08T03:24:15Z"}
{"fields.time":"2018-12-08T03:24:25Z","latency":10458,"level":"debug","method":"GET","msg":"","remote":"10.30.36.21:41468","request":"/","request-id":"1E3RnmwtUmT2sjZUAQAowirMDE9","time":"2018-12-08T03:24:25Z"}
{"fields.time":"2018-12-08T03:24:35Z","latency":10723,"level":"debug","method":"GET","msg":"","remote":"10.30.36.21:41606","request":"/","request-id":"1E3Rp7JQ1ldap0Tk9FPWAQL7Ke6","time":"2018-12-08T03:24:35Z"}
{"fields.time":"2018-12-08T03:24:45Z","latency":11236,"level":"debug","method":"GET","msg":"","remote":"10.30.36.21:41742","request":"/","request-id":"1E3RqKDSVD0yZz3YDpBPKm7NxFg","time":"2018-12-08T03:24:45Z"}
{"fields.time":"2018-12-08T03:24:55Z","latency":13072,"level":"debug","method":"GET","msg":"","remote":"10.30.36.21:41894","request":"/","request-id":"1E3RrdzLn0dExWVLvIG3TyfDFff","time":"2018-12-08T03:24:55Z"}
{"fields.time":"2018-12-08T03:25:05Z","latency":21336,"level":"debug","method":"GET","msg":"","remote":"10.30.36.21:42040","request":"/","request-id":"1E3RsrKjCjc6Gav4D9a1n1kP2pi","time":"2018-12-08T03:25:05Z"}
{"fields.time":"2018-12-08T03:25:15Z","latency":10713,"level":"debug","method":"GET","msg":"","remote":"10.30.36.21:42170","request":"/","request-id":"1E3Ru8WamLEwLrzGRSYp23CtsuY","time":"2018-12-08T03:25:15Z"}
{"fields.time":"2018-12-08T03:25:25Z","latency":12727,"level":"debug","method":"GET","msg":"","remote":"10.30.36.21:42318","request":"/","request-id":"1E3RvJxdbgbhyzwJ3PuDRSC34qm","time":"2018-12-08T03:25:25Z"}
{"fields.time":"2018-12-08T03:25:35Z","latency":10951,"level":"debug","method":"GET","msg":"","remote":"10.30.36.21:42464","request":"/","request-id":"1E3RwZgrmOVpPXiLixakPpPMwdq","time":"2018-12-08T03:25:35Z"}
{"fields.time":"2018-12-08T03:25:45Z","latency":12663,"level":"debug","method":"GET","msg":"","remote":"10.30.36.21:42592","request":"/","request-id":"1E3Rxrk660hILZUpJU5yztZsieL","time":"2018-12-08T03:25:45Z"}
2

ah, I think this is another miss in my documentation. You need to set the following values:

DRONE_RPC_PROTO=https
DRONE_RPC_HOST=drone.example.com

These values should be optional and should default to the DRONE_SERVER_HOST and DRONE_SERVER_PROTO if unset. This is an oversight on my part. I will submit a patch to correct this behavior, so that you are not required to set the parameters by default.

Thank you for providing such detailed information. This made it really easy to spot the problem :slight_smile:

3

also just a heads up, I have not hooked up secret plugins yet for Drone for Kubernetes (so DRONE_SECRET_ENDPOINT won’t work). I will fix this first thing tomorrow!!!

4

YW and thanks for your hard work. I can’t wait to move large swathes of my jenkins pipeline into drone… and burn my jenkins groovy code in fire.

Also, I figured kubernetes-secrets hadn’t been included in this release, just yet. I left it in, anticipating its return. :wink:

5

Not to be a nudnik … but I wanted to also explicitly call out the false positive the job returned. There was an error, but it seems the process exited 0. I’m sure that’s going to bite someone soon.

Also, the status of the job never changed. The hour glass continued to jiggle, in the UI.

6

yep, the Job should ideally return a non-zero exit code when we encounter an unexpected error (like inability to reach the server). I will make this update. Also, I patched the previous issue and setting the RPC host and server is no longer required in the latest image.

Also, the status of the job never changed. The hour glass continued to jiggle, in the UI.

yes this is actually expected, because in this case the Pipeline Controller was never able to send back an error to the server, because it did not have the server address.

7

I’m an excitable person, so I manually set the RPC env vars, to see what would happen, memory issues.

job logs

$ kubectl -n drone logs drone-job-rgfjd-mbnsl
{"arch":"amd64","build":1,"error":"runtime error: invalid memory address or nil pointer dereference","level":"error","machine":"ip-10-30-36-21.ec2.internal","msg":"runner: unexpected panic","os":"linux","pipeline":"default","repo":"Adaptly/connector","stage":1,"time":"2018-12-08T05:35:28Z"}

job env

      DRONE_STAGE_ID:     1
      DRONE_STAGE_ID:     1
      DRONE_LOGS_DEBUG:   true
      DRONE_LOGS_TRACE:   false
      DRONE_LOGS_COLOR:   false
      DRONE_LOGS_PRETTY:  false
      DRONE_LOGS_TEXT:    false
      DRONE_STAGE_ID:     1
      DRONE_RPC_PROTO:    https
      DRONE_RPC_HOST:     drone.example.com
      DRONE_RPC_SERVER:   drone
      DRONE_RPC_SECRET:   ...
      DRONE_RPC_DEBUG:    false
      KUBERNETES_NODE:     (v1:spec.nodeName)
      DRONE_RUNNER_NAME:   (v1:spec.nodeName)
8

hmm, very strange, I haven’t hit that issue yet. I will see if I can reproduce. Unfortunately it looks like the panic does not include the full stack trace (I think this is my fault with how I handled it) which could make this a bit more difficult to hunt down … can you share the yaml you used that caused the problem?

9

Two env vars are… null? Because I commented out the secret keys for the external data, since they couldn’t be set.

kind: pipeline
name: default
steps:
  - name: test
    image: clojure
    pull: always
    commands:
      - env
      - ./scripts/test.sh
    environment:
      POSTGRES_PORT_5432_TCP_ADDR: postgres
      POSTGRES_PORT_5432_TCP_PORT: 5432
      REDIS_PORT_6379_TCP_ADDR: redis
      REDIS_PORT_6379_TCP_PORT: 6379
      LEIN_USERNAME:
      LEIN_PASSPHRASE:
10

I’m not sure I understand the question. Can you clarify?

11

Sorry, I was pointing out that in my yaml I’d temporarily commented out

environment:
  LEIN_USERNAME:
    from_secret: foo
  LEIN_PASSPHRASE:
    from_secret: bar

This rendered the environment variables values null/nil/empty.

  environment:
    LEIN_USERNAME:
    LEIN_PASSPHRASE:

This is what lead to the invalid memory address/nil pointer error. Once I removed these null value env vars, the error went away.

12

Now I have a new issue, the job created uses the default service account for the namespace. But my default service account has not been granted admin access to the cluster. I created a new service account, for this purpose. So, we need some means to pass the service account the job will use.

In the meantime, I can force the default account to be granted cluster admin.

job pod

$ kubectl -n drone describe po drone-job-rsvvg-8zhn6
...
    Environment:
      DRONE_STAGE_ID:     2
      DRONE_STAGE_ID:     2
      DRONE_LOGS_DEBUG:   true
      DRONE_LOGS_TRACE:   false
      DRONE_LOGS_COLOR:   false
      DRONE_LOGS_PRETTY:  false
      DRONE_LOGS_TEXT:    false
      DRONE_STAGE_ID:     2
      DRONE_RPC_PROTO:    https
      DRONE_RPC_HOST:     drone.example.com
      DRONE_RPC_SERVER:   drone
      DRONE_RPC_SECRET:   ...
      DRONE_RPC_DEBUG:    false
      KUBERNETES_NODE:     (v1:spec.nodeName)
      DRONE_RUNNER_NAME:   (v1:spec.nodeName)
...
Volumes:
  default-token-kkkrt:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-kkkrt
    Optional:    false

job logs

$ kubectl -n drone logs -f drone-job-rsvvg-8zhn6
{"arch":"amd64","build":2,"level":"info","machine":"ip-10-30-36-21.ec2.internal","msg":"runner: start execution","os":"linux","pipeline":"default","repo":"ORG/REPO","stage":1,"time":"2018-12-09T18:17:32Z"}
{"arch":"amd64","build":2,"level":"debug","machine":"ip-10-30-36-21.ec2.internal","msg":"runner: watch for kill signal","os":"linux","pipeline":"default","repo":"ORG/REPO","stage":1,"time":"2018-12-09T18:17:32Z"}
{"arch":"amd64","build":2,"error":"namespaces is forbidden: User \"system:serviceaccount:drone:default\" cannot create namespaces at the cluster scope","level":"info","machine":"ip-10-30-36-21.ec2.internal","msg":"runner: execution failed","os":"linux","pipeline":"default","repo":"ORG/REPO","stage":1,"time":"2018-12-09T18:17:32Z"}
{"arch":"amd64","build":2,"error":"namespaces is forbidden: User \"system:serviceaccount:drone:default\" cannot create namespaces at the cluster scope","level":"debug","machine":"ip-10-30-36-21.ec2.internal","msg":"runner: received kill signal","os":"linux","pipeline":"default","repo":"ORG/REPO","stage":1,"time":"2018-12-09T18:17:32Z"}
13

ah I see. thanks, I will make sure we handle this edge case.

EDIT: I created this issue.

14

In the meantime, I can force the default account to be granted cluster admin.

Yes, this is what we currently recommend although I definitely understand it may not be ideal:
https://docs.drone.io/installation/github/kubernetes/#namespaces-is-forbidden

So, we need some means to pass the service account the job will use.

Can you provide me a reference to how one would do this? Is it specified somewhere in the Job yaml specification? This should be relatively easy to implement, but I’m still learning Kubernetes so I could use some direction :slight_smile:

15

If you specify serviceAccount serviceAccountName for the pod spec you should be able to control the token used. So long as the server takes an env var, to configure the service account, this should be easy peasy.

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

16

ok thanks, I see, I am going to patch 1.0.0-rc.2 to include DRONE_KUBERNETES_SERVICE_ACCOUNT so that you can set the service account. A new snapshot should be available in ~15 minutes.