Hi all, I’m have trouble getting my first pipeline to actually build. I see the build start, but the steps never execute and the console log just shows Loading... forever. Let me provide some details. I have an install of drone and gitea in a local raspberry pi 4 k3s cluster. In my k3s cluster I have a drone-server pod and drone-runner pod. They both use the same secret drone-secrets to set environment variables (which of course includes the DRONE_RPC_SECRET). Drone is configured to use a local postgres DB (I did have to modify the schema for Drone to work though, for some reason the tokens I was getting from gitea were larger than 500 characters (they were 838 characters long) so I had to increase the oauth token and oauth refresh field sizes - this was a strange issue but I’d doubt it is related). Drone is also configured to connect to gitea. I have a small branch for a hello world website that when I push changes to, Drone sees the changes and starts a new build. The pipeline .drone.yml specifies the platform os and arch and for a while I had a label specification as well for arm64 (as I intend to add an amd64 node and try doing architecture specfic builds). It seems this pipeline yaml is seen by Drone as in the UI, I’ll see builds kicking off with its name. In the logs for the drone-server, I see logs that say request queue item and context canceled alot. Whereas in the drone-runner logs, I see a lot of no content returned messages. It seems like the drone server is requesting for a job to start and the runner looks for a job but doesn’t see it or ignores it or something. There seems to be some break down in the handoff from drone-server to drone-runner. Ok that is my best understanding so far. I could be totally wrong. Now let me provide actual console outputs rather than just a colloquial summary.
drone-secrets
root@atlas00:~# k describe secret -n drone drone-secrets
Name: drone-secrets
Namespace: drone
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
DOCKER_PASSWORD: 32 bytes
DOCKER_USERNAME: 5 bytes
DRONE_DATABASE_DATASOURCE: 90 bytes
DRONE_GITEA_CLIENT_ID: 36 bytes
DRONE_GITEA_CLIENT_SECRET: 44 bytes
DRONE_POSTGRES_PASSWORD: 32 bytes
DRONE_POSTGRES_USERNAME: 5 bytes
DRONE_RPC_SECRET: 32 bytes
drone-runner-arm64 deployment
root@atlas00:~# cat drone-runner-arm64-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: drone-runner-arm64
namespace: drone
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: drone-runner-arm64
template:
metadata:
labels:
app: drone-runner-arm64
spec:
serviceAccountName: drone-runner
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/arch
operator: In
values:
- arm64
containers:
- name: drone-runner
image: drone/drone-runner-docker:latest
envFrom:
- secretRef:
name: drone-secrets
env:
- name: DRONE_RPC_HOST
value: "drone.drone.svc.cluster.local"
- name: DRONE_RPC_PROTO
value: "http"
- name: DRONE_RUNNER_LABELS
value: architecture:arm64
- name: DRONE_RUNNER_CAPACITY
value: "1"
- name: DRONE_RUNNER_NAMESPACE
value: drone
- name: DRONE_LOGS_TRACE
value: "true"
- name: DRONE_LOGS_DEBUG
value: "true"
- name: DRONE_RUNNER_PLATFORM_OS
value: "linux"
- name: DRONE_RUNNER_PLATFORM_ARCH
value: "arm64"
volumeMounts:
- name: runner-storage-arm64
mountPath: /drone-runner-storage
- name: docker-socket
mountPath: /var/run/docker.sock
volumes:
- name: runner-storage-arm64
persistentVolumeClaim:
claimName: drone-runner-storage-arm64
- name: docker-socket
hostPath:
path: /var/run/docker.sock
root@atlas00:~# k describe deployment -n drone drone-runner
Name: drone-runner-arm64
Namespace: drone
CreationTimestamp: Fri, 12 May 2023 08:52:54 -0500
Labels: <none>
Annotations: deployment.kubernetes.io/revision: 10
Selector: app=drone-runner-arm64
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: Recreate
MinReadySeconds: 0
Pod Template:
Labels: app=drone-runner-arm64
Service Account: drone-runner
Containers:
drone-runner:
Image: drone/drone-runner-docker:latest
Port: <none>
Host Port: <none>
Environment Variables from:
drone-secrets Secret Optional: false
Environment:
DRONE_RPC_HOST: drone.drone.svc.cluster.local
DRONE_RPC_PROTO: http
DRONE_RUNNER_LABELS: architecture:arm64
DRONE_RUNNER_CAPACITY: 1
DRONE_RUNNER_NAMESPACE: drone
DRONE_LOGS_TRACE: true
DRONE_LOGS_DEBUG: true
DRONE_RUNNER_PLATFORM_OS: linux
DRONE_RUNNER_PLATFORM_ARCH: arm64
Mounts:
/drone-runner-storage from runner-storage-arm64 (rw)
/var/run/docker.sock from docker-socket (rw)
Volumes:
runner-storage-arm64:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: drone-runner-storage-arm64
ReadOnly: false
docker-socket:
Type: HostPath (bare host directory volume)
Path: /var/run/docker.sock
HostPathType:
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: drone-runner-arm64-7777bd7d95 (1/1 replicas created)
Events: <none>
drone-runner logs
root@atlas00:~# k logs -n drone drone-runner-arm64-7777bd7d95-rh26z
time="2023-05-13T17:22:29Z" level=debug msg="successfully pinged the docker daemon"
time="2023-05-13T17:22:29Z" level=info msg="starting the server" addr=":3000"
time="2023-05-13T17:22:29Z" level=info msg="successfully pinged the remote server"
time="2023-05-13T17:22:29Z" level=info msg="polling the remote server" arch=arm64 capacity=1 endpoint="http://drone.drone.svc.cluster.local" kind=pipeline os=linux type=docker
time="2023-05-13T17:22:29Z" level=debug msg="poller: request stage from remote server" thread=1
time="2023-05-13T17:22:59Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2023-05-13T17:23:39Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2023-05-13T17:24:19Z" level=trace msg="http: no content returned: re-connect and re-try"
...
time="2023-05-13T18:31:40Z" level=trace msg="http: no content returned: re-connect and re-try"
drone-server deployment
root@atlas00:~# cat drone-server-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: drone-server
namespace: drone
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: drone-server
template:
metadata:
labels:
app: drone-server
spec:
containers:
- name: drone-server
image: drone/drone:latest
envFrom:
- secretRef:
name: drone-secrets
- configMapRef:
name: drone-config
env:
- name: DRONE_LOGS_TRACE
value: "true"
- name: DRONE_LOGS_DEBUG
value: "true"
ports:
- containerPort: 80
volumeMounts:
- name: server-storage
mountPath: /var/lib/drone
volumes:
- name: server-storage
persistentVolumeClaim:
claimName: drone-server-storage
root@atlas00:~# k describe deployment -n drone drone-server
Name: drone-server
Namespace: drone
CreationTimestamp: Thu, 11 May 2023 14:47:30 -0500
Labels: <none>
Annotations: deployment.kubernetes.io/revision: 2
Selector: app=drone-server
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: Recreate
MinReadySeconds: 0
Pod Template:
Labels: app=drone-server
Containers:
drone-server:
Image: drone/drone:latest
Port: 80/TCP
Host Port: 0/TCP
Environment Variables from:
drone-secrets Secret Optional: false
drone-config ConfigMap Optional: false
Environment:
DRONE_LOGS_TRACE: true
DRONE_LOGS_DEBUG: true
Mounts:
/var/lib/drone from server-storage (rw)
Volumes:
server-storage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: drone-server-storage
ReadOnly: false
Conditions:
Type Status Reason
---- ------ ------
Progressing True NewReplicaSetAvailable
Available True MinimumReplicasAvailable
OldReplicaSets: <none>
NewReplicaSet: drone-server-5c9fc74f67 (1/1 replicas created)
Events: <none>
drone-server logs
{"build.limit":0,"expires":"0001-01-01T00:00:00Z","kind":"trial","level":"debug","msg":"main: license loaded","repo.limit":0,"time":"2023-05-13T17:51:35Z","user.limit":0}
{"interval":"30m0s","level":"info","msg":"starting the cron scheduler","time":"2023-05-13T17:51:35Z"}
{"interval":"24h0m0s","level":"info","msg":"starting the zombie build reaper","time":"2023-05-13T17:51:35Z"}
{"acme":false,"host":"drone.bstein.dev","level":"info","msg":"starting the http server","port":":80","proto":"https","time":"2023-05-13T17:51:35Z","url":"https://drone.bstein.dev"}
{"arch":"arm64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"linux","time":"2023-05-13T17:51:49Z","type":"docker","variant":""}
{"arch":"arm64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: context canceled","os":"linux","time":"2023-05-13T17:52:19Z","type":"docker","variant":""}
...
{"commit":"ea822e1eb2f1a8b67a0f0cec2f225d9d9acd4078","event":"push","level":"debug","msg":"webhook parsed","name":"bstein-dev-backend","namespace":"brad_stein","time":"2023-05-13T17:53:03Z"}
{"commit":"ea822e1eb2f1a8b67a0f0cec2f225d9d9acd4078","event":"push","level":"debug","msg":"trigger: received","ref":"refs/heads/main","repo":"brad_stein/bstein-dev-backend","time":"2023-05-13T17:53:03Z"}
{"authtype":"cookie","fields.time":"2023-05-13T17:53:03Z","latency":296651662,"level":"debug","method":"POST","msg":"","remote":"10.42.8.11:43132","request":"/hook?secret=phkHY8ioMge760nmPPCYk9GjSSqrNrzv","request-id":"2PkU4f6O0gEsPDAS1uDbO47aoxf","time":"2023-05-13T17:53:03Z"}
{"arch":"arm64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"linux","time":"2023-05-13T17:53:10Z","type":"docker","variant":""}
{"arch":"arm64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: context canceled","os":"linux","time":"2023-05-13T17:53:40Z","type":"docker","variant":""}
{"arch":"arm64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"linux","time":"2023-05-13T17:53:50Z","type":"docker","variant":""}
{"arch":"arm64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: context canceled","os":"linux","time":"2023-05-13T17:54:20Z","type":"docker","variant":""}
...
{"level":"debug","msg":"cron: begin process pending jobs","time":"2023-05-13T18:21:35Z"}
{"level":"debug","msg":"cron: found 0 pending jobs","time":"2023-05-13T18:21:35Z"}
{"level":"debug","msg":"cron: finished processing jobs","time":"2023-05-13T18:21:35Z"}
...
{"arch":"arm64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"linux","time":"2023-05-13T18:33:10Z","type":"docker","variant":""}
{"arch":"arm64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: context canceled","os":"linux","time":"2023-05-13T18:33:40Z","type":"docker","variant":""}
{"arch":"arm64","kernel":"","kind":"pipeline","level":"debug","msg":"manager: request queue item","os":"linux","time":"2023-05-13T18:33:50Z","type":"docker","variant":""}
.drone.yml pipeline file
kind: pipeline
type: docker
name: build-arm64
platform:
os: linux
arch: arm64
# labels:
# architecture: arm64
steps:
- name: build_and_push_arm64
image: docker
environment:
DOCKER_USERNAME:
from_secret: docker_username
DOCKER_PASSWORD:
from_secret: docker_password
commands:
- docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD registry.bstein.dev
- docker build -f Dockerfile.arm64 -t registry.bstein.dev/bstein-dev-backend:${DRONE_COMMIT_SHA}-arm64 -t registry.bstein.dev/bstein-dev-backend:arm64 .
- docker push registry.bstein.dev/bstein-dev-backend:${DRONE_COMMIT_SHA}-arm64
- docker push registry.bstein.dev/bstein-dev-backend:arm64
# ---
# kind: pipeline
# type: docker
# name: build-x86_64
# platform:
# os: linux
# arch: amd64
# labels:
# architecture: amd64
# steps:
# - name: build_and_push_x86_64
# image: docker
# environment:
# DOCKER_USERNAME:
# from_secret: docker_username
# DOCKER_PASSWORD:
# from_secret: docker_password
# commands:
# - docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD registry.bstein.dev
# - docker build -f Dockerfile.x86-64 -t registry.bstein.dev/bstein-dev-backend:${DRONE_COMMIT_SHA}-x86_64 -t registry.bstein.dev/bstein-dev-backend:x86_64 .
# - docker push registry.bstein.dev/bstein-dev-backend:${DRONE_COMMIT_SHA}-x86_64
# - docker push registry.bstein.dev/bstein-dev-backend:x86_64
There are some additional entries in the drone-server log that I omitted because I think they are just rest calls from interacting with the UI. Hopefully there is enough information for someone here who is more familiar with drone to see what I did wrong or what is missing and point it out to me. I’m sure it is something small and simple. I’ve tried for a couple days to figure it out and I’m on my third re-install trying to figure it out. Any insight as to why I can see the build being kicked off in the Drone web UI but it never gets beyond Loading... in the console log of the web ui for each build, would be most appreciated. Thanks.