Problem of deployment on two nodes

moyamos · August 10, 2019, 12:13pm

Hello,

I want to deploy my container on two different machines (nodes). I have run a drone “Multi-Machine” container on a server. Also, I have run a runner container on the same machine of Multi-Machine server. Another runner is on a different machine. I am encountering a strange issue. After pushing on my GitLab repository only an instance of nodes randomly start to deploy. Never two deployments at the same time! In addition, if I stop one of the node runners the deployment stuck in the first clone step. However, if I mention only one node in the node section of my pipeline it works fine for both servers!

Here I am sharing my commands to run the server and two runners, pipeline, and logs of the server and runners to receive your idea of how can I resolve the problem.

Running a server:

docker run --volume=/var/run/docker.sock:/var/run/docker.sock 
           --volume=/var/lib/drone:/data 
           --env=DRONE_GIT_ALWAYS_AUTH=false 
           --env=DRONE_GITLAB_SERVER=http://x.x.x.x -
           -env=DRONE_GITLAB_CLIENT_ID=<ID>
           --env=DRONE_GITLAB_CLIENT_SECRET=<Secret>
           --env=DRONE_RPC_SECRET=<RPC_Secret>
           --env=DRONE_SERVER_HOST=drone.server.com:1234
           --env=DRONE_SERVER_PROTO=http 
           --env=DRONE_AGENTS_ENABLED=true 
           --env=DRONE_TLS_AUTOCERT=false 
           --env=DRONE_USER_CREATE=username:yaghobi,admin:true 
           --publish=3310:80 
           --publish=3311:443 
           --restart=always 
           --detach=true 
           --name=drone_multi 
           drone/drone:1

First runner:

docker run -d 
           -e DRONE_RPC_PROTO=http 
           -e DRONE_RPC_HOST=drone.server.com:1234   
           -e DRONE_RPC_SECRET=<RPC_Secret>   
           -e DRONE_RUNNER_CAPACITY=2   
           -e DRONE_RUNNER_NAME=srv1   
           -e DRONE_RUNNER_LABELS=server:srv1   
           -e DRONE_LOGS_DEBUG=true   
           -e DRONE_RPC_DEBUG=true  
           -p 3000:3000   
           -v /var/run/docker.sock:/var/run/docker.sock   
           --restart always   
           --name runner   
           drone/agent:1

Second runner:

docker run -d 
           -e DRONE_RPC_PROTO=http 
           -e DRONE_RPC_HOST=drone.server.com:1234   
           -e DRONE_RPC_SECRET=<RPC_Secret>   
           -e DRONE_RUNNER_CAPACITY=2   
           -e DRONE_RUNNER_NAME=srv2
           -e DRONE_RUNNER_LABELS=server:srv2
           -e DRONE_LOGS_DEBUG=true   
           -e DRONE_RPC_DEBUG=true
           -p 3000:3000   
           -v /var/run/docker.sock:/var/run/docker.sock   
           --restart always   
           --name runner   
           drone/agent:1

The pipeline:

kind: pipeline
name: Deploy Nginx on two servers of Test

platform:
    os: linux

steps:
- name: Build nginx Image
  image: docker
  pull: if-not-exists
  commands:
    - docker build -t nginx_img_drone .
  volumes:
    - name: docker_sock
      path: /var/run/docker.sock
      host: /var/run/docker.sock

- name: Run Nginx Container
  image: docker
  commands:
    - docker ps | grep nginx_cont_drone && docker stop nginx_cont_drone
    - docker ps -a | grep nginx_cont_drone && docker rm nginx_cont_drone
    - docker run --name nginx_cont_drone -it -d -v /home/test/nginx/conf:/etc/nginx/conf.d -v /storage/:/home -v /storage/test:/var/log/nginx -v /home/test/logrotate_nginx/logrotate.d/:/etc/logrotate.d -v /home/test/logrotate_nginx/statusDir/:/var/lib/logrotate/ -v /etc/localtime:/etc/localtime --dns=8.8.8.8 --dns=77.77.77.77 --dns=185.98.113.113 nginx_img_drone
  volumes:
    - name: docker_sock
      path: /var/run/docker.sock
      host: /var/run/docker.sock

volumes:
  - name: docker_sock
    host: 
      path: /var/run/docker.sock

node:
    server: srv1
    server: srv2

Server log:

{"level":"info","msg":"main: internal scheduler enabled","time":"2019-08-08T07:15:24Z"}
{"interval":"30m0s","level":"info","msg":"starting the cron scheduler","time":"2019-08-08T07:15:24Z"}
{"acme":false,"host":"drone.server.com:1234","level":"info","msg":"starting the http server","port":":80","proto":"http","time":"2019-08-08T07:15:24Z","url":"drone.server.com:1234"}
{"build.id":221,"build.number":221,"error":"Cannot transition status via :enqueue from :pending (Reason(s): Status cannot transition via \"enqueue\")","level":"warning","msg":"manager: cannot publish status","repo.id":4,"stage.id":206,"time":"2019-08-08T07:20:10Z"}

First runner log:

{"level":"debug","msg":"successfully pinged the docker daemon","time":"2019-08-08T07:19:49Z"}
{"arch":"amd64","level":"debug","machine":"srv1","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:19:49Z"}
2019/08/08 07:19:49 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
{"arch":"amd64","level":"debug","machine":"srv1","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:19:49Z"}
2019/08/08 07:19:49 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
2019/08/08 07:19:54 [DEBUG] POST http://drone.server.com:1234/rpc/v1/accept
2019/08/08 07:19:54 [DEBUG] POST http://drone.server.com:1234/rpc/v1/accept
{"arch":"amd64","level":"debug","machine":"srv1","msg":"runner: get stage details from server","os":"linux","stage-id":206,"time":"2019-08-08T07:19:59Z"}
2019/08/08 07:19:59 [DEBUG] POST http://drone.server.com:1234/rpc/v1/details
{"arch":"amd64","level":"debug","machine":"srv1","msg":"runner: watch for cancel signal","os":"linux","time":"2019-08-08T07:19:59Z"}
2019/08/08 07:19:59 [DEBUG] POST http://drone.server.com:1234/rpc/v1/watch
{"arch":"amd64","level":"debug","machine":"srv1","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:19:59Z"}
2019/08/08 07:19:59 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
2019/08/08 07:20:04 [DEBUG] POST http://drone.server.com:1234/rpc/v1/netrc
2019/08/08 07:20:05 [DEBUG] POST http://drone.server.com:1234/rpc/v1/beforeAll
{"arch":"amd64","build":221,"level":"info","machine":"srv1","msg":"runner: start execution","os":"linux","pipeline":"Deploy Nginx on two servers of Test 30","repo":"yaghobi/docker_nginx","stage":1,"stage-id":206,"time":"2019-08-08T07:20:10Z"}
2019/08/08 07:20:10 [DEBUG] POST http://drone.server.com:1234/rpc/v1/before
2019/08/08 07:20:11 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:20:11 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:20:16 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:20:17 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:20:22 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:20:27 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:20:32 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:20:32 [DEBUG] POST http://drone.server.com:1234/rpc/v1/upload?id=444
2019/08/08 07:20:34 [DEBUG] POST http://drone.server.com:1234/rpc/v1/watch (status: 524): retrying in 1s (30 left)
2019/08/08 07:20:34 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:20:37 [DEBUG] POST http://drone.server.com:1234/rpc/v1/after
2019/08/08 07:20:38 [DEBUG] POST http://drone.server.com:1234/rpc/v1/before
2019/08/08 07:20:44 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:20:44 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:20:49 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:20:49 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:20:54 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:20:59 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:20:59 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:20:59 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
{"arch":"amd64","level":"debug","machine":"srv1","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:20:59Z"}
2019/08/08 07:20:59 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
2019/08/08 07:21:04 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:21:04 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:21:04 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:21:07 [DEBUG] POST http://drone.server.com:1234/rpc/v1/watch (status: 524): retrying in 2s (29 left)
2019/08/08 07:21:10 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:21:10 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:21:15 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:21:20 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:21:20 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:21:25 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:21:25 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:21:34 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:21:40 [DEBUG] POST http://drone.server.com:1234/rpc/v1/watch (status: 524): retrying in 4s (28 left)
2019/08/08 07:21:40 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:21:45 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:21:46 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:21:51 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:21:56 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:21:56 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:21:59 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
{"arch":"amd64","level":"debug","machine":"srv1","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:21:59Z"}
2019/08/08 07:21:59 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
2019/08/08 07:22:01 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:22:15 [DEBUG] POST http://drone.server.com:1234/rpc/v1/watch (status: 524): retrying in 8s (27 left)
2019/08/08 07:22:16 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:22:21 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:22:26 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:22:26 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:22:31 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:22:31 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:22:36 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:22:37 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:22:42 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:22:42 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:22:47 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:22:47 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:22:52 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:22:56 [DEBUG] POST http://drone.server.com:1234/rpc/v1/watch (status: 524): retrying in 10s (26 left)
2019/08/08 07:22:57 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:22:59 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
{"arch":"amd64","level":"debug","machine":"srv1","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:22:59Z"}
2019/08/08 07:22:59 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
2019/08/08 07:23:02 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:23:07 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:23:17 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:23:22 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:23:23 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:23:28 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:23:28 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:23:32 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:23:33 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:23:33 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:23:37 [DEBUG] POST http://drone.server.com:1234/rpc/v1/watch (status: 524): retrying in 10s (25 left)
2019/08/08 07:23:38 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:23:38 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:23:38 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:23:43 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:23:43 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:23:49 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:23:49 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:23:54 [DEBUG] POST http://drone.server.com:1234/rpc/v1/upload?id=445
2019/08/08 07:23:54 [DEBUG] POST http://drone.server.com:1234/rpc/v1/after
2019/08/08 07:23:59 [DEBUG] POST http://drone.server.com:1234/rpc/v1/before
2019/08/08 07:23:59 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
{"arch":"amd64","level":"debug","machine":"srv1","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:23:59Z"}
2019/08/08 07:23:59 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
2019/08/08 07:24:07 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:24:07 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:24:12 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:24:12 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:24:17 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:24:17 [DEBUG] POST http://drone.server.com:1234/rpc/v1/write
2019/08/08 07:24:19 [DEBUG] POST http://drone.server.com:1234/rpc/v1/watch (status: 524): retrying in 10s (24 left)
2019/08/08 07:24:22 [DEBUG] POST http://drone.server.com:1234/rpc/v1/upload?id=446
2019/08/08 07:24:23 [DEBUG] POST http://drone.server.com:1234/rpc/v1/after
{"arch":"amd64","build":221,"error":null,"level":"info","machine":"srv1","msg":"runner: execution complete","os":"linux","pipeline":"Deploy Nginx on two servers of Test 30","repo":"yaghobi/docker_nginx","stage":1,"stage-id":206,"time":"2019-08-08T07:24:28Z"}
2019/08/08 07:24:28 [DEBUG] POST http://drone.server.com:1234/rpc/v1/afterAll
{"arch":"amd64","level":"debug","machine":"srv1","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:24:28Z"}
2019/08/08 07:24:28 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
2019/08/08 07:24:29 [ERR] POST http://drone.server.com:1234/rpc/v1/watch request failed: Post http://drone.server.com:1234/rpc/v1/watch: context canceled
{"arch":"amd64","level":"debug","machine":"srv1","msg":"runner: done listening for cancel signals","os":"linux","time":"2019-08-08T07:24:29Z"}
2019/08/08 07:24:34 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:24:59 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
{"arch":"amd64","level":"debug","machine":"srv1","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:24:59Z"}
2019/08/08 07:24:59 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
2019/08/08 07:25:04 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:25:28 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
{"arch":"amd64","level":"debug","machine":"srv1","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:25:28Z"}
2019/08/08 07:25:28 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
2019/08/08 07:25:29 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:25:59 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:25:59 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
{"arch":"amd64","level":"debug","machine":"srv1","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:25:59Z"}
2019/08/08 07:25:59 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
2019/08/08 07:26:28 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
{"arch":"amd64","level":"debug","machine":"srv1","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:26:28Z"}

The second runner log:

{"level":"debug","msg":"successfully pinged the docker daemon","time":"2019-08-08T07:20:19Z"}
{"arch":"amd64","level":"debug","machine":"srv2","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:20:19Z"}
2019/08/08 07:20:19 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
{"arch":"amd64","level":"debug","machine":"srv2","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:20:19Z"}
2019/08/08 07:20:19 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
2019/08/08 07:20:49 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:20:49 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:21:19 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
{"arch":"amd64","level":"debug","machine":"srv2","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:21:19Z"}
2019/08/08 07:21:19 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
2019/08/08 07:21:19 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
{"arch":"amd64","level":"debug","machine":"srv2","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:21:19Z"}
2019/08/08 07:21:19 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
2019/08/08 07:21:49 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:21:49 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:22:19 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
2019/08/08 07:22:19 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
{"arch":"amd64","level":"debug","machine":"srv2","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:22:19Z"}
2019/08/08 07:22:19 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
{"arch":"amd64","level":"debug","machine":"srv2","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:22:19Z"}
2019/08/08 07:22:19 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
2019/08/08 07:22:50 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:22:50 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:23:19 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
2019/08/08 07:23:19 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
{"arch":"amd64","level":"debug","machine":"srv2","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:23:19Z"}
2019/08/08 07:23:19 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
{"arch":"amd64","level":"debug","machine":"srv2","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:23:19Z"}
2019/08/08 07:23:19 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
2019/08/08 07:23:49 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:23:49 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:24:19 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
2019/08/08 07:24:19 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
{"arch":"amd64","level":"debug","machine":"srv2","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:24:19Z"}
2019/08/08 07:24:19 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
{"arch":"amd64","level":"debug","machine":"srv2","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:24:19Z"}
2019/08/08 07:24:19 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
2019/08/08 07:24:49 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:24:49 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:25:19 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
2019/08/08 07:25:19 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
{"arch":"amd64","level":"debug","machine":"srv2","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:25:19Z"}
2019/08/08 07:25:19 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
{"arch":"amd64","level":"debug","machine":"srv2","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:25:19Z"}
2019/08/08 07:25:19 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
2019/08/08 07:25:49 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:25:49 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:26:19 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
2019/08/08 07:26:19 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
{"arch":"amd64","level":"debug","machine":"srv2","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:26:19Z"}
2019/08/08 07:26:19 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
{"arch":"amd64","level":"debug","machine":"srv2","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:26:19Z"}
2019/08/08 07:26:19 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request
2019/08/08 07:26:49 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:26:49 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request (status: 524): retrying in 1s (30 left)
2019/08/08 07:27:19 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
2019/08/08 07:27:19 [ERR] POST http://drone.server.com:1234/rpc/v1/request request failed: Post http://drone.server.com:1234/rpc/v1/request: context deadline exceeded
{"arch":"amd64","level":"debug","machine":"srv2","msg":"runner: polling queue","os":"linux","time":"2019-08-08T07:27:19Z"}
2019/08/08 07:27:19 [DEBUG] POST http://drone.server.com:1234/rpc/v1/request

moyamos · August 11, 2019, 10:02am

I have added

DRONE_LOGS_DEBUG=true

for running the server container and I am seeing following errors:

{"error":"Optimistic Lock Error","level":"debug","machine":"srv1","msg":"manager: stage processed by another agent","stage-id":209,"time":"2019-08-11T09:48:33Z"}
{"build.id":224,"build.number":224,"error":"Cannot transition status via :enqueue from :pending (Reason(s): Status cannot transition via \"enqueue\")","level":"warning","msg":"manager: cannot publish status","repo.id":4,"stage.id":209,"time":"2019-08-11T09:48:44Z"}

Is there any sign of a problem in the above-mentioned errors? Where I have to look to resolve the problem? Do I need to consider a problem of connections between the node and the server?

Any hint is appreciated and welcomed.

P.S.:

1- Currently, mentioned pipeline only deploys on a node with label “srv1”, although I have already had experience of random deploy on both nodes. It seems when a node accepts a request another node is not capable to accept/notice a push on the GitLab repository!

2- I have changed the order of labels in the “node” section of the pipeline. Now, “srv2” has started to deploy and “srv1” do not perform anything. Did I use correct syntax and format for defining node’s label during running the runners and using in the node section?

3- I notice that the deploy process on “srv2” is considerably faster than “srv1”!