Drone-docker plugin issue on kubernetes deployment

Drone Support
1

Hi everyone,

Recently we have been having issues with the drone-docker plugin in our cluster. It seemed that it cannot connect/auth to docker. We can connect/login to it from our machines and our cluster is able to use it as well -so the registry seems OK.

After trying quite a few solutions I’ve run out of ideas. Here is the blocking error that the plugin gets - it seems like a docker in docker permissions error, but I’m not sure how to fix it or if there’s a setting that I may be missing.

Drone.yml file:
pipeline:

  build-using-vanilla:
    image: plugins/docker
    debug: true
    launch_debug: true
    registry: myregistry.com
    secrets: [ docker_username, docker_password ]
    repo: myregistry/image
    tags:
      - latest
      - ${DRONE_BRANCH##feature/}
      - ${DRONE_BUILD_NUMBER}

  build-using-17-05:
    image: plugins/docker:17.05
    debug: true
    launch_debug: true
    registry: myregistry.com
    secrets: [ docker_username, docker_password ]
    repo: myregistry/image
    tags:
      - latest
      - ${DRONE_BRANCH##feature/}
      - ${DRONE_BUILD_NUMBER}

  build-using-latest:
    image: plugins/docker:latest
    debug: true
    launch_debug: true
    registry: myregistry.com
    secrets: [ docker_username, docker_password ]
    repo: myregistry/image
    tags:
      - latest
      - ${DRONE_BRANCH##feature/}
      - ${DRONE_BUILD_NUMBER}

When running plugins/docker:

+ /usr/local/bin/dockerd -g /var/lib/docker
time="2018-03-14T01:57:33Z" level=warning msg="The \"-g / --graph\" flag is deprecated. Please use \"--data-root\" instead"
time="2018-03-14T01:57:33.018227178Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
time="2018-03-14T01:57:33.019789050Z" level=info msg="libcontainerd: started new docker-containerd process" pid=20
time="2018-03-14T01:57:33Z" level=info msg="starting containerd" module=containerd revision=89623f28b87a6004d4b785663257362d1658a729 version=v1.0.0 
time="2018-03-14T01:57:33Z" level=info msg="setting subreaper..." module=containerd 
time="2018-03-14T01:57:33Z" level=info msg="changing OOM score to -500" module=containerd 
containerd: write /proc/20/oom_score_adj: permission denied
time="2018-03-14T01:57:33.031937208Z" level=error msg="containerd did not exit successfully" error="exit status 1" module=libcontainerd
Failed to connect to containerd: failed to dial "/var/run/docker/containerd/docker-containerd.sock": dial unix:///var/run/docker/containerd/docker-containerd.sock: timeout
time="2018-03-14T01:58:07Z" level=fatal msg="Error authenticating: exit status 1"

When running plugins/docker:17.05:

+ /usr/local/bin/dockerd -g /var/lib/docker
time="2018-03-14T01:58:34Z" level=warning msg="The \"-g / --graph\" flag is deprecated. Please use \"--data-root\" instead"
time="2018-03-14T01:58:34.940648011Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
time="2018-03-14T01:58:34.941710548Z" level=info msg="libcontainerd: started new docker-containerd process" pid=18
time="2018-03-14T01:58:34Z" level=info msg="starting containerd" module=containerd revision=89623f28b87a6004d4b785663257362d1658a729 version=v1.0.0 
time="2018-03-14T01:58:34Z" level=info msg="setting subreaper..." module=containerd 
time="2018-03-14T01:58:34Z" level=info msg="changing OOM score to -500" module=containerd 
containerd: write /proc/18/oom_score_adj: permission denied
time="2018-03-14T01:58:34.950754131Z" level=error msg="containerd did not exit successfully" error="exit status 1" module=libcontainerd
Failed to connect to containerd: failed to dial "/var/run/docker/containerd/docker-containerd.sock": dial unix:///var/run/docker/containerd/docker-containerd.sock: timeout
time="2018-03-14T01:59:09Z" level=fatal msg="Error authenticating: exit status 1"

When running plugins/docker:latest:

+ /usr/local/bin/dockerd -g /var/lib/docker
time="2018-03-14T01:57:33Z" level=warning msg="The \"-g / --graph\" flag is deprecated. Please use \"--data-root\" instead"
time="2018-03-14T01:57:33.040146753Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
time="2018-03-14T01:57:33.041208439Z" level=info msg="libcontainerd: started new docker-containerd process" pid=18
time="2018-03-14T01:57:33Z" level=info msg="starting containerd" module=containerd revision=89623f28b87a6004d4b785663257362d1658a729 version=v1.0.0 
time="2018-03-14T01:57:33Z" level=info msg="setting subreaper..." module=containerd 
time="2018-03-14T01:57:33Z" level=info msg="changing OOM score to -500" module=containerd 
containerd: write /proc/18/oom_score_adj: permission denied
time="2018-03-14T01:57:33.052930817Z" level=error msg="containerd did not exit successfully" error="exit status 1" module=libcontainerd
Failed to connect to containerd: failed to dial "/var/run/docker/containerd/docker-containerd.sock": dial unix:///var/run/docker/containerd/docker-containerd.sock: timeout
time="2018-03-14T01:58:07Z" level=fatal msg="Error authenticating: exit status 1"

Drone Agent ConfigMap yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: drone-agent-config
  namespace: ops
data:
  agent.debug: "true"
  agent.debug.pretty: "true"
  agent.max.procs: "1"
  agent.healthcheck: "true"

Drone Agent Deployment yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    name: drone-agent
  name: drone-agent
  namespace: ops
spec:
  replicas: 1
  selector:
    matchLabels:
      name: drone-agent
  template:
    metadata:
      labels:
        name: drone-agent
    spec:
      containers:
      - env:
        # tried 1.27 which I thought was right
        # tried removing this and it did not work
        - name: DOCKER_API_VERSION
          value: "1.24"
        - name: DRONE_PLUGIN_PULL
          value: "true"
        - name: DRONE_SERVER
          value: drone-server:9000
        - name: DRONE_SECRET
          valueFrom:
            secretKeyRef:
              key: server.secret
              name: drone-secrets
        - name: DRONE_MAX_PROCS
          valueFrom:
            configMapKeyRef:
              name: drone-agent-config
              key: agent.max.procs
        - name: DRONE_DEBUG
          valueFrom:
            configMapKeyRef:
              name: drone-agent-config
              key: agent.debug
        - name: DRONE_DEBUG_PRETTY
          valueFrom:
            configMapKeyRef:
              name: drone-agent-config
              key: agent.debug.pretty
        - name: DRONE_HEALTHCHECK
          valueFrom:
            configMapKeyRef:
              name: drone-agent-config
              key: agent.healthcheck
        # same output with 0.8.4                                     
        image: drone/agent:latest
        imagePullPolicy: Always
        name: drone-agent
        resources:
          requests:
            cpu: 100m
            memory: 64Mi
        livenessProbe:
          httpGet:
            path: /healthz
            port: 3000
          initialDelaySeconds: 3
          periodSeconds: 3
        volumeMounts:
        - mountPath: /var/run/docker.sock
          name: docker-socket
        securityContext:
          privileged: true  
          allowPrivilegeEscalation: true
      dnsPolicy: ClusterFirst   
      volumes:
      - hostPath:
          path: /var/run/docker.sock
        name: docker-socket

Kubernetes Docker Version:

Client:
 Version:      17.03.2-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 02:09:56 2017
 OS/Arch:      linux/amd64
Server:
 Version:      17.03.2-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 02:09:56 2017
 OS/Arch:      linux/amd64
 Experimental: false

And if it helps:

 # uname -a
 Linux ip-10-0-144-238 4.4.102-k8s #1 SMP Sun Nov 26 23:32:43 UTC 2017 x86_64 GNU/Linux

Kubernetes Version:

v1.8.4

Kubenetes AMI:

image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2017-12-02

Any help is greatly appreciated.

Thanks in advance,
Elliot

2

Have you tried running the official docker:dind image on your kubernetes cluster (in a pod, not in drone)? The docker plugin uses the docker:dind as its base. So if you can get the docker:dind image working, you should be able to get the plugins/docker image working. And conversely if you cannot get docker:dind to run, we known the plugins/docker image will fail.

If you cannot get the docker:dind image working, you should be able to engage kubernetes or docker support, who will have a more specialized understanding of the problem, and should be better suited to help triage the problem.

3

Thanks for the reply! This was going to be the next thing I tried. I just did it and here’s what’s going on.

Here is the configuration for the dind-test deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: dind-test
  name: dind-test
  namespace: ops
spec:
  replicas: 1
  selector:
    matchLabels:
      run: dind-test
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        run: dind-test
    spec:
      containers:
      - env:
        - name: DOCKER_DRIVER
          value: overlay
        image: docker:17.05.0-ce-dind
        imagePullPolicy: IfNotPresent
        name: dind-test
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /var/lib/docker
          name: docker-graph-storage
      restartPolicy: Always
      volumes:
      - emptyDir: {}
        name: docker-graph-storage

The vanilla dind container starts and outputs this:

time="2018-03-14T13:38:15.084578905Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
time="2018-03-14T13:38:15.084752559Z" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
time="2018-03-14T13:38:15.085553221Z" level=info msg="libcontainerd: new containerd process, pid: 18"
time="2018-03-14T13:38:16.118486048Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
time="2018-03-14T13:38:16.118781692Z" level=warning msg="Your kernel does not support swap memory limit"
time="2018-03-14T13:38:16.118840782Z" level=warning msg="Your kernel does not support kernel memory limit"
time="2018-03-14T13:38:16.118884424Z" level=warning msg="Your kernel does not support cgroup rt period"
time="2018-03-14T13:38:16.118900028Z" level=warning msg="Your kernel does not support cgroup rt runtime"
time="2018-03-14T13:38:16.119258508Z" level=info msg="Loading containers: start."
time="2018-03-14T13:38:16.120173684Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: modprobe: can't change directory to '/lib/modules': No such file or directory\n, error: exit status 1"
time="2018-03-14T13:38:16.120820111Z" level=warning msg="Running modprobe nf_nat failed with message: `modprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1"
time="2018-03-14T13:38:16.121518334Z" level=warning msg="Running modprobe xt_conntrack failed with message: `modprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1"
time="2018-03-14T13:38:16.164093548Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
time="2018-03-14T13:38:16.187860580Z" level=info msg="Loading containers: done."
time="2018-03-14T13:38:16.193596284Z" level=info msg="Daemon has completed initialization"
time="2018-03-14T13:38:16.193628432Z" level=info msg="Docker daemon" commit=89658be graphdriver=overlay version=17.05.0-ce
time="2018-03-14T13:38:16.207879240Z" level=info msg="API listen on [::]:2375"
time="2018-03-14T13:38:16.207945618Z" level=info msg="API listen on /var/run/docker.sock"

I changed my drone-agent deployment to look like this considering the modules error above:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    name: drone-agent
  name: drone-agent
  namespace: ops
spec:
  replicas: 1
  selector:
    matchLabels:
      name: drone-agent
  template:
    metadata:
      labels:
        name: drone-agent
    spec:
      containers:
      - name: drone-agent
        env:
        - name: DRONE_PLUGIN_PULL
          value: "true"
        - name: DRONE_SERVER
          value: drone-server:9000
        - name: DRONE_SECRET
          valueFrom:
            secretKeyRef:
              key: server.secret
              name: drone-secrets
        - name: DRONE_MAX_PROCS
          valueFrom:
            configMapKeyRef:
              name: drone-agent-config
              key: agent.max.procs
        - name: DRONE_DEBUG
          valueFrom:
            configMapKeyRef:
              name: drone-agent-config
              key: agent.debug
        - name: DRONE_DEBUG_PRETTY
          valueFrom:
            configMapKeyRef:
              name: drone-agent-config
              key: agent.debug.pretty
        - name: DRONE_HEALTHCHECK
          valueFrom:
            configMapKeyRef:
              name: drone-agent-config
              key: agent.healthcheck 
        - name: DOCKER_HOST
          value: tcp://localhost:2375                                    
        image: drone/agent:latest
        imagePullPolicy: Always
        resources:
          requests:
            cpu: 100m
            memory: 64Mi
        livenessProbe:
          httpGet:
            path: /healthz
            port: 3000
          initialDelaySeconds: 3
          periodSeconds: 3
      - env:
        - name: DOCKER_DRIVER
          value: overlay
        image: docker:17.05.0-ce-dind
        imagePullPolicy: IfNotPresent
        name: drone-agent-dind
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /var/lib/docker
          name: docker-graph-storage
        - mountPath: /lib/modules
          name: docker-modules
      restartPolicy: Always
      volumes:
      - emptyDir: {}
        name: docker-graph-storage
      - hostPath: 
          path: /lib/modules
        name: docker-modules
      tolerations:
      - effect: NoSchedule
        key: purpose
        value: ops
      - effect: NoSchedule
        key: dedicated
        value: ops
      - effect: NoExecute
        key: dedicated
        value: ops

Now the output from the dind container running in drone (trying to do a docker build) looks like this:

time="2018-03-14T13:48:33.404582870Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
time="2018-03-14T13:48:33.404910527Z" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
time="2018-03-14T13:48:33.406357192Z" level=info msg="libcontainerd: new containerd process, pid: 20"
time="2018-03-14T13:48:34.462850635Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
time="2018-03-14T13:48:34.463248492Z" level=warning msg="Your kernel does not support swap memory limit"
time="2018-03-14T13:48:34.463317054Z" level=warning msg="Your kernel does not support kernel memory limit"
time="2018-03-14T13:48:34.463395231Z" level=warning msg="Your kernel does not support cgroup rt period"
time="2018-03-14T13:48:34.463424283Z" level=warning msg="Your kernel does not support cgroup rt runtime"
time="2018-03-14T13:48:34.463946265Z" level=info msg="Loading containers: start."
time="2018-03-14T13:48:34.540633164Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
time="2018-03-14T13:48:34.572601158Z" level=info msg="Loading containers: done."
time="2018-03-14T13:48:34.579829273Z" level=info msg="Daemon has completed initialization"
time="2018-03-14T13:48:34.579930090Z" level=info msg="Docker daemon" commit=89658be graphdriver=overlay version=17.05.0-ce
time="2018-03-14T13:48:34.593240184Z" level=info msg="API listen on [::]:2375"
time="2018-03-14T13:48:34.593389869Z" level=info msg="API listen on /var/run/docker.sock"
time="2018-03-14T13:48:50.597705498Z" level=error msg="Error closing logger: invalid argument"
time="2018-03-14T13:49:30.905165685Z" level=error msg="Error closing logger: invalid argument"
time="2018-03-14T13:49:31.236525920Z" level=error msg="Handler for POST /v1.26/containers/0_4976107370189178316_clone/kill returned error: Cannot kill container 0_4976107370189178316_clone: Container a902438347503376f490ddf34e302677edd5c7908d43d41bf019fc92c4cc7005 is not running"
time="2018-03-14T13:49:31.249567059Z" level=error msg="Handler for POST /v1.26/containers/0_4976107370189178316_step_0/kill returned error: Cannot kill container 0_4976107370189178316_step_0: Container 7217dd66b9145a3158846d3d09b8ff4f64aa35b26fa71c3a7aaf5b9027179d1c is not running"

I’m wondering now if this is because my docker version and the dind version do not line up. I used to have the following flag set, and am wondering if maybe there is a new value I should be using.

    - name: DOCKER_API_VERSION
      value: "1.24"

Thanks again for your help!

4

Just as an update. I tried setting the api version to 1.26, but that didn’t help. I put the dind container into debug mode and this is the output (timestamps omitted to fit in character limit):

debug msg="Registering DELETE, /nodes/{id}"
debug msg="Registering POST, /nodes/{id}/update"
debug msg="Registering GET, /tasks"
debug msg="Registering GET, /tasks/{id}"
debug msg="Registering GET, /secrets"
debug msg="Registering POST, /secrets/create"
debug msg="Registering DELETE, /secrets/{id}"
debug msg="Registering GET, /secrets/{id}"
debug msg="Registering POST, /secrets/{id}/update"
debug msg="Registering GET, /plugins"
debug msg="Registering GET, /plugins/{name:.*}/json"
debug msg="Registering GET, /plugins/privileges"
debug msg="Registering DELETE, /plugins/{name:.*}"
debug msg="Registering POST, /plugins/{name:.*}/enable"
debug msg="Registering POST, /plugins/{name:.*}/disable"
debug msg="Registering POST, /plugins/pull"
debug msg="Registering POST, /plugins/{name:.*}/push"
debug msg="Registering POST, /plugins/{name:.*}/upgrade"
debug msg="Registering POST, /plugins/{name:.*}/set"
debug msg="Registering POST, /plugins/create"
debug msg="Registering GET, /networks"
debug msg="Registering GET, /networks/"
debug msg="Registering GET, /networks/{id:.+}"
debug msg="Registering POST, /networks/create"
debug msg="Registering POST, /networks/{id:.*}/connect"
debug msg="Registering POST, /networks/{id:.*}/disconnect"
debug msg="Registering POST, /networks/prune"
debug msg="Registering DELETE, /networks/{id:.*}"
info msg="API listen on [::]:2375"
info msg="API listen on /var/run/docker.sock"
debug msg="Calling POST /v1.26/volumes/create"
debug msg="form data: {\"Driver\":\"local\",\"DriverOpts\":null,\"Labels\":null,\"Name\":\"0_6424036164612164775_default\"}"
debug msg="Registering new volume reference: driver \"local\", name \"0_6424036164612164775_default\""
debug msg="Calling POST /v1.26/networks/create"
debug msg="form data: {\"Attachable\":false,\"CheckDuplicate\":false,\"Driver\":\"bridge\",\"EnableIPv6\":false,\"IPAM\":null,\"Internal\":false,\"Labels\":null,\"Name\":\"0_6424036164612164775_default\",\"Options\":null}"
debug msg="Allocating IPv4 pools for network 0_6424036164612164775_default (c7765c6c80c1c64b7d3fadba9e17ce661c1863119f2e4aad7d4929456b11ea06)"
debug msg="RequestPool(LocalDefault, , , map[], false)"
debug msg="RequestAddress(LocalDefault/172.18.0.0/16, <nil>, map[RequestAddressType:com.docker.network.gateway])"
debug msg="Did not find any interface with name br-c7765c6c80c1: Link not found"
debug msg="Setting bridge mac address to 02:42:ac:de:21:9d"
debug msg="Assigning address to bridge interface br-c7765c6c80c1: 172.18.0.1/16"
debug msg="/sbin/iptables, [--wait -t nat -C POSTROUTING -s 172.18.0.0/16 ! -o br-c7765c6c80c1 -j MASQUERADE]"
debug msg="/sbin/iptables, [--wait -t nat -I POSTROUTING -s 172.18.0.0/16 ! -o br-c7765c6c80c1 -j MASQUERADE]"
debug msg="/sbin/iptables, [--wait -t nat -C DOCKER -i br-c7765c6c80c1 -j RETURN]"
debug msg="/sbin/iptables, [--wait -t nat -I DOCKER -i br-c7765c6c80c1 -j RETURN]"
debug msg="/sbin/iptables, [--wait -D FORWARD -i br-c7765c6c80c1 -o br-c7765c6c80c1 -j DROP]"
debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -i br-c7765c6c80c1 -o br-c7765c6c80c1 -j ACCEPT]"
debug msg="/sbin/iptables, [--wait -I FORWARD -i br-c7765c6c80c1 -o br-c7765c6c80c1 -j ACCEPT]"
debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -i br-c7765c6c80c1 ! -o br-c7765c6c80c1 -j ACCEPT]"
debug msg="/sbin/iptables, [--wait -I FORWARD -i br-c7765c6c80c1 ! -o br-c7765c6c80c1 -j ACCEPT]"
debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -o br-c7765c6c80c1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT]"
debug msg="/sbin/iptables, [--wait -I FORWARD -o br-c7765c6c80c1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT]"
debug msg="/sbin/iptables, [--wait -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]"
debug msg="/sbin/iptables, [--wait -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER]"
debug msg="/sbin/iptables, [--wait -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8]"
debug msg="/sbin/iptables, [--wait -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8]"
debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -o br-c7765c6c80c1 -j DOCKER]"
debug msg="/sbin/iptables, [--wait -I FORWARD -o br-c7765c6c80c1 -j DOCKER]"
debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -j DOCKER-ISOLATION]"
debug msg="/sbin/iptables, [--wait -D FORWARD -j DOCKER-ISOLATION]"
debug msg="/sbin/iptables, [--wait -I FORWARD -j DOCKER-ISOLATION]"
debug msg="/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION -i br-c7765c6c80c1 -o docker0 -j DROP]"
debug msg="/sbin/iptables, [--wait -I DOCKER-ISOLATION -i br-c7765c6c80c1 -o docker0 -j DROP]"
debug msg="/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION -i docker0 -o br-c7765c6c80c1 -j DROP]"
debug msg="/sbin/iptables, [--wait -I DOCKER-ISOLATION -i docker0 -o br-c7765c6c80c1 -j DROP]"
debug msg="Calling POST /v1.26/containers/create?name=0_6424036164612164775_clone"
error msg="Handler for POST /v1.26/containers/create returned error: No such image: plugins/git:latest"
debug msg="Calling POST /v1.26/images/create?fromImage=plugins%2Fgit&tag=latest"
debug msg="Trying to pull plugins/git from https://registry-1.docker.io v2"
debug msg="Pulling ref from V2 registry: plugins/git:latest"
debug msg="pulling blob \"sha256:c0fd9486e124b8a3154a58a8720e6d47ba289e22fa4c7a2881cfd9b67fe32d34\""
debug msg="pulling blob \"sha256:b56ae66c29370df48e7377c8f9baa744a3958058a766793f821dadcb144a4647\""
debug msg="pulling blob \"sha256:a76803713564fbfc2a1a1f0e9a68db94550def032bd772ef53924aa065db446c\""
debug msg="Downloaded b56ae66c2937 to tempfile /var/lib/docker/tmp/GetImageBlob122745934"
debug msg="Downloaded c0fd9486e124 to tempfile /var/lib/docker/tmp/GetImageBlob040528875"
debug msg="pulling blob \"sha256:aa982143412bf5dce988358db194fe9cc05dc4455f33812765620edcde7d8b1b\""
debug msg="Start untar layer"
debug msg="Downloaded a76803713564 to tempfile /var/lib/docker/tmp/GetImageBlob008992085"
debug msg="Downloaded aa982143412b to tempfile /var/lib/docker/tmp/GetImageBlob603162288"
debug msg="Untar time: 0.43927891900000005s"
debug msg="Applied tar sha256:2aebd096e0e237b447781353379722157e6c2d434b9ec5a0d63f2a6f07cf90c2 to ccbf8b2577933d9106aa8ec2d87c01aaa0f4a2e7fc00c56b3f534ee4acb211cc, size: 3961589"
debug msg="Applied tar sha256:0edd9b27914eb7937d5e077aa0b3f47ebad461476e5a6fefc93822dfbb17602b to 01133c64121f0b43c4d32c6078e1df66620429ac682147dbbac5a88fd71a7272, size: 674251"
debug msg="Applied tar sha256:aa6425b484cd4de9b50de05bb98ef14f50b6d61d1793cc02bd19601c5f797d9e to 9c428f2abf3fbc0e85c438e840ae2be1ee42f3a7eac523397524776cb551ae2b, size: 58261315"
debug msg="Applied tar sha256:551c4ad2b9851f6ea4121d13a5996620429895380f165123f56bf30a1d38f30e to 026f295bdfb3b192694989e2228c6f9a2f07b20cc9337dda9199d9adc822bffd, size: 3916554"
debug msg="Calling POST /v1.26/containers/create?name=0_6424036164612164775_clone"
debug msg="container mounted via layerStore: /var/lib/docker/overlay/e8b35d4fa08795237ee054bef4643f69962d6ccd3bb64416e998a676679be916/merged"
debug msg="copying image data from 0c519f446c19bf5a05dfabd59fef6d328e1e380b3e694923e868e12038964523:/drone, to 0_6424036164612164775_default"
debug msg="Creating dest directory: /var/lib/docker/volumes/0_6424036164612164775_default/_data"
debug msg="Calling TarUntar(/var/lib/docker/overlay/e8b35d4fa08795237ee054bef4643f69962d6ccd3bb64416e998a676679be916/merged/drone, /var/lib/docker/volumes/0_6424036164612164775_default/_data)"
debug msg="TarUntar(/var/lib/docker/overlay/e8b35d4fa08795237ee054bef4643f69962d6ccd3bb64416e998a676679be916/merged/drone /var/lib/docker/volumes/0_6424036164612164775_default/_data)"
debug msg="Calling POST /v1.26/networks/0_6424036164612164775_default/connect"
debug msg="form data: {\"Container\":\"0_6424036164612164775_clone\",\"EndpointConfig\":{\"Aliases\":[\"clone\"],\"EndpointID\":\"\",\"Gateway\":\"\",\"GlobalIPv6Address\":\"\",\"GlobalIPv6PrefixLen\":0,\"IPAMConfig\":null,\"IPAddress\":\"\",\"IPPrefixLen\":0,\"IPv6Gateway\":\"\",\"Links\":null,\"MacAddress\":\"\",\"NetworkID\":\"\"}}"
debug msg="Calling POST /v1.26/containers/0_6424036164612164775_clone/start"
debug msg="container mounted via layerStore: /var/lib/docker/overlay/e8b35d4fa08795237ee054bef4643f69962d6ccd3bb64416e998a676679be916/merged"
debug msg="Assigning addresses for endpoint 0_6424036164612164775_clone's interface on network bridge"
debug msg="RequestAddress(LocalDefault/172.17.0.0/16, <nil>, map[])"
debug msg="Assigning addresses for endpoint 0_6424036164612164775_clone's interface on network bridge"
debug msg="Programming external connectivity on endpoint 0_6424036164612164775_clone (330066be329a4ca08d47d5c3dac0df9647de8daadd4d37020aef0f4806207482)"
debug msg="Assigning addresses for endpoint 0_6424036164612164775_clone's interface on network 0_6424036164612164775_default"
debug msg="RequestAddress(LocalDefault/172.18.0.0/16, <nil>, map[])"
debug msg="Assigning addresses for endpoint 0_6424036164612164775_clone's interface on network 0_6424036164612164775_default"
debug msg="Revoking external connectivity on endpoint 0_6424036164612164775_clone (330066be329a4ca08d47d5c3dac0df9647de8daadd4d37020aef0f4806207482)"
debug msg="Programming external connectivity on endpoint 0_6424036164612164775_clone (ffaa6260fdee0d5b0ed353dded67bd5e80e721ca36565dc42efc34f610a56ceb)"
debug msg="sandbox set key processing took 117.891688ms for container 0c519f446c19bf5a05dfabd59fef6d328e1e380b3e694923e868e12038964523"
debug msg="libcontainerd: received containerd event: &types.Event{Type:\"start-container\", Id:\"0c519f446c19bf5a05dfabd59fef6d328e1e380b3e694923e868e12038964523\", Status:0x0, Pid:\"\", Timestamp:(*timestamp.Timestamp)(0xc420e8fd70)}"
debug msg="libcontainerd: event unhandled: type:\"start-container\" id:\"0c519f446c19bf5a05dfabd59fef6d328e1e380b3e694923e868e12038964523\" timestamp:<seconds:1521036682 nanos:818400407 > "
debug msg="Calling GET /v1.26/containers/0_6424036164612164775_clone/logs?follow=1&stderr=1&stdout=1&tail="
debug msg="logs: begin stream"
debug msg="Name To resolve: bitbucket.org.ops.svc.cluster.local."
debug msg="Query bitbucket.org.ops.svc.cluster.local.[1] from 172.18.0.2:53396, forwarding to udp:100.64.0.10"
debug msg="Name To resolve: bitbucket.org.ops.svc.cluster.local."
debug msg="Query bitbucket.org.ops.svc.cluster.local.[28] from 172.18.0.2:56855, forwarding to udp:100.64.0.10"
debug msg="Name To resolve: bitbucket.org.svc.cluster.local."
debug msg="Query bitbucket.org.svc.cluster.local.[1] from 172.18.0.2:36953, forwarding to udp:100.64.0.10"
debug msg="Name To resolve: bitbucket.org.svc.cluster.local."
debug msg="Query bitbucket.org.svc.cluster.local.[28] from 172.18.0.2:35262, forwarding to udp:100.64.0.10"
debug msg="Name To resolve: bitbucket.org.cluster.local."
debug msg="Calling POST /v1.26/containers/0_6424036164612164775_clone/wait"
debug msg="Name To resolve: bitbucket.org.cluster.local."
debug msg="Query bitbucket.org.cluster.local.[1] from 172.18.0.2:38791, forwarding to udp:100.64.0.10"
debug msg="Query bitbucket.org.cluster.local.[28] from 172.18.0.2:37017, forwarding to udp:100.64.0.10"
debug msg="Name To resolve: bitbucket.org.ec2.internal."
debug msg="Name To resolve: bitbucket.org.ec2.internal."
debug msg="Query bitbucket.org.ec2.internal.[1] from 172.18.0.2:49167, forwarding to udp:100.64.0.10"
debug msg="Query bitbucket.org.ec2.internal.[28] from 172.18.0.2:51853, forwarding to udp:100.64.0.10"
debug msg="Name To resolve: bitbucket.org."
debug msg="Query bitbucket.org.[28] from 172.18.0.2:40799, forwarding to udp:100.64.0.10"
debug msg="Name To resolve: bitbucket.org."
debug msg="Query bitbucket.org.[1] from 172.18.0.2:54235, forwarding to udp:100.64.0.10"
debug msg="containerd: process exited" id=0c519f446c19bf5a05dfabd59fef6d328e1e380b3e694923e868e12038964523 pid=init status=0 systemPid=154
debug msg="libcontainerd: received containerd event: &types.Event{Type:\"exit\", Id:\"0c519f446c19bf5a05dfabd59fef6d328e1e380b3e694923e868e12038964523\", Status:0x0, Pid:\"init\", Timestamp:(*timestamp.Timestamp)(0xc420e752d0)}"
debug msg="Revoking external connectivity on endpoint 0_6424036164612164775_clone (ffaa6260fdee0d5b0ed353dded67bd5e80e721ca36565dc42efc34f610a56ceb)"
debug msg="logs: end stream"
error msg="Error closing logger: invalid argument"
debug msg="Programming external connectivity on endpoint 0_6424036164612164775_clone (330066be329a4ca08d47d5c3dac0df9647de8daadd4d37020aef0f4806207482)"
debug msg="Releasing addresses for endpoint 0_6424036164612164775_clone's interface on network 0_6424036164612164775_default"
debug msg="ReleaseAddress(LocalDefault/172.18.0.0/16, 172.18.0.2)"
debug msg="Revoking external connectivity on endpoint 0_6424036164612164775_clone (330066be329a4ca08d47d5c3dac0df9647de8daadd4d37020aef0f4806207482)"
debug msg="Releasing addresses for endpoint 0_6424036164612164775_clone's interface on network bridge"
debug msg="ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.2)"
debug msg="Calling GET /v1.26/containers/0_6424036164612164775_clone/json"
debug msg="Calling POST /v1.26/containers/create?name=0_6424036164612164775_step_0"
error msg="Handler for POST /v1.26/containers/create returned error: No such image: plugins/docker:latest"
debug msg="Calling POST /v1.26/images/create?fromImage=plugins%2Fdocker&tag=latest"
debug msg="Trying to pull plugins/docker from https://registry-1.docker.io v2"
debug msg="Pulling ref from V2 registry: plugins/docker:latest"
debug msg="pulling blob \"sha256:ff3a5c916c92643ff77519ffa742d3ec61b7f591b6b7504599d95a4a41134e28\""
debug msg="pulling blob \"sha256:1a649ea86bcaa0acdca25d22520d9d7b6d6373c3e4a385a21b48c2757e8ec6ac\""
debug msg="pulling blob \"sha256:ce35f4d5f86ae68ae9e5cb6590ecdcca2ae5257cbedb85fe4bfd2efa05f73b2f\""
debug msg="Downloaded ce35f4d5f86a to tempfile /var/lib/docker/tmp/GetImageBlob082274425"
debug msg="pulling blob \"sha256:b6206661264b13cd55781f8d72e87dac4a18fa6a712b5bae924e27687214546f\""
debug msg="Downloaded 1a649ea86bca to tempfile /var/lib/docker/tmp/GetImageBlob530078978"
debug msg="pulling blob \"sha256:b8b71dba24d36787b78c71e16eeec00238f613506c782e701cbb93de54a1194a\""
debug msg="Downloaded b8b71dba24d3 to tempfile /var/lib/docker/tmp/GetImageBlob131461907"
debug msg="pulling blob \"sha256:3873004a68eeb9534774a8fef1a9300aabebc2843f5b7d4e9c067eda4ab6ba26\""
debug msg="Downloaded 3873004a68ee to tempfile /var/lib/docker/tmp/GetImageBlob859391318"
debug msg="pulling blob \"sha256:f4205b132661966c453fc62160d4e952bded5cb788db27bdf7cf81545c81a725\""
debug msg="Downloaded ff3a5c916c92 to tempfile /var/lib/docker/tmp/GetImageBlob672716911"
debug msg="pulling blob \"sha256:91a85eeeb2572e6d9a9b430028a0bd0a9ca474168a2a2f85544ef60d04fe1a7f\""
debug msg="Start untar layer"
debug msg="Downloaded b6206661264b to tempfile /var/lib/docker/tmp/GetImageBlob192978308"
debug msg="pulling blob \"sha256:93c96d032b32e2f8db15b2fa0729011c8b57ff6287cd7b7cdbd3e347ec901a88\""
debug msg="Downloaded 91a85eeeb257 to tempfile /var/lib/docker/tmp/GetImageBlob254797048"
debug msg="pulling blob \"sha256:bb761748d6e12eae9527ee97723b46285b5eb13cf387ce30a33c50f3f7484c0e\""
debug msg="Downloaded f4205b132661 to tempfile /var/lib/docker/tmp/GetImageBlob043394237"
debug msg="pulling blob \"sha256:0a22375fd12c7f03bd3bd686b07138a95e53600b7940bdfcf54da09ffb582f5a\""
debug msg="Downloaded 93c96d032b32 to tempfile /var/lib/docker/tmp/GetImageBlob799478506"
debug msg="Downloaded bb761748d6e1 to tempfile /var/lib/docker/tmp/GetImageBlob515645761"
debug msg="Downloaded 0a22375fd12c to tempfile /var/lib/docker/tmp/GetImageBlob844300716"
debug msg="Untar time: 0.594156974s"
debug msg="Applied tar sha256:cd7100a72410606589a54b932cabd804a17f9ae5b42a1882bd56d263e02b6215 to 68b9efc4d8c8ddfdd9f776803b855deea486eee097d12fa1af3cfe2285b38f1a, size: 4143449"
debug msg="Applied tar sha256:1ed7a5e2d1b39c343c5d78eb53a60949f3ddace917df1b56e9ee1ffc39a33fc1 to c721093a104e7628b6535b7f13a4aab678eaaba307351a286433263d398066b1, size: 538410"
debug msg="Applied tar sha256:b09386d6aa0f4107347e107e6187bde67ecf74b9f11d8692365bc2b0c8b1e0f6 to 2b04a88ce259b2bf5f3b812a6c4fac01171a019b6295dad1f6938d144238cf97, size: 17"
debug msg="Applied tar sha256:08d5dabfb40f1195dcfd8775ec4ae4f1b1f8aae52719911fdf10c970f792dd6b to e33946122cdda848e72760b1a6c960e4aa8b6f4b1d821c276277cd94ed9e1fd6, size: 109693792"
debug msg="Applied tar sha256:c9b8ec8c0bb98170db07337d1f6efd55565c0521108f641ba29706d9a93396a1 to 90761e11f8adb6017f03b2fd38b2e961428b3d7eb2ed4719b32d534eadc40d6e, size: 587"
debug msg="Applied tar sha256:8a79ac99b0a49d16a60601e56b9408a845a6a8a908b6382f2aa07ce5d16b225b to e919f42972462cbfeb30583977b65d54299dc1890d5fc7bf56c015729b205f1f, size: 933"
debug msg="Applied tar sha256:030b1b37b5451e2592fa975f52802801225f1ee2037a0b61fbe09f602ad54006 to 1b5ce3ceb5e0414c59919d9aa2676a0c97ffe7a52e7ded5fa9614815ad59ea8b, size: 11115324"
debug msg="Applied tar sha256:a6be646b27f09444452fef13688df8be0d4f87b9e14eb7b9adc6b37e2d68d1d1 to d20483a394ebefb8df5789be5f4281d9ec9f1f28bf0774a8acc4995708fdca04, size: 4903"
debug msg="Applied tar sha256:97b49b805bca0a4aae9b6f150275945b92b7c3ac455ddcb096b4db9e2eec4507 to 678e155db20d9dca83ef6e7128e1a9d22fe304d636e9518a010d96d1d3a64edf, size: 68190"
debug msg="Applied tar sha256:93ec9936d7f06540b168ab97181089b6af241149e231f98ebee47c3b4dd8958b to 52ccdbe06177d7e3b36c7087b98ec422b8cff2641f7083337cde0a0a5375018f, size: 457"
debug msg="Applied tar sha256:26a53ce03513736d42a963f1180e9709b005e0477505ed0828bf39803905c28b to 6fdf5c60ae21c41465dfbc5ea621b97174ff4ca71c4559d07fb2e4ccf7c7a590, size: 4130667"
debug msg="Calling POST /v1.26/containers/create?name=0_6424036164612164775_step_0"
debug msg="container mounted via layerStore: /var/lib/docker/overlay/421b72f1cbd14cf97ee783da6d9f960e6efe940b543b4bfeb8857598f803d853/merged"
debug msg="Probing all drivers for volume with name: 327af57240114b2e75c5071ebaaf2b48f9a2ce7bcaefeb674fd77ec54af67a07"
debug msg="Registering new volume reference: driver \"local\", name \"327af57240114b2e75c5071ebaaf2b48f9a2ce7bcaefeb674fd77ec54af67a07\""
debug msg="copying image data from 77b83112db408c676a45b953f1a1927cd18314a5745ebf487309f8589793965d:/drone, to 0_6424036164612164775_default"
debug msg="copying image data from 77b83112db408c676a45b953f1a1927cd18314a5745ebf487309f8589793965d:/var/lib/docker, to 327af57240114b2e75c5071ebaaf2b48f9a2ce7bcaefeb674fd77ec54af67a07"
debug msg="Calling POST /v1.26/networks/0_6424036164612164775_default/connect"
debug msg="form data: {\"Container\":\"0_6424036164612164775_step_0\",\"EndpointConfig\":{\"Aliases\":[\"build-vanilla-image\"],\"EndpointID\":\"\",\"Gateway\":\"\",\"GlobalIPv6Address\":\"\",\"GlobalIPv6PrefixLen\":0,\"IPAMConfig\":null,\"IPAddress\":\"\",\"IPPrefixLen\":0,\"IPv6Gateway\":\"\",\"Links\":null,\"MacAddress\":\"\",\"NetworkID\":\"\"}}"
debug msg="Calling POST /v1.26/containers/0_6424036164612164775_step_0/start"
debug msg="container mounted via layerStore: /var/lib/docker/overlay/421b72f1cbd14cf97ee783da6d9f960e6efe940b543b4bfeb8857598f803d853/merged"
debug msg="Assigning addresses for endpoint 0_6424036164612164775_step_0's interface on network bridge"
debug msg="RequestAddress(LocalDefault/172.17.0.0/16, <nil>, map[])"
debug msg="Assigning addresses for endpoint 0_6424036164612164775_step_0's interface on network bridge"
debug msg="Programming external connectivity on endpoint 0_6424036164612164775_step_0 (916882334f1f4443bd4aa1cbb0bc99e8cd4f4165afa797961039b85d59dd4958)"
debug msg="Assigning addresses for endpoint 0_6424036164612164775_step_0's interface on network 0_6424036164612164775_default"
debug msg="RequestAddress(LocalDefault/172.18.0.0/16, <nil>, map[])"
debug msg="Assigning addresses for endpoint 0_6424036164612164775_step_0's interface on network 0_6424036164612164775_default"
debug msg="Revoking external connectivity on endpoint 0_6424036164612164775_step_0 (916882334f1f4443bd4aa1cbb0bc99e8cd4f4165afa797961039b85d59dd4958)"
debug msg="Programming external connectivity on endpoint 0_6424036164612164775_step_0 (35b3c644dc1bfc9b6ee4a7cb3ca4c705553efb10a237fdc427cb1094479a7dcd)"
debug msg="sandbox set key processing took 121.162972ms for container 77b83112db408c676a45b953f1a1927cd18314a5745ebf487309f8589793965d"
debug msg="libcontainerd: received containerd event: &types.Event{Type:\"start-container\", Id:\"77b83112db408c676a45b953f1a1927cd18314a5745ebf487309f8589793965d\", Status:0x0, Pid:\"\", Timestamp:(*timestamp.Timestamp)(0xc42116b7e0)}"
debug msg="libcontainerd: event unhandled: type:\"start-container\" id:\"77b83112db408c676a45b953f1a1927cd18314a5745ebf487309f8589793965d\" timestamp:<seconds:1521036690 nanos:477606706 > "
debug msg="Calling GET /v1.26/containers/0_6424036164612164775_step_0/logs?follow=1&stderr=1&stdout=1&tail="
debug msg="logs: begin stream"
debug msg="Calling POST /v1.26/containers/0_6424036164612164775_step_0/wait"
debug msg="containerd: process exited" id=77b83112db408c676a45b953f1a1927cd18314a5745ebf487309f8589793965d pid=init status=1 systemPid=338
debug msg="libcontainerd: received containerd event: &types.Event{Type:\"exit\", Id:\"77b83112db408c676a45b953f1a1927cd18314a5745ebf487309f8589793965d\", Status:0x1, Pid:\"init\", Timestamp:(*timestamp.Timestamp)(0xc4207a9c10)}"
debug msg="Revoking external connectivity on endpoint 0_6424036164612164775_step_0 (35b3c644dc1bfc9b6ee4a7cb3ca4c705553efb10a237fdc427cb1094479a7dcd)"
debug msg="logs: end stream"
error msg="Error closing logger: invalid argument"
debug msg="Programming external connectivity on endpoint 0_6424036164612164775_step_0 (916882334f1f4443bd4aa1cbb0bc99e8cd4f4165afa797961039b85d59dd4958)"
debug msg="Releasing addresses for endpoint 0_6424036164612164775_step_0's interface on network 0_6424036164612164775_default"
debug msg="ReleaseAddress(LocalDefault/172.18.0.0/16, 172.18.0.2)"
debug msg="Revoking external connectivity on endpoint 0_6424036164612164775_step_0 (916882334f1f4443bd4aa1cbb0bc99e8cd4f4165afa797961039b85d59dd4958)"
debug msg="Releasing addresses for endpoint 0_6424036164612164775_step_0's interface on network bridge"
debug msg="ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.2)"
debug msg="Calling GET /v1.26/containers/0_6424036164612164775_step_0/json"
debug msg="Calling POST /v1.26/containers/0_6424036164612164775_clone/kill?signal=9"
error msg="Handler for POST /v1.26/containers/0_6424036164612164775_clone/kill returned error: Cannot kill container 0_6424036164612164775_clone: Container 0c519f446c19bf5a05dfabd59fef6d328e1e380b3e694923e868e12038964523 is not running"
debug msg="Calling DELETE /v1.26/containers/0_6424036164612164775_clone?v=1"
debug msg="Calling POST /v1.26/containers/0_6424036164612164775_step_0/kill?signal=9"
error msg="Handler for POST /v1.26/containers/0_6424036164612164775_step_0/kill returned error: Cannot kill container 0_6424036164612164775_step_0: Container 77b83112db408c676a45b953f1a1927cd18314a5745ebf487309f8589793965d is not running"
debug msg="Calling DELETE /v1.26/containers/0_6424036164612164775_step_0?v=1"
debug msg="Removing volume reference: driver local, name 327af57240114b2e75c5071ebaaf2b48f9a2ce7bcaefeb674fd77ec54af67a07"
debug msg="Calling POST /v1.26/containers/0_6424036164612164775_step_1/kill?signal=9"
error msg="Handler for POST /v1.26/containers/0_6424036164612164775_step_1/kill returned error: Cannot kill container 0_6424036164612164775_step_1: No such container: 0_6424036164612164775_step_1"
debug msg="Calling DELETE /v1.26/containers/0_6424036164612164775_step_1?v=1"
error msg="Handler for DELETE /v1.26/containers/0_6424036164612164775_step_1 returned error: No such container: 0_6424036164612164775_step_1"
debug msg="Calling POST /v1.26/containers/0_6424036164612164775_step_2/kill?signal=9"
error msg="Handler for POST /v1.26/containers/0_6424036164612164775_step_2/kill returned error: Cannot kill container 0_6424036164612164775_step_2: No such container: 0_6424036164612164775_step_2"
debug msg="Calling DELETE /v1.26/containers/0_6424036164612164775_step_2?v=1"
error msg="Handler for DELETE /v1.26/containers/0_6424036164612164775_step_2 returned error: No such container: 0_6424036164612164775_step_2"
debug msg="Calling DELETE /v1.26/volumes/0_6424036164612164775_default?force=1"
debug msg="Removing volume reference: driver local, name 0_6424036164612164775_default"
debug msg="Calling DELETE /v1.26/networks/0_6424036164612164775_default"
debug msg="/sbin/iptables, [--wait -t nat -C POSTROUTING -s 172.18.0.0/16 ! -o br-c7765c6c80c1 -j MASQUERADE]"
debug msg="/sbin/iptables, [--wait -t nat -D POSTROUTING -s 172.18.0.0/16 ! -o br-c7765c6c80c1 -j MASQUERADE]"
debug msg="/sbin/iptables, [--wait -t nat -C DOCKER -i br-c7765c6c80c1 -j RETURN]"
debug msg="/sbin/iptables, [--wait -t nat -D DOCKER -i br-c7765c6c80c1 -j RETURN]"
debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -i br-c7765c6c80c1 -o br-c7765c6c80c1 -j ACCEPT]"
debug msg="/sbin/iptables, [--wait -D FORWARD -i br-c7765c6c80c1 -o br-c7765c6c80c1 -j ACCEPT]"
debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -i br-c7765c6c80c1 ! -o br-c7765c6c80c1 -j ACCEPT]"
debug msg="/sbin/iptables, [--wait -D FORWARD -i br-c7765c6c80c1 ! -o br-c7765c6c80c1 -j ACCEPT]"
debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -o br-c7765c6c80c1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT]"
debug msg="/sbin/iptables, [--wait -D FORWARD -o br-c7765c6c80c1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT]"
debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -o br-c7765c6c80c1 -j DOCKER]"
debug msg="/sbin/iptables, [--wait -t filter -C FORWARD -o br-c7765c6c80c1 -j DOCKER]"
debug msg="/sbin/iptables, [--wait -D FORWARD -o br-c7765c6c80c1 -j DOCKER]"
debug msg="/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION -i br-c7765c6c80c1 -o docker0 -j DROP]"
debug msg="/sbin/iptables, [--wait -D DOCKER-ISOLATION -i br-c7765c6c80c1 -o docker0 -j DROP]"
debug msg="/sbin/iptables, [--wait -t filter -C DOCKER-ISOLATION -i docker0 -o br-c7765c6c80c1 -j DROP]"
debug msg="/sbin/iptables, [--wait -D DOCKER-ISOLATION -i docker0 -o br-c7765c6c80c1 -j DROP]"
debug msg="releasing IPv4 pools from network 0_6424036164612164775_default (c7765c6c80c1c64b7d3fadba9e17ce661c1863119f2e4aad7d4929456b11ea06)"
debug msg="ReleaseAddress(LocalDefault/172.18.0.0/16, 172.18.0.1)"
debug msg="ReleasePool(LocalDefault/172.18.0.0/16)"
5

It seems that the agent does not tell the plugin that it is running in privileged mode. I had to make the repo trusted then I just got this to work by adding the following flag to all the pipeline steps:

privileged: true

Is this expected for anything that uses the plugin to be set on every single pipeline step, or is there some server wide setting I can apply this?

We did not have to do this before, so I’m not sure if this is something changed or if it was just something we should have been doing all along.

6

Drone automatically configures the plugins/dockage image to run in privileged mode, as long as certain criteria are met. Specifically you cannot try to override the entrypoint or command, for security reasons.

Drone decides which images to escalate to privileged mode using this list:

We did not have to do this before, so I’m not sure if this is something changed or if it was just something we should have been doing all along.

There have been no changes to drone or to the image that would impact the above logic. Nor have we received reports of similar issues. If a regression were introduced, we would be unable to build drone or any of the drone plugins so we would probably catch it pretty quickly …

7

Aha! There we go! That flag was being set to one of our custom build containers. Once I removed that it seems to have fixed the aforementioned issues. Thanks for your help. I think this resolves my issue across the board.