Kubernetes docker build failed

metal3d · May 25, 2019, 9:56am

Hello
I spent a long time with 2 kubernetes clusters to try to build images with Drone.

I installed Drone with helm chart (official) and I’m using Gitea. Everything is fine until I try to build image.

My .drone.yml file is:

kind: pipeline
name: default

steps:
  - name: test
    image: plugins/docker
    commands:
      - docker ps

I also tried “docker” image and “docker:dind”. What I see is:

+ docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

My repository is “Trusted” and I set up drone with my admin account that is the user on gitea which is mine.

The problem is that, later, I want to use “source to image” to build image and push it on my private repository, so I will need that kind of commands:

- s2i build . my-base-image 10.x.x.x:5000/my-app-image
- docker push 10.x.x.x:5000/my-app-image

And it needs docker to pull base image, then push my image.

Here is the lines in drone deployment:

  - env:
    - name: DRONE_KUBERNETES_ENABLED
      value: "true"
    - name: DRONE_KUBERNETES_NAMESPACE
      value: drone
    - name: DRONE_KUBERNETES_SERVICE_ACCOUNT
      value: drone-drone-pipeline
    - name: DRONE_GIT_ALWAYS_AUTH
      value: "false"
    # I hide that
    - name: DRONE_SERVER_HOST
      value: drone.XXXXX
    - name: DRONE_SERVER_PROTO
      value: https
    - name: DRONE_USER_CREATE
      value: username:metal3d,machine:false,admin:true
    - name: DRONE_RPC_SECRET
      valueFrom:
        secretKeyRef:
          key: secret
          name: drone-drone
    - name: DRONE_DATABASE_DATASOURCE
      value: /var/lib/drone/drone.sqlite
    - name: DRONE_DATABASE_DRIVER
      value: sqlite3
    - name: DRONE_LOGS_DEBUG
      value: "false"
    # I hide that
    - name: DRONE_GITEA_SERVER
      value: https://XXXXXX

Also, note that I didn’t found any /var/lib/docker mount point in the deployment, so I really wonder how Drone can build image in Kubernetes “pods”.

Thanks

metal3d · May 25, 2019, 9:59am

I forget to say that several repositories, where I’m not the owner, will need to build images with s2i. I really wonder how I will be able to make that working.

narze · May 25, 2019, 1:23pm

This drone configuration works for me on official helm chart.
Note that I have to use volumes on every pipelines to make with docker layer caching working.
But I’m still stuck at no space left on device issue. I’m going to stop using k8s method for now and use https://autoscale.drone.io/ instead.

kind: pipeline
name: prepare

clone:
  disable: true

failure: ignore

steps:
  - name: ensure-cache-path
    image: busybox
    commands:
      - df -h
      - ls -la /drone/src
    volumes:
      - name: cache
        path: /cache

services:
  - name: docker
    image: docker:dind
    privileged: true
    environment:
      DOCKER_LAUNCH_DEBUG:
        true
      DOCKER_HOST:
        unix:///var/run/host.sock
    volumes:
      - name: dockersock
        path: /var/run
      - name: docker
        path: /var/lib/docker

volumes:
  - name: docker
    host:
      path: /var/cache/${DRONE_REPO}/docker
  - name: dockersock
temp: {}

---
kind: pipeline
name: build

depends_on:
  - prepare

clone:
  depth: 1

globals:
  - &docker_creds
    username:
      from_secret: docker_username
    password:
      from_secret: docker_password

steps:
  - name: ls
    image: busybox
    commands:
      - ls -la /drone/src
  - name: build-docker
    image: plugins/docker
    settings:
      tags:
        - ${DRONE_BRANCH//\//-}-${DRONE_COMMIT_SHA:0:8}
      repo: xxxx/yyyy
      use_cache: true
      <<: *docker_creds
    volumes:
      - name: docker
        path: /var/lib/docker

volumes:
  - name: cache
    host:
      path: /var/cache
  - name: docker
    host:
      path: /var/cache/${DRONE_REPO}/docker

metal3d · May 25, 2019, 2:01pm

Hi, thanks a lot, I will try that.

In the same time I realized that plugins/docker is working, the problem is that I wanted to use s2i. So I managed a “fake s2i” like that:

kind: pipeline
name: default

steps:
  - name: prepare-s2i
    image: alpine
    commands:
      - |
        cat > Dockerfile << EOF
        FROM 10.233.16.154:5000/metal3d/nginx:1.15-s2i
        ADD . ./
        RUN mkdir -p /tmp/src; /usr/libexec/s2i/assemble 
        CMD ["/usr/libexec/s2i/run"]
        EOF
      - cat Dockerfile
 
  - name: build
    image: plugins/docker
    settings:
      registry: 10.233.16.154:5000
      repo: 10.233.16.154:5000/metal3d/httptest
      insecure: true

This is a “trick”, I only prepare a Dockerfile that makes exactly what s2i does: take an image, build sources with “assemble” and set the command to “run”.

Note that the image is now well built and pushed to the private registry (only accessible through Kubernetes cluster, I didn’t tried to use service name but it should work)

That is very simple, maybe more adapted to my workflow. But, anyway, I will take a look on your solution that can help me a lot later. One more time, thanks a lot !

bradrydzewski · May 25, 2019, 3:26pm

EDIT looks like you figured out how to get the Docker plugin working. The below explanations may still be useful to understand why the various configurations previously attempted did not work.

NOTE the recommended approach to building and publishing docker images is using the plugins/docker plugin, as you have done in your updated example. This can be done without requiring trusted mode and is therefore more safe. Other options that require trusted mode are of course viable alternatives, but trusted mode inherently introduces security concerns since it effectively grants the build unrestricted root access to your host machine.

There are two documented examples for how to build docker images using drone:
https://docs.drone.io/examples/service/docker_dind/
https://docs.drone.io/examples/service/docker/

One issue with your example is that there is no docker daemon running inside plugins/docker because you are overriding the default plugin entrypoint when you set commands. If you are going to use this plugin image, it should be used in the following manner http://plugins.drone.io/drone-plugins/drone-docker/

I also tried “docker” image and “docker:dind”

Here are some root causes for the error you described:

the docker image does not start a docker daemon
the docker:dind starts a docker daemon, but the image requires privileged: true
the docker:dind starts a docker daemon as the entrypoint, but when you set commands you are overriding the default container entrypoint, and as a result no docker daemon starts.

My repository is “Trusted” and I set up drone with my admin account that is the user on gitea which is mine.

setting a repository as trusted is the first step. The second step is that you need to run containers in privileged mode, using privileged: true

metal3d · May 25, 2019, 4:02pm

Thanks, that’s what I’ve understood later.

What I really need, actually, is to create a plugin that uses “s2i” command - this command needs a docker daemon somewhere to pull base image, and to push resulting image.

What I did is a “trick” to avoid that command usage, but s2i allows several options.

I already tried to build a “s2i” image using “FROM plugins/docker” but it seems that I’ve got many problems to override it…

My plugin Dockerfile makes:

FROM plugins/docker

ENV S2I_VERSION=1.1.14 \
    S2I_GITCOMMIT=874754de

RUN set -xe; \
    wget -O - https://github.com/openshift/source-to-image/releases/download/v${S2I_VERSION}/source-to-image-v${S2I_VERSION}-${S2I_GITCOMMIT}-linux-amd64.tar.gz | tar -C /usr/local/bin -zxf - ./s2i

ADD s2ibuild.sh /usr/local/bin/s2ibuild.sh

ENTRYPOINT ["/usr/local/bin/dockerd-entrypoint.sh", "/usr/local/bin/s2ibuild.sh"]

And the builder is:
#!/bin/sh -e

set -e

s2i build /drone/src ${PLUGIN_CONTEXT-./} ${PLUGIN_IMAGE} ${PLUGIN_TARGET}
docker push ${PLUGIN_TARGET}

And the .drone.yml file is:

kind: pipeline
name: default

steps:
  - name: s2i-build
    image: metal3d/drone-plugin-s2i
    settings:
      registry: docker-docker-registry.registry:5000
      image: docker-docker-registry.registry:5000/metal3d/nginx:1.15-s2i
      target: docker-docker-registry.registry:5000/metal3d/httptest
      insecure: true

And of course… I’ve got the error:

FATAL: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Something is missing in my image, but I don’t find what… If you can help ?

bradrydzewski · May 25, 2019, 6:33pm

Here are some tips:

you need to set privileged: true for your pipeline step. The plugins/docker image is whitelisted and automatically run in privileged mode, but your custom plugin will not.
where are you starting the docker daemon? If you look at plugins/docker source code you will see that we start the docker daemon in the Go code [1]

Please note that you can test plugins locally [2] which makes it easier to debug them and triage these sorts of issues. I recommend studying the docker plugin source code [3] more closely which may help you proceed.

[1] https://github.com/drone-plugins/drone-docker/blob/master/docker.go#L70:L84
[2] https://github.com/drone-plugins/drone-docker#usage
[3] https://github.com/drone-plugins/drone-docker

metal3d · May 25, 2019, 10:09pm

This is what I didn’t undertand until I’ve taken a look on the drone-docker sources (just minutes before your answer )

This is, IMHO, one point that Drone is missing… I’m pretty sure you’re very busy and you will not create such whitelisted plugin. But if one day you want to make it possible, please consider s2i to be a very useful plugin. I know Go and I probably can build such plugin myself, but I am running out of time (and I don’t know the structure as well as you, I will spend so much more time than you)

Even if I was able to find time to develop such of plugin, it will not be whitelisted and my teammate will not be able to activate it (no privilege) - so I have to use my workaround for now. It’s usable, that’s the essential goal

S2I would be very interesting to avoid my teammate to overwrite Dockerfile, and let a “base image” to assemble the image. The project sources doesn’t need to have a Dockerfile. It’s a bit more secured and it can make incremental build as well.

It’s not so bad, as I’m able to fake the s2i build by generating a Dockerfile, I will be able to leave that for now.

PS: Drone is very impressive, it works very well now in Kubernetes, and I’m very happy to use it. This little worry does not spoil my enthusiasm to use it at all.

bradrydzewski · May 25, 2019, 10:42pm

you can add to the white-list by setting the following environment variable:
https://docs.drone.io/reference/agent/drone-runner-privileged-images/

metal3d · May 25, 2019, 10:44pm

Ho great !

OK, so maybe I wll be able to launch docker daemon my self, maybe in bash… I will take a look later.

Thanks a lot for all that help, for Drone, and so on

metal3d · May 28, 2019, 11:08pm

@bradrydzewski my god it’s now ok.

I now use “plugins/docker” as base image, and I can start docker daemon without privilege mode (I whitelisted my plugin)
I can now do that:
kind: pipeline
name: default

steps:
  - name: s2i-build
    image: metal3d/drone-plugin-s2i
    pull: always
    settings:
      registry: docker-docker-registry.registry:5000
      insecure: true
      image: docker-docker-registry.registry:5000/metal3d/nginx:1.15-s2i 
      target: docker-docker-registry.registry:5000/metal3d/httptest

My entrypoint makes some tests, then starts daemon and waits docker to be ready (using nc on unix socket, and getting docker version)

After docker is ready, it launches:

s2i build ${DRONE_WORKSPACE_BASE} \
    --context-dir=${PLUGIN_CONTEXT-./} \
   ${PLUGIN_IMAGE} ${PLUGIN_TARGET}

So the build process is now able to get a “s2i base image” and build a target image.

I have to setup some params to push image or not, but it works great.

One more time, I’m so happy to have chosen Drone as CD solution

@bradrydzewski thanks a lot for your help

EDIT: I will give my plugin with opensource licence - wait some minutes

metal3d · May 28, 2019, 11:41pm

Plugin is ready:

https://hub.docker.com/r/metal3d/drone-plugin-s2i

I will need to implement docker login, of someone can help ?