Problem building docker image, alpine upgrade

Drone Support
1

Hey!
I’ve got a problem during a build with Drone. I use a kubernetes setup and installed Drone and drone-runner-docker via Helm. Beside my other problem regarding HPA everything’s working so far except for this error: I cannot run RUN apk upgrade --no-cache, it hangs until the build is running in timeout.

Step 2/7 : RUN apk upgrade --no-cache
 ---> Running in e40c2117c10b
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz

If I exec into the runner dind pod, I am enable to contact alpinelinux.org:

$ kubectl exec -it  drone-runner-docker-669547bf49-sjklc -n drone -- /bin/sh
Defaulted container "dind" out of: dind, gc, drone-runner-docker
/ # apk upgrade --no-cache
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz
(1/2) Upgrading musl-utils (1.2.3-r1 -> 1.2.3-r2)
(2/2) Upgrading krb5-libs (1.19.3-r0 -> 1.19.4-r0)
Executing busybox-1.35.0-r17.trigger
OK: 31 MiB in 59 packages
/ #

Locally the Dockerfile is getting build without a problem. What am I doing wrong?

Best regards,
Panda

2

Hello @panda

Thanks for reporting this. I wrote the drone-runner-docker Helm chart, so I appreciate the feedback.

Can you try running a step in a debian or ubuntu image, and just try running the apt-get update command? The problem you are seeing might be related to recent versions of alpine on docker. If that works, I need to do some research.

3

Hey!

I tested another image as you suggested:

Step 2/7 : RUN apt update && apt upgrade -y
 ---> Running in 57a3dffe6d48
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Get:1 http://nginx.org/packages/debian buster InRelease [3601 B]
Get:2 http://nginx.org/packages/debian buster/nginx Sources [16.7 kB]
Get:3 http://nginx.org/packages/debian buster/nginx amd64 Packages [24.8 kB]
Get:4 http://deb.debian.org/debian buster InRelease [122 kB]
Err:4 http://deb.debian.org/debian buster InRelease
  Connection failed [IP: 199.232.138.132 80]
Get:5 http://deb.debian.org/debian-security buster/updates InRelease [34.8 kB]
Get:5 http://deb.debian.org/debian-security buster/updates InRelease [34.8 kB]
Get:5 http://deb.debian.org/debian-security buster/updates InRelease [34.8 kB]
Get:5 http://deb.debian.org/debian-security buster/updates InRelease [34.8 kB]
Err:5 http://deb.debian.org/debian-security buster/updates InRelease
  Could not connect to deb.debian.org:80 (199.232.138.132), connection timed out [IP: 199.232.138.132 80]
Err:6 http://deb.debian.org/debian buster-updates InRelease
  Unable to connect to deb.debian.org:http: [IP: 199.232.138.132 80]
Fetched 45.1 kB in 3min 31s (214 B/s)
Reading package lists...
Building dependency tree...
Reading state information...
All packages are up to date.

W: Failed to fetch http://deb.debian.org/debian/dists/buster/InRelease  Connection failed [IP: 199.232.138.132 80]
W: Failed to fetch http://deb.debian.org/debian-security/dists/buster/updates/InRelease  Could not connect to deb.debian.org:80 (199.232.138.132), connection timed out [IP: 199.232.138.132 80]
W: Failed to fetch http://deb.debian.org/debian/dists/buster-updates/InRelease  Unable to connect to deb.debian.org:http: [IP: 199.232.138.132 80]
W: Some index files failed to download. They have been ignored, or old ones used instead.

At least aptitude isn’t waiting :smile: I also switched dind image from alpine to Ubuntu I guess. No change. Next thing I gonna try is this. Maybe a MTU size Problem?

BR
panda

Tried these with mtu 1400 and 1280 - no success so far…

dind:
  registry: docker.io
  repository: docker
  # tag: 20.10.21-dind
  # tag: 20.10.21-dind-alpine3.16
  tag: 20-dind
  pullPolicy: IfNotPresent
  command:
    - "dockerd"
  commandArgs:
    - "--host"
    - "tcp://localhost:2375"
    - "--mtu=1280"
4

Indeed, it’s an MTU problem… Adding mtu: 1280 inside the step helped.

steps:
- name: docker
  image: plugins/docker
  settings:
    username:
      from_secret: REGISTRY_USER
    password:
      from_secret: REGISTRY_PASSWORD
    registry: ***
    repo: ***
    tags: latest
    mtu: 1280

Now my build is updated & green! Thank you for your help :slight_smile:

5

A-ha! MTU! I should have thought of that…

This is definitely something we need to document, I’ll write something up tomorrow.

1 Like
6

@panda I created a PR with documentation updates related to the MTU [SKIP CI] adding documentation related to MTU by jimsheldon · Pull Request #109 · drone/charts · GitHub

There is some info not covered in our conversation here, such as setting com.docker.network.driver.mtu for the docker runner. I believe that without that setting, other steps (not plugins/docker) in your pipeline would also have networking issues.

Let me know if the PR looks good and I’ll merge it.

Thanks again for the help!

1 Like
7

Thank you! For me it looks good. Added com.docker.network.driver.mtu to my deployment and have no issues so far. Every build’s still green :slight_smile:

1 Like