Docker mtu problem

Drone Support
1

The environment I use is openstack vm, The network card mtu is 1454 ,not 1500
When I build with drone, I find that it creates a new container and the mtu of the network card
is 1500 . so the container was not able to access the Internet

by the way ,during the use, I discovered that the drone created a bridge network using docker and the mtu is 1500, I need to set the mtu of this network for 1400 to get online

I’ve done the following, but it doesn’t work

networks:
default:
driver: bridge
driver_opts: io.drone.network.pod=true

My drone version is 0.8.1

What else do I need to write, thank you.

2

You can try to change the docker daemon setting by modifying /etc/docker/daemon.json with :

{
'mtu': '1400'
}

or change direcetly the value in you compose file in the network section :

networks:
  default:
    driver: bridge
    driver_opts:
      com.docker.network.driver.mtu: 1400
3

Actually it’s not exactly Drone issues, but Docker (not possible to set default MTU for all new networks). Even if you will change MTU settings in daemon.json it will be applied only to default bridge. Settings inside docker-compose.yml will be applied only to networks that was created for drone-server/agent. But Drone will create additional network and that network will have default MTU 1500. So in result I end up with modifying Drone source code to be able to pass network options to this Drone-created network.

And I want to ask you @bradrydzewski do you interested in such Pull Request to 2 repos: drone/drone and cncd/pipeline to add ability to pass options for Drone created networks?

2 Likes
4

I am having the same issue. I am running Drone 0.8.2 inside the OpenStack VM which runs the Kubernetes with Weave Net CNI (Container Network Interface) which is running with the MTU of 1360 bytes.

At first, I thought that having the --mtu set to 1360 bytes for the drone-dind (docker:17.12.0-ce-dind) would have solved the issue, only if it was keeping that --mtu=1360 as the default for the newly created docker network bridges. Unfortunately, this is not the case [1].

I believe this should not be very hard to extend the Drone and let the users set their options such as the MTU value for the Docker network [2].

At the moment, my quick and dirty workaround is to leverage the “clamp the MSS” (Linux kernel feature in the network stack that calculates the proper MSS for the link) [3].

This is the piece of my configuration for the Kubernetes deployment:

I am using the DOCKER-USER iptables chain as the Docker is trying to place it as the first rule under the FORWARD chain.

Please also note that I am using --host=tcp://127.0.0.1:2375 instead of --host=tcp://0.0.0.0:2375 dockerd argument as the latter one allows an unprivileged container (ran in Drone) to escape to the drone-dind.

      - name: drone-dind
        image: "docker:17.12.0-ce-dind"
        imagePullPolicy: IfNotPresent
        command: ["/bin/sh"]
        args: ["-c", "iptables -N DOCKER-USER; iptables -I DOCKER-USER -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu; dockerd --host=unix:///var/run/docker.sock --host=tcp://127.0.0.1:2375"]
1 Like
6

Thank you. I’ve solved this problem by setting the network_mode property

cicd

7

Thank you for your reply
I’ve tried it, but it doesn’t work.

8

The network_mode property works only when you set the Trusted repo option which can only be done by Drone admin. Since the trusted repo also allows running containers in a privileged mode, this might not be a solution to many people, including me.

9

Hi, is there any solution for this in the current Drone CI? I am trying to run it using Docker Swarm on OpenStack where MTU is 1442. Then the runners always fail on fetching the repository :man_facepalming:

Initialized empty Git repository in /drone/src/.git/
+ git fetch origin +refs/heads/drone:
fatal: unable to access 'https://github.com/MarekSuchanek/engine-backend.git/': LibreSSL 
SSL_connect: SSL_ERROR_SYSCALL in connection to github.com:443
10

I pushed this patch that allows you to set global docker network options. This patch is available in the latest version of the runner image. You can set global network options as follows:

DRONE_RUNNER_NETWORK_OPTS=com.docker.network.drive.mtu:1442

This patch should work based on the docker documentation, but I do not have an environment that I can use to test this. If you experience any issues we would kindly ask that you send a pull request that improves this code.

1 Like
11

Thanks! It worked for me, just I needed to use:

DRONE_RUNNER_NETWORK_OPTS=com.docker.network.driver.mtu:1442

(typo, driver instead of drive - if someone finds the solution here)

12

This works for the docker runner, but how about docker exec? When using a VPN locally, the MTU of the newly created docker networks always defaults to 1500.

13

Did you ever get this sorted? I’m facing the same issue with newly created containers using a different docker network with MTU reverted back to 1500.