MTU problems (Original title: Github oauth flow breaks, cannot login)

Drone Support
1

EDIT: Problems were understood and solved, see last two posts.

Hi everybody,

After running drone in the office for some time now, I’m really happy with it and want to setup a small instance on my mini server at home. I’m trying to setup drone on the following:

  • small server on ARM cpu with 4GB of RAM
  • Ubuntu 20.04
  • Docker 20.10.11
  • Caddy as reverse proxy in front

I followed the official guide from here closely: GitHub | Drone

Sadly, the login doesn’t work and I’m out of ideas. It will properly redirect me to github, I can grant access and am then redirected to my-drone.host.com/login with the code issued by github. This then takes forever until something times out and I get this error message:

oauth: cannot exchange code: [code here]: Post \"https://github.com/login/oauth/access_token\": read tcp 172.20.0.2:48520->140.82.121.4:443: read: connection reset by peer

I already checked a lot of things. Outgoing connections to github work, also from inside the drone container I can ping github.com. I restarted the docker service and even tried to disable ufw to see if there is some connection issue, but it should actually be working.

Is this a known issue? What else could I try to get it working? Thanks for your help!

2

there are no known issues with github logins.

read: connection reset by peer

this indicates a networking (or maybe dns) error

This then takes forever until something times out and I get this error message:

this likely indicates a TCP timeout which would indicate a possible networking or dns issue

Outgoing connections to github work

keep in mind the redirect to github is client-side (in your browser) and is not indicative of the ability to connect with github from inside the container network.

also from inside the drone container I can ping github.com

are you sure it is pinging github? If this is a dns issue it could be pinging a valid address, but not the github address. Perhaps try using curl to connect with github from inside the container? We have seen this exact scenario where ping was successful, but curl failed, indicating a problem with docker networking (not an issue with drone). See this thread http://discuss.harness.io/t/cannot-access-public-facing-loadbalancer-from-spawned-container/1700 where someone faced a similar issue and was able to ping github.com but not curl github.com, which helped them rule out issues with Drone.

3

Thanks for your quick reply. Sadly, it seems not to be a DNS issue. I fiddled around with DNS a lot now and am very sure, that it works fine. From the drone server container, I can wget google.com and get the google page, same for any other website. The only thing that does not work is wget github.com. Github just won’t answer.

Based on my research I am pretty sure that the problem must be somewhere in the networking setup that docker-compose adds. When running docker run -it alpine I can easily access github.com, as soon as I do docker run -it --network=drone_network alpine I can’t. The network in the docker-compose file is just a default bridge network, nothing special. Still there must be a difference.

I’ll try to create a new network to see if I can fix this.

4

Found it. There must be some strange fiddling with my data connection by the provider. Had to set the mtu on the network to 1300 to make it work.

In case anybody else stumbles into this: First make sure DNS works properly. To do this, you can run docker exec [drone_server_container] wget -O- anywebsite.com. If only github doesn’t answer, DNS should be fine. If not, fix DNS.

If you suspect that the MTU might be the problem, you can lower it. If you want to try my solution: set the MTU on the network to something lower than 1500 (1300 worked for me, google MTU to understand what might work), I did it in my docker-compose.yml:

networks:
  drone:
    name: drone_network
    driver: bridge
    driver_opts:
      com.docker.network.driver.mtu: 1300

I then ended up killing and removing all networks manually and also recreating the docker container for drone. After that, I got it working with the recreated network.

If you are not sure whether or not your network took the settings, run docker network inspect [network_name], this should give you a dump including the settings.

Thanks @bradrydzewski for the help and for drone. :slight_smile:

EDIT: See post below if you run into similar issues after fixing the first one.

5

Followup information:

You might run into downstream problems caused by changing the MTU. The nested containers and networks created by drone also need to fix the MTU to make connection work all the way down.

To fix it in the network that is created by drone, add this to your runner (Again, MTU as you need):

DRONE_RUNNER_NETWORK_OPTS=com.docker.network.driver.mtu:1300

And then if running dind, you’ll again have to fix the MTU. So when building images with Docker | Drone , make sure to set the mtu parameter in the plugin settings.