Plugins/docker fails DNS queries

Drone Support
1

My Dockerfile which is setup to run a bunch of tests, nslookup and curl fail, but I can have no issue pinging the servers. Has anyone experienced something similar? The gist contains the full output.

FROM davidneudorfer/testkid:0.0.3 (ubuntu with added tools)

RUN ping -c5 8.8.8.8
RUN ping -c5 208.67.222.222
RUN ifconfig
RUN uname -a
RUN cat /etc/resolv.conf
RUN route
RUN ip address
RUN ip route
RUN nslookup google.com
RUN curl -IL google.com
RUN curl ifconfig.me
2

As soon as I started using my VPC’s private dns things started working

  docker_test:
    image: plugins/docker
    custom_dns: [ 10.10.0.2 ]
1 Like
3

Pretty sure I’m also seeing similar issues too, but just straight up using the plugins/docker plugin to build Docker images. I added some extra logging to a custom version of the plugin and I see errors like this (which currently are obscured from the user):

Error authenticating to Docker daemon (retrying 538/600): exit status 1
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://r.mills.io/v2/: dial tcp: lookup r.mills.io on 127.0.0.11:53: read udp 127.0.0.1:60798->127.0.0.11:53: read: connection refused

I know it’s not a networking problem, because I also put some extra long retries in the custom version of plugins/docker so I can docker exec into the spaned runner container to see wtf is going on. Here’s what I find:

  • Networking works fine, local LAN as well as Internet. i.e: ping <ip> responds just fine.
  • DNS fails with the above.

Once I edit /etc/resolv.conf in the container and swap nameserver 127.0.0.11 with my local DNS server on the LAN (e.g: 10.0.0.1) DNS works (obviously).

It seems the builtin DNS server in the Docker daemon is not responding for some reason and I’ve been debugging this for weeks with no succeed. I haven’t been able to find any good hints as why either as this is a really hard topic to web search.

:confused:

@bradrydzewski You got any ideas?

4

Sorry, this is outside of my area of expertise and outside of the scope of support that I am able to provide. The best place to ask questions about docker / report issues with docker dns / report issues with the docker:dind image would probably be the docker issue tracker. The core docker maintainers are going to be your best resource.

5

@bradrydzewski That’s okay! I figured as much. There’s no hint of what’s going on when you run with debug: true either which spits out a lot of debugging information about the Docker daemon startup.

I’ll keep debugging this on my own, debug the Docker daemon code if I have to and report back if I manage to solve this :smiley:

6

Some additional info from today’s debugging.

  • Running the exact same docker run command as the drone-runner would run results in a non producer. i.e: I can’t seem to reproduce this outside of the Drone Runner (itself running as a Docker container). I even tried running the drone-docker plugin from inside a drone-runner instance, still couldn’t reproduce it.

This makes me very curious as to how the drone-runner is running the plugins? Hmmm? :thinking:

As a work-around I’ve found that the plugins/kaniko plugin for building docker images works perfectly with minimal changes to your pipeline; just swap the image: out.

7

Drone creates a user-defined network for every pipeline execution. If you are testing with docker run you are probably using the default bridge network. The default bridge network uses a different dns resolution strategy than user-defined networks [1]

By default, a container inherits the DNS settings of the host, as defined in the /etc/resolv.conf configuration file. Containers that use the default bridge network get a copy of this file, whereas containers that use a custom network use Docker’s embedded DNS server, which forwards external DNS lookups to the DNS servers configured on the host.

This topic comes up every few years [2] and while I empathize that you are experiencing issues, I am not sure there is anything new for me to add to this discussion. I am glad to hear the Kaniko plugin is working for you. If you (or anyone reading this thread) wants to continue triaging Docker networking or dns issues further, we would continue to recommend reaching out to the Docker maintainers for assistance. If they come up with any novel ideas, we are happy to incorporate them into the plugin or the documentation (or both).

[1] https://docs.docker.com/config/containers/container-networking/#dns-services
[2] http://discuss.harness.io/t/dind-container-not-receiving-host-resolv-conf-settings/811/2

8

Came here to say I had the same issue as above and found @prologic 's solution to be the best (and only). Using the banzaicloud/drone-kaniko image instead of the plugin/docker image worked immediately.

At first I could fix this issue by attaching bridge and a custom network br0 (specifically in that order) to the runner’s DRONE_RUNNER_NETWORKS, but once I set up VLANs with a new router this stopped working.

  • I could ping IPs and hostnames from the plugin/docker container itself while it was executing the Dockerfile.
  • I tried forcing DNS on the containers themselves, in the .drome.yml, and in the Dockerfile during the build process itself.

The only thing I didn’t try was messing with the Daemon DNS running in the ephemeral container executing the Dockerfile commands. Wasn’t sure how to do this, I would expect the custom_dns for the .drone.yml would do this but it didn’t.