Following Scenario: I have a Git server (Gitea) and Drone on the same host running with Rancher. One loadbalancer (haproxy) in front of everything which routes based on domains and handles ssl termination.
Drones spawned containers cannot acces the public facing services (made public by ranchers loadbalancer) when trying to access the public domain/ip of the server, however they have internet access and can reach the server via ping. Dns don’t seem to be the issue.
The Build fails on the first step (clone) with a timeout.
Doing a scan with nmap from inside of one of these contaiers gets me the following result:
Host is up (0.00013s latency).
Not shown: 995 closed ports
PORT STATE SERVICE VERSION
80/tcp filtered http
111/tcp open rpcbind 2-4 (RPC #100000)
443/tcp filtered https
8080/tcp open tcpwrapped
The interesting part is in port 80/443, these are the two ports handeled by the rancher loadbalancer. Doing curl on them from inside a drone build container results in a timeout.
I also tried running manually a container on the host (outside of rancher), whithin this i can access everything perfectly normal.
After digging a bit, I think this is probably an issue with ranchers iptables configuration, I think it would explain why I could ping the host but don’t access services provided by it’s loadbalancer.
I also created an issue on the rancher repo: https://github.com/rancher/rancher/issues/11288
I also tried running manually a container on the host (outside of rancher), whithin this i can access everything perfectly normal.
Drone creates user-defined networks (eg docker network create) for each pipeline execution. So if you are testing on the host machine, you should structure your test similar to the below commands, in order to get an apples to apples comparison:
docker network create foo
docker run --network=foo -t -i alpine ping -c 1 github.com
I would not be surprised if the default bridge network has some special network or DNS configuration that does not apply to user-defined networks. Just a guess, though, based on similar issues we’ve seen in the past.
DNS doesn’t seem to be the issue here, ping works just fine, with my host as well as with any other public host.
However, curl results in this:
$ docker run --network=foo -t -i appropriate/curl git.kolaente.de
curl: (7) Failed to connect to git.kolaente.de port 80: Operation timed out
But only on my own host (the one the container is running on). Doing curl on any other frontfacing webservice (such as github) works perfectly fine, I only get timouts when doing curl on my own services which run on the same host but with rancher.
Doing the same command from another server works without any issue.
Sorry, not sure I am completely following. I can only confirm that Drone creates user-defined networks and does not alter, override or implement any custom networking magic beyond standard Docker defaults. If the standard user-defined Docker networks are not working as expected, I would encourage you to work with Rancher folks, since this is outside my area of expertise.
With that being said, I definitely encourage you to continue to use this forum to interact with other Drone+Rancher users in search of a solution. I can help if you have any Drone-specific questions or if you identify a solution for improved support of Rancher networking within Drone.
Yeah sure, you already helped me. I havent thought of the network thing, I only tested in a containter started by the drone agent, so I thought this could be a drone issue. But as it is the same with any container running in a custom network, this seems to be a rancher issue. I’ll ask in Ranchers forum for help.
Keep on with the good work, Drone is a really great product!