The steps labeled mongo, memcached, mysql, solr and redis are all services that need to start in order for this pipeline to succeed. But after the clone step runs, only the first two services start and at this point it hangs indefinitely. Inspecting logs for drone-server shows no activity and no errors during this time. I do see the new namespace when I inspect with kubectl and inside that namespace I see nodes for the completed clone step and the two running steps, but no others:
kubectl get all --namespace 5df4qu2u0efmrrgsk0h2oua7atj0yv02
NAME READY STATUS RESTARTS AGE
pod/bmztcu9hgmopsgvb3t6vys44jtgpucy9 0/1 Completed 0 9m8s
pod/hozuoms4ag2alixz76uwj9qgz262zjgs 1/1 Running 0 8m37s
pod/v9z8sipfya0d06619eyq112gw7xf5fd9 1/1 Running 0 8m26s
I believe I figured out the problem. Drone will not carry on until there is something on STDOUT after having run a service. In this case, the image I’m using is the official memcached image from: https://hub.docker.com/_/memcached (memcached/memcached:1.5-alpine).
I can work around the issue by forcing the container to emit something on STDOUT by adding a command section to .drone.yml inside the memcached service definition:
- name: memcached
- echo "Starting memcached on port 11211" &&
Notice the code returns an io.ReadCloser from Kubernetes and then Drone performs an io.Copy that exits when the Kubernetes client closes the log stream. Would this imply Kubernetes does not close the io.ReadCloser until some output is written? If so, that would sound like a Kubernetes client bug?
I’m not sure yet what the mechanism is behind this issue. I think the most helpful thing I can do to start with is to share some simple steps to reproduce. Here’s a repo I just created to illustrate what I’m seeing:
Hi Steve, thanks for the link to reproduce. Unfortunately I do not have any bandwidth right now to deep dive on the issue (I have more immediate priorities), however I will certainly provide as much support as possible if you are interested in debugging further and possibly authoring a patch.
Note that in the meantime you can always fallback to Drone agents.