All my builds are freezing and then timing out. I was on drone 0.5 and upgraded to 0.8.1, but i’m still having the same issue. Any ideas?
ERROR: terminal inactive for 15m0s, build cancelled
The code that fails a build for terminal inactivity was removed as of 0.6, so this would not be possible in 0.8.1
Have you looked at your server and agent logs? Do they provide any useful debugging information? I am happy provide some high-level support, however, it will require more technical details to proceed.
Yeah, i’m not getting that error anymore. But the builds are still stalling.
In dind i see some errors like
time="2017-10-10T16:46:45Z" level=warning msg="Running modprobe xt_conntrack failed with message: `modprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1"
Other than that i don’t see any error logs
While this may be an accurate high-level description of the issue, this does not provide enough low-level detail for me to understand the problem and where it occurs in the system. What is your yaml? How far does it get in the build process? What do you see in the UI when it stalls? At which step does it stall? What do the agent logs (eg docker logs agent) show at the time it stall?
Other than that i don’t see any error logs
The agent provide very detailed debug logging. Have you enabled debug logging?
It is failing during the publish action, usually while installing apk, rubygems, or npm dependencies. The build starts and moves along nicely, then will randomly stall while installing one package or another. In most cases, it’s never the same gem, or package. Mostly around the same area, but sometimes it get’s a bit further than a previous build when manually restarting the drone build.
This was working for me for several months and this issue just started popping up across most of my builds.
Maybe your logs are exceeding the 2mb limit (reduced from 5mb) and being truncated. In this case it might look like the build is stalled when in fact it is still running.
Would that have caused the inactive terminal error on 0.5?
Nope. The only thing that would cause an inactive terminal would be the following:
- the docker logs command is blocking (this would be a docker daemon bug)
- something in your build script is blocking
I would not typically triage either of the above issues as drone issues. This is because once drone starts, it has no control over your build. It is simply waiting for the docker container to exit and collecting the logs.
# start your container
docker run -d <your-build-image> /bin/sh -c <your-build-commands>
# wait for your container to finish
docker wait <container>
# collect the container logs
docker logs <container>
The fact that a build abruptly stops while downloading dependencies sounds more like a network issue to me. Maybe the connection is blocking or timing out (a tcp connection can take 15 minutes to timeout)