Builds not canceled after agent scale-down

Drone Support
1

We run drone agents on spot instances in AWS, and scale the number of spot instances based on the day and time of day.

We’ve noticed that if a build is executing on a node that gets scaled out, these builds stall, and have to be canceled manually. Sometimes those executing a build don’t notice this, because builds so often succeed - and they check back hours later, only to find that the build is still “executing”, but really stalled out.

Is there a way for Drone to cancel these stalled out builds automatically?

2

We do not recommend using Drone with spot instances because they are volatile and can be terminated unexpectedly while pipelines are running, leaving pipelines in a suspended state. With that being said, the latest version of Drone (1.8.1) will clear out pipelines that are in a suspended state automatically, every 24 hours, I believe.

3

Thanks @ashwilliams1. Is that period of time configurable?

4

the cleanup routine is run every 24 hours by default and can be configured:

DRONE_CLEANUP_INTERVAL=24h

the default timeout period for a stuck pending job is 24 hours and can be configured:

DRONE_CLEANUP_DEADLINE_PENDING=24h

the default timeout period for a stuck running job is 24 hours and can be configured:

DRONE_CLEANUP_DEADLINE_RUNNING=24h

hope that helps!

5

Thanks @bradrydzewski, it certainly does!