Autoscaler not destroying servers

Hi all,

Firstly, thanks for the amazing work on the autoscaler and drone in general. Super valuable.

Question about the autoscaler. It does not seem to want to destroy two instances even though by all metrics these should be scheduled for deletion. Any ideas would be very appreciated.
As you can see the POOL_MIN is 0. Running/Pending jobs is 0. But the server count remains at 2.
environment:
- DRONE_POOL_MIN=0
- DRONE_POOL_MAX=5
- DRONE_SERVER_HOST=${DRONE_SERVER_HOST}
- DRONE_SERVER_PROTO=${DRONE_SERVER_PROTO}
- DRONE_SERVER_TOKEN=${DRONE_AUTOSCALER_TOKEN}
- DRONE_AGENT_TOKEN=${DRONE_RPC_SECRET}
- DRONE_HETZNERCLOUD_TOKEN=${DRONE_HETZNERCLOUD_TOKEN}
- DRONE_HETZNERCLOUD_IMAGE=ubuntu-20.04
- DRONE_INTERVAL=30s
- DRONE_TLS_AUTOCERT=false
- DRONE_HETZNERCLOUD_TYPE=cpx21
- DRONE_HOST=0.0.0.0

{"id":"yhfMaggHsrlLckxb","level":"debug","max-pool":5,"min-pool":0,"msg":"check capacity","pending-builds":0,"running-builds":0,"server-buffer":0,"server-capacity":4,"server-count":2,"time":"2021-05-04T09:55:34Z"}
{"id":"yhfMaggHsrlLckxb","level":"debug","msg":"terminate 2 servers","time":"2021-05-04T09:55:34Z"}
{"id":"yhfMaggHsrlLckxb","level":"debug","min-pool":0,"msg":"abort terminating instances to ensure minimum capacity met","servers-running":1,"servers-to-terminate":2,"time":"2021-05-04T09:55:34Z"}
{"id":"yhfMaggHsrlLckxb","level":"debug","msg":"check capacity complete","time":"2021-05-04T09:55:34Z"}

I don’t really have enough information to advise, but generally speaking, the system will not automatically destroy servers that are in an error state [1][2]. Have you use the CLI to inspect the existing servers to understand more about their state?

I also recommend auditing the source code here. We have solid test coverage in place for various scenarios in this file, but if there is a scenario or edge case that is missing, we would appreciate if you could add a test case that represents your state that demonstrates an issue.

[1] http://discuss.harness.io/t/failed-servers-not-being-killed/8494/6
[2] https://github.com/drone/autoscaler/pull/51

I will dig into it more deeply and try what you suggest. Thanks