Docker runner not cleaning up volumes

It seems that the docker runner does not clean up empty volumes. On our machine there are now more than a thousand volumes (ls -d /var/lib/docker/volumes/drone-* | wc -l → 1269) which have not been cleaned up. While they all seem to be empty they still take up some space.

I already looked around a bit in the code and found a check for vol.EmptyDir (which seems correspond to the temp: ... attribute). Maybe the check triggers for temp: {} and skips deleting the volume?

Our volume section looks like the following:

- name: dockersock
    path: /var/run/docker.sock
- name: database
    medium: memory
- name: tmp_dir
  temp: {}
- name: coverage
  temp: {}

And a docker volume inspect yields:

        "CreatedAt": "2022-01-11T07:18:24+01:00",
        "Driver": "local",
        "Labels": {
            "io.drone": "true",
            "": "2729",
            "io.drone.created": "1641881904",
            "io.drone.expires": "1641887304",
            "io.drone.protected": "false",
            "": "cdedb2",
            "io.drone.repo.namespace": "cdedb",
            "io.drone.repo.slug": "cdedb/cdedb2",
            "": "default",
            "io.drone.stage.number": "1",
            "": "<...>",
            "io.drone.system.proto": "https",
            "io.drone.system.version": "2.3.1",
            "io.drone.ttl": "30m0s"
        "Mountpoint": "/var/lib/docker/volumes/drone-ZmiDI8DX28VMWcJKNQYR/_data",
        "Name": "drone-ZmiDI8DX28VMWcJKNQYR",
        "Options": null,
        "Scope": "local"

There are two known reasons where a volume would not be removed:

  1. the runner is force-killed while pipelines are running which prevents cleanup
  2. docker daemon is unable to delete the volume due to internal errors, in which case a bug needs to be filed with the docker project

The runner removes volumes when removing containers by setting the RemoveVolumes tag to true

The runner further makes an explicit call to delete each volume, just in case it was not removed when the container was deleted:

Drone makes multiple attempts to remove temporary volumes. We have seen Docker fail to remove the volume but this is very rare (maybe once every 6 months) and has been due to internal Docker issues that we (Drone) cannot control.

With that being said, we do provide a garbage collection utility that is compatible with Drone, and can be used to cleanup orphaned docker resources. Please keep in mind that, if Docker refuses to delete a volume due to an internal error, this utility may not be of use.