Docker runner not cleaning up volumes

Drone Support
1

It seems that the docker runner does not clean up empty volumes. On our machine there are now more than a thousand volumes (ls -d /var/lib/docker/volumes/drone-* | wc -l → 1269) which have not been cleaned up. While they all seem to be empty they still take up some space.

I already looked around a bit in the code and found a check for vol.EmptyDir (which seems correspond to the temp: ... attribute). Maybe the check triggers for temp: {} and skips deleting the volume?

Our volume section looks like the following:

volumes:
- name: dockersock
  host:
    path: /var/run/docker.sock
- name: database
  temp:
    medium: memory
- name: tmp_dir
  temp: {}
- name: coverage
  temp: {}

And a docker volume inspect yields:

[
    {
        "CreatedAt": "2022-01-11T07:18:24+01:00",
        "Driver": "local",
        "Labels": {
            "io.drone": "true",
            "io.drone.build.number": "2729",
            "io.drone.created": "1641881904",
            "io.drone.expires": "1641887304",
            "io.drone.protected": "false",
            "io.drone.repo.name": "cdedb2",
            "io.drone.repo.namespace": "cdedb",
            "io.drone.repo.slug": "cdedb/cdedb2",
            "io.drone.stage.name": "default",
            "io.drone.stage.number": "1",
            "io.drone.system.host": "<...>",
            "io.drone.system.proto": "https",
            "io.drone.system.version": "2.3.1",
            "io.drone.ttl": "30m0s"
        },
        "Mountpoint": "/var/lib/docker/volumes/drone-ZmiDI8DX28VMWcJKNQYR/_data",
        "Name": "drone-ZmiDI8DX28VMWcJKNQYR",
        "Options": null,
        "Scope": "local"
    }
]
2

There are two known reasons where a volume would not be removed:

  1. the runner is force-killed while pipelines are running which prevents cleanup
  2. docker daemon is unable to delete the volume due to internal errors, in which case a bug needs to be filed with the docker project

The runner removes volumes when removing containers by setting the RemoveVolumes tag to true
https://github.com/drone-runners/drone-runner-docker/blob/master/engine/engine.go#L133

The runner further makes an explicit call to delete each volume, just in case it was not removed when the container was deleted:
https://github.com/drone-runners/drone-runner-docker/blob/master/engine/engine.go#L156

Drone makes multiple attempts to remove temporary volumes. We have seen Docker fail to remove the volume but this is very rare (maybe once every 6 months) and has been due to internal Docker issues that we (Drone) cannot control.

With that being said, we do provide a garbage collection utility that is compatible with Drone, and can be used to cleanup orphaned docker resources. Please keep in mind that, if Docker refuses to delete a volume due to an internal error, this utility may not be of use.
https://github.com/drone/drone-gc