Dynamically generate docker images

Drone Support
1

We use Drone to build and test our Python projects. To this end, Drone pulls a Debian image with Python pre-installed. What’s not installed, however, are the many pip dependencies of the Python project. Every time a build starts, Drone will re-install these which takes a considerable amount of time and wastes bandwidth.

Building the images manually with their dependencies is not really an option as they may change from time to time, and taking out human time to make new Docker images with the correct dependencies (and selecting them in drone.yml) is even more costly.

We looked into generating Docker images on the fly. This would work as follows:

  • pull the repo
  • calculate a hash from the dependency list (requirements.txt/setup.cfg/…)
  • if an image with the matching hash does not exist, pull the base image, install the dependencies and push it to the repository
  • pull the image with the matching hash
  • continue normal Drone CI jobs

However, we are worried about possible race conditions when multiple builds run concurrently and try to build the image and push to the repo at the same time.

Does anyone have experience with this or other lore that can help with this particular attempt at optimisation?

2

I would suggest this option

in the docker build step: (using plugin/docker)

  • pull the image from image:latest in cache-from
  • push the image to image:latest if the commit is in the master branch (and not a pr)
  • push the image to image:$DRONE_SHA in any case

in the next steps:
use image:$DRONE_SHA

In most cases, the tag $DRONE_SHA would just be another tag on an already existing image, so it will not take up a lot of space. Add some pruning script on the registry api.