We use Drone to build and test our Python projects. To this end, Drone pulls a Debian image with Python pre-installed. What’s not installed, however, are the many pip dependencies of the Python project. Every time a build starts, Drone will re-install these which takes a considerable amount of time and wastes bandwidth.
Building the images manually with their dependencies is not really an option as they may change from time to time, and taking out human time to make new Docker images with the correct dependencies (and selecting them in drone.yml) is even more costly.
We looked into generating Docker images on the fly. This would work as follows:
- pull the repo
- calculate a hash from the dependency list (requirements.txt/setup.cfg/…)
- if an image with the matching hash does not exist, pull the base image, install the dependencies and push it to the repository
- pull the image with the matching hash
- continue normal Drone CI jobs
However, we are worried about possible race conditions when multiple builds run concurrently and try to build the image and push to the repo at the same time.
Does anyone have experience with this or other lore that can help with this particular attempt at optimisation?