How can we properly tag and re-use base images in drone.io multi-stage Dockerfiles to
1. utilizing docker layer-caching
2. tagging base image with appropriate tag (:latest, v1.2.3-rc.X)
3. re-use this tag for actual apps being built
Let’s consider a simple example, like a multi-stage Dockerfile with some base step for dependencies which finally builds two different applications:
FROM python:3.8-slim as build-base
...dependencies...
FROM python:3.8-slim as base
COPY --from=build-base ...dependencies...
FROM build-base as app1
...config for app1...
FROM build-base as app2
...config for app2...
If we would introduce a drone.yml now like this:
x-docker: &docker
image: plugins/docker
settings:
registry: my-registry.com
dockerfile: multistage.Dockerfile
steps:
- name: base
<<: *docker
settings:
repo: my-registry.com/myproject/base
target: base
- name: app1
<<: *docker
settings:
repo: my-registry.com/myproject/app1
target: app1
- name: app2
<<: *docker
settings:
repo: my-registry.com/myproject/app2
target: app2
…the drone build steps will execute all of the multistage.Dockerfile at once, not using any layer caching and thus producing a lot of mess.
If we add cache_from: my-registry.com/myproject/base we effectively utilize layer caching, but we created new problems:
A. The base image would always be built and pushed against :latest, thus, if we build some alpha stuff on dev.., some release on master, and some point release on release/* branches we might accidentally use a dev... base image on master build if dev... pushed first and master downloads at from registry at this very moment.
B. Drone executes /usr/local/bin/docker system prune -f at the end of a plugin/docker build step, thus app1 and app2 will both download base individually. One can imagine, if we would have 10, 20, … build steps that this totally gets out of hand w.r.t. network traffic.
For solving A we would, at drone execution time, need something to tell us how base was tagged and pushed in the beginning. This works if we only use auto_tag: true and thus stick with :latest or semantic versioning scheme since we can utilize ${DRONE_TAG} then…but what if we have additional triggers for, let’s say, feature branches, pull requests, … whatever. Even a test tag like :test-123 would be problematic already.
For B I do not see any simple fix other than coming up with an own plugin/docker image or migrating to using commands and declaring all docker commands yourself.
So…any advice for nice CI pipelines that accomplish 1. to 3. while not falling into problems A or B?