How to effectively use cached layers for docker build steps?

How can we properly tag and re-use base images in multi-stage Dockerfiles to

1. utilizing docker layer-caching
2. tagging base image with appropriate tag (:latest, v1.2.3-rc.X)
3. re-use this tag for actual apps being built

Let’s consider a simple example, like a multi-stage Dockerfile with some base step for dependencies which finally builds two different applications:

FROM python:3.8-slim as build-base


FROM python:3.8-slim as base

COPY --from=build-base ...dependencies...

FROM build-base as app1

...config for app1...

FROM build-base as app2

...config for app2...

If we would introduce a drone.yml now like this:

x-docker: &docker
  image: plugins/docker
    dockerfile: multistage.Dockerfile

  - name: base
    <<: *docker
      target: base

  - name: app1
    <<: *docker
      target: app1

  - name: app2
    <<: *docker
      target: app2

…the drone build steps will execute all of the multistage.Dockerfile at once, not using any layer caching and thus producing a lot of mess.

If we add cache_from: we effectively utilize layer caching, but we created new problems:

A. The base image would always be built and pushed against :latest, thus, if we build some alpha stuff on dev.., some release on master, and some point release on release/* branches we might accidentally use a dev... base image on master build if dev... pushed first and master downloads at from registry at this very moment.

B. Drone executes /usr/local/bin/docker system prune -f at the end of a plugin/docker build step, thus app1 and app2 will both download base individually. One can imagine, if we would have 10, 20, … build steps that this totally gets out of hand w.r.t. network traffic.

For solving A we would, at drone execution time, need something to tell us how base was tagged and pushed in the beginning. This works if we only use auto_tag: true and thus stick with :latest or semantic versioning scheme since we can utilize ${DRONE_TAG} then…but what if we have additional triggers for, let’s say, feature branches, pull requests, … whatever. Even a test tag like :test-123 would be problematic already.

For B I do not see any simple fix other than coming up with an own plugin/docker image or migrating to using commands and declaring all docker commands yourself.

So…any advice for nice CI pipelines that accomplish 1. to 3. while not falling into problems A or B?

The docker plugin uses docker-in-docker which means layer caching happens inside docker (and is therefore destroyed when the pipeline is complete). The docker plugin is explicitly designed to use docker-in-docker to ensure full isolation from the host. See my comment here:

If you want to use caching with the docker plugin, you should use cache-from as described in this blog post

If the ephemeral, isolated nature of the docker plugin does not meet your exact needs, you have the option to create your own custom plugin or interact directly with the host machine docker daemon as shown in this example