I played around with this general topic a couple months ago and was trying to find a good balance of quick dev builds on my local workstation, but with clean (–no-cache) builds on the CI server. I find the way Docker caches each command tough to reason about sometimes, so I actually like doing important builds in a clean dind environment.
I’m not sure if it’s perfectly on topic, but I’ll add a couple things I noticed and that I was able to get working. First, I noticed that tools like skopeo (assuming crane is similar) will push images into a local registry using different hashes than the Docker daemon:
docker pull alpine
docker tag alpine example.com/alpine:docker
docker tag alpine example.com/alpine:skopeo
docker push example.com/alpine:docker
skopeo copy docker-daemon:example.com/alpine:skopeo docker://example.com/alpine:skopeo
That will get you completely different hashes on both images because the Docker daemon doesn’t expose the original hashes (skopeo issue). I’m not sure if there’s an actual concern there, but I have it noted for more research in case it causes traceability issues. I pull official images and push them to my local registry untouched and that may be the only scenario where it’s even useful to have the original hashes since FROM xxxxxxx in the build logs gives me a way to find the official image by referencing the hash.
As for building with a local cache, I tested out having two different Docker configs in the same build and switching between them at build time. I use TLS for both. For local builds on my workstation I use my workstation’s docker daemon via a --env file. For CI builds on a CI agent I use the dind-rootless image. I also use a local pull through cache for Docker, so I didn’t give much consideration to caching.
I don’t use this on anything production. It was just some playing around to see what was possible. Here’s roughly what I do for the Drone config (copied straight from my test project).
---
kind: 'pipeline'
name: 'default'
type: 'docker'
volumes:
- name: certs
temp: { }
- name: hostsock
host:
path: /var/run/docker.sock
steps:
- name: generate-certs
image: registry.example.com/library/docker:19-dind-rootless
pull: always
privileged: true
environment: &tlsconf
DOCKER_HOST: tcp://dind.localhost:2376
DOCKER_TLS_SAN: DNS:dind.localhost,DNS:docker.localhost
DOCKER_TLS_CERTDIR: /certs # server
DOCKER_TLS_VERIFY: 1 # clients
DOCKER_CERT_PATH: /certs/client # clients
volumes:
- &certs
name: certs
path: /certs
command:
- --version
# The `dind` image in the next step will regenerate certificates if the CA
# key is available. Deleting it ensures this doesn't happen.
- name: delete-ca-key
image: registry.example.com/alpine
volumes:
- <<: *certs
commands:
- rm -f /certs/ca/key.pem
# The ability to run `dind-rootless` may require extra config on the
# runner host.
- name: dind.localhost
image: registry.example.com/library/docker:19-dind-rootless
pull: always
privileged: true
detach: true
environment:
<<: *tlsconf
volumes:
- <<: *certs
command:
- --experimental
# The *.* wildcard for exclude will match all hosts / domains, so it matches
# all runners. The instance hostname isn't set when running drone exec, so
# this will run locally, but not on a CI server.
- name: docker.localhost
image: registry.example.com/nginx:alpine
privileged: true
detach: true
when:
instance:
exclude:
- "*.*"
volumes:
- <<: *certs
- name: hostsock
path: /var/run/docker.sock
commands:
- nginx -g "daemon off;" -c "$${PWD}/nginx.conf"
# Wait for the daemon to become responsive and output info which shows details
# for any errors.
- name: wait-for-docker
image: registry.example.com/docker/compose
pull: always
environment:
<<: *tlsconf
volumes:
- <<: *certs
commands:
- timeout 15s sh -c --
'while (! docker info > /dev/null 2>&1); do :; done;' || true
- docker info
- . .env.sh
- env
# Build using docker-compose
- name: build
image: registry.example.com/docker/compose
pull: always
environment:
<<: *tlsconf
volumes:
- <<: *certs
commands:
- . .env.sh
- docker-compose build --pull --no-cache
- docker image ls
# Tag the image. These tags are set by the .env.sh script.
- name: tag
image: registry.example.com/docker/compose
pull: always
environment:
<<: *tlsconf
volumes:
- <<: *certs
commands:
- . .env.sh
- docker tag "$${LOCAL_IMG}:$${BUILD_TAG}" "$${IMG}"
- docker tag "$${LOCAL_IMG}:$${BUILD_TAG}" "$${IMG}:$${TAG}"
- docker tag "$${LOCAL_IMG}:$${BUILD_TAG}" "$${IMG}:$${UNIQUE_TAG}"
- docker image ls
# Push to the local registry.
- name: push
image: registry.example.com/docker/compose
pull: always
environment:
<<: *tlsconf
NPCR_USERNAME:
from_secret: npcr_username
NPCR_PASSWORD:
from_secret: npcr_password
volumes:
- <<: *certs
when:
branch:
- master
instance:
include:
- drone.example.com
commands:
- . .env.sh
- echo "$${NPCR_PASSWORD}" | docker login
--username "$${NPCR_USERNAME}" --password-stdin "$${REGISTRY}"
- docker push "$${IMG}"
The nginx.conf I use to proxy the local daemon:
user root;
worker_processes 1;
events {
worker_connections 1024;
}
stream {
server {
listen 2376 ssl;
ssl_client_certificate /certs/server/ca.pem;
ssl_certificate /certs/server/cert.pem;
ssl_certificate_key /certs/server/key.pem;
ssl_verify_client on;
proxy_pass unix:/var/run/docker.sock;
}
}
A local .env file (.drone-local.env) that sets the daemon to the locally proxied socket:
DOCKER_HOST=tcp://docker.localhost:2376
By default, that uses dind-rootless to build on the CI agent. If I’m running locally I think I was using something like this to use the proxied local daemon instead:
drone exec --trusted --env .drone-local.env --exclude "dind.localhost"
That’s from memory, so I hope I got the syntax right. I don’t know if it’s useful, but my thinking was that maybe you could set up the Docker plugin so there’s a switch that allows choosing between using the host daemon and a clean dind environment. I know dind isn’t considered a great solution, but I like knowing the environment is completely clean and that I’m not accidentally / unintentionally using cached resources during an important build.
I’m also using docker-compose for building and end up with what I think is a pretty reasonable set of options for quick, local iteration. Running docker-compose build is very fast. Running drone exec --trusted --env .drone-local.env is still fast, but does a full build via Drone. Running drone exec --trusted does a full, clean build using dind-rootless which should be the same as what runs on the CI agent.
Another difference to the current plugin is that builds are scoped to the pipeline, not to each step. That’s one of the reasons I prefer Drone over GitLab CI. Scoping Docker builds to a pipeline makes way more sense to me than scoping to each step.
Again, I know I might be perfectly on topic, but I think the overall goal (faster builds under certain circumstances) is similar enough that it may be useful.
