Large (20-60GB) cache

cbrake · May 24, 2019, 7:49pm

Hi, I’m looking for a CI system to build Yocto images. The build directory is large and runs from 20-60GB. The build directory must be preserved between builds (cached). Will a drone volume-cache (I assume it is just a docker volume) be practical for something this large?

We currently use containers to do builds, but we just map in the build directory from the host file system into the container.

github.com

YoeDistro/yoe-distro/blob/master/envsetup.sh#L466


  unset MAP_DL_DIR
  if [ -n "$DL_DIR" ]; then
    MAP_DL_DIR="-v $DL_DIR:$DL_DIR"
  fi


  if [ -n "$SSH_AUTH_SOCK" ]; then
    SSH_AUTH_DIR=$(readlink -f $SSH_AUTH_SOCK)
  fi


  docker run --rm -it \
    -v $(pwd):$(pwd) \
    -v ~/.ssh:/home/build/.ssh \
    -v ~/.gitconfig:/home/build/.gitconfig \
    -v /stash/downloads:/stash/downloads \
    $MAP_DL_DIR \
    -v $SSH_AUTH_DIR:/ssh-agent \
    -e SSH_AUTH_SOCK=/ssh-agent \
    -e MACHINE=$MACHINE \
    --user $(id -u):$(id -g) \
    ${DOCKER_REPO} /bin/bash -c "cd $(pwd) && . envsetup.sh && $CMD $2 $3 $4 $5 $6 $7 $8"
}

Cliff

bradrydzewski · May 24, 2019, 7:57pm

I cannot speak to performance characteristics of existing plugins, since they typically tar or gzip the directory when they cache on the host machine. However, you always have the option to create your own custom plugin that is optimized for your use case. Creating a plugin is pretty simple [1]. The volume plugin, for example, is under 100 lines of bash code [2].

[1] https://docs.drone.io/plugins/examples/bash/
[2] https://github.com/Drillster/drone-volume-cache/blob/master/cacher.sh

cbrake · May 24, 2019, 9:07pm

Thanks Brad – I’ll set things up and start learning.

For this use case, copying or zipping the cache won’t work – seems like it has to be a docker volume or bind mount to the host. Ideally, I’d like to be able to bind mount a directory on the host into the container running a task.

bradrydzewski · May 24, 2019, 10:54pm

To mount host machine volumes see https://docs.drone.io/user-guide/pipeline/volumes/

Note that you need to host your own drone installation to mount volumes. For security reasons we do not allowing mounting host machine volumes in Drone Cloud. Also our servers would not support such high disk space usage for a single pipeline.

cbrake · May 28, 2019, 2:16am

its working:

github.com

YoeDistro/yoe-distro/blob/feature-drone/.drone.yml

# test 12
kind: pipeline
name: default

# TODO, move most of this to a drone plugin
# Env vars:
#   - DRONE_YOE_REPO (can we get this from drone env??)
#   - DRONE_YOE_REPO_BRANCH (can we get this from drone env??)
#   - DRONE_YOE_DOCKER_IMAGE
#   - DRONE_YOE_CACHE_DIR
#   - DRONE_YOE_IMAGE
#   - DRONE_YOE_ENVSETUP

clone:
  depth: 50

steps:
  - name: build
    image: yoedistro/yoe-build:stretch
    volumes:

This file has been truncated. show original

Still some details to work through, but I’m liking Drone a lot.

Thanks for the great support.

cbrake · November 6, 2019, 2:06pm

The exec runner (https://docs.drone.io/pipeline/exec/overview/) is also a good fit for this type of build. I think it was released after our initial discussion above.