This morning my kube-runner pipelines started failing where there is a Docker image build step with errors like:
Step 7/13 : WORKDIR /app
---> Running in efb08b109c2a
Removing intermediate container efb08b109c2a
---> 2747b41f6275
Step 8/13 : RUN echo "${NPMRC}" | base64 -d > ~/.npmrc && yarn install && rm -f ~/.npmrc
---> Running in 91d2bd476bce
io.containerd.runc.v2: failed to adjust OOM score for shim: set shim OOM score: write /proc/348/oom_score_adj: invalid argument
: exit status 1: unknown
time="2020-12-14T16:29:54Z" level=fatal msg="exit status 1"
The pipeline uses a service for Docker-in-Docker using the image docker:dind. This tag was recently updated to use the newly released Docker 20.10.0 version. I’m not sure what exactly the issue is here, I tried adding the SYS_RESOURCE capability via PodSecurityPolicy but I’ve been unable to remedy it that way. Using the last major dind tag, 19.03.14-dind works normally so I guess 20.10.0 is doing something new with OOM adjustments.
The valid range for oom_score_adj is between -1000 and 1000. By default Kubernetes uses 1000 for BestEffort. What’s happening is the logic in https://github.com/containerd/containerd/blob/master/runtime/v2/shim/util_unix.go#L62 in this set-up sets the value to 1001 resulting in the invalid argument . (NOTE: This is not reproducible with Docker Desktop where it uses -500 for BestEffort!)
You can try and set the values between the highest and lowest range according to you and re-try:
# set the highest priority
echo 1000 > /proc/[PID]/oom_score_adj
# set the lowest priority
echo -1000 > /proc/[PID]/oom_score_adj
Also Please confirm if you run a docker build command on the same docker version outside drone, does it give the same result?
@mattlqx this is a known regression in the latest major version of docker:
The solution is to use a tagged version of docker that works with your cluster. We have tagged releases of the docker plugin for docker version 18, 19, etc. I recommend using the latest 19.x release: