Docker Hub Limits PLUGIN_CONFIG not working?

Drone Support
1

Going off the FAQ posted here: How to prevent DockerHub pull rate limit errors

We are definitely Scenario 2 noted there. We build everything and then use plugin/ecr to push up to ecr, but our images need to pull dependent images from docker hub.

I generated a json PLUGIN_CONFIG (as detailed in k8s docs) and am injecting that into the drone runners (everything is in k8s). Here is what the json looks like:
{"auths":{"https://index.docker.io/v1/":{"username":"xxx","password":"xxx","email":"xxx","auth":"xxx"}}}

I will post the drone config below, but here is the ecr build step. Note that it says it detected the registry credentials, but I’m still getting the rate limit message from docker.

+ /usr/local/bin/dockerd --data-root /var/lib/docker --host=unix:///var/run/docker.sock
Detected registry credentials
+ /usr/local/bin/docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 01:22:56 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b7f0
  Built:            Wed Mar 11 01:30:32 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
+ /usr/local/bin/docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.14.165-133.209.amzn2.x86_64
 Operating System: Alpine Linux v3.11 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.501GiB
 Name: drone-x5sdumyhvqwn0s0htzrh
 ID: DNQF:2RYF:UB6E:HX7S:FM6X:7BJX:PQF3:TGQ6:2QYE:2CGB:7BY4:NZG5
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

+ /usr/local/bin/docker build --rm=true -f Dockerfile -t 41a57434bc03279f11fbfecf27749a6625373e6c . --pull=true --label org.opencontainers.image.created=2021-03-19T14:08:46Z --label org.opencontainers.image.revision=41a57434bc03279f11fbfecf27749a6625373e6c --label org.opencontainers.image.source=https://github.com/coin-lion/coinlion-front.git --label org.opencontainers.image.url=https://github.com/coin-lion/coinlion-front
Sending build context to Docker daemon     32MB

Step 1/7 : FROM coinlion/node-server:v0.1.4
toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
time="2021-03-19T14:08:48Z" level=fatal msg="exit status 1"

.drone.yml:

---
kind: pipeline
type: kubernetes
name: default

platform:
  os: linux
  arch: amd64

steps:
- name: ecr-tag
  image: alpine
  commands:
  - echo -n ${DRONE_TAG##v} > .tags
  when:
    event:
      - tag

- name: ecr-sha
  image: alpine
  commands:
  - echo -n "sha-${DRONE_BRANCH}-${DRONE_COMMIT_SHA:0:8}" > .tags
  when:
    event:
      - push

- name: build-platform
  image: node:10.15.0
  commands:
  - cd app
  - yarn install
  - yarn build
  settings:
    group: build
  environment:
    COINLION_SITENAME: platform
  when:
    event:
    - push
    - tag

- name: ecr-platform
  image: plugins/ecr
  depends_on: [ ecr-sha, ecr-tag, build-platform ]
  settings:
    registry: 1234.dkr.ecr.us-east-1.amazonaws.com
    repo: 1234.dkr.ecr.us-east-1.amazonaws.com/coinlion-front
  when:
    event:
    - push
    - tag

- name: build-admin
  image: node:10.15.0
  depends_on: [ ecr-platform ]
  commands:
  - cd app
  - yarn install
  - yarn build
  settings:
    group: build
  environment:
    COINLION_SITENAME: admin
  when:
    event:
    - push
    - tag

- name: ecr-admin
  image: plugins/ecr
  depends_on: [ build-admin ]
  settings:
    registry: 1234.dkr.ecr.us-east-1.amazonaws.com
    repo: 1234.dkr.ecr.us-east-1.amazonaws.com/coinlion-admin
  when:
    event:
    - push
    - tag
2

Hello @finsterdexter,

Thanks for reaching Harness Support.
Could you please let us know where in drone yaml you are referring the secret?

Regards,
Harness Support

3

This would generally happen if PLUGIG_CONFIG is not being properly passed to the runner using the DRONE_RUNNER_ENV_FILE variable. Please provide an example of your kubernetes configuration and an env file (with the password redacted) that demonstrates how you are passing this value.

You can use the below pipeline to test for the value in your pipeline. If the value is missing or is malformed it would indicate improper configuration. Please run the below pipeline and report back your findings.

kind: pipeline
type: kubernetes
name: default

steps:
- name: dump
  image: alpine
  commands:
  - env
4

Yeah, the DRONE_RUNNER_ENV_FILE is not set up properly. I set that up, and I believe it’s working now. Thanks, I didn’t catch that env var.

5

Well, we’re still getting this in our ecr steps:

Step 1/12 : FROM alpine AS alpine
toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
time="2021-04-08T22:59:15Z" level=fatal msg="exit status 1"

I verified that the PLUGIN_CONFIG is indeed being injected into the drone runners when they spin up, via the corrected DRONE_RUNNER_ENV_FILE value. Not sure what to do from here? :cold_sweat:

Is there an issue with the ecr plugin? I assume that the plugin is running in a drone runner, and should be using the same docker env as any other drone runner but maybe that is a wrong assumption?

6

I’ve been digging into this, and I think I have it fixed. What happened is that the k8s docs say to use https://index.docker.io/v2/ instead of https://index.docker.io/v1/. The drone docs say to use v1 so I switched the config to v1 and it still wasn’t working. However, I failed to restart the drone runner. When I scaled down the drone runner and scaled it back up, my builds seem to be working correctly.

I think this is fully solved, hopefully.