[solved] Slow log ingestion issues

Drone Support
1

Hi, everyone
Would it be possible for someone to give me some pointers about how to correctly configure Drone CI for better log ingestion speed? I have tried the basic set up using docker-compose and it works but I am getting very slow log ingestion when running pipelines.

The log ingestion is so slow that it takes longer than the time it takes to run the pipeline. One particular step takes 7 minutes to complete when the logs output is muted but 30 minutes when the logs are piped to stdout.
Imgur

I am using Caddy as a reverse proxy and TLS terminator inside a EC2 instance at AWS, using docker-compose.

Drone version: 1.9 from docker hub
Caddy version: v2.2.1
Ubuntu 20.04 x86_64 x86_64
Docker version 19.03.8, build afacb8b7f0
docker-compose version 1.25.0
systemd 245 (245.4-4ubuntu3.3)

I have tried running drone directly exposed to the Internet without using a reverse proxy, I have also tried running drone using sqlite and postgres but the behavior was the same.

Thanks in advance

2

In order to receive support with configuration, can you please provide your full docker-compose file so we can see more details about how you have installed Drone? For example, it is not clear which runner you are using and which version.

3

On server-side:

version: "3"

services:
  caddy:
    build:
      context: ./caddy
    restart: unless-stopped
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./caddy-data:/data
    networks:
      external_network:
      internal_network:

   postgres:
    image: postgres:13
    restart: unless-stopped
    ports:
      - 5432
    volumes:
      - ./pg-data:/var/lib/postgresql/data
    environment:
      - POSTGRES_USER=secret
      - POSTGRES_PASSWORD=secret
    networks:
      internal_network:

  drone:
    image: drone/drone:1
    restart: unless-stopped
    ports:
      - 80
      - 443
    environment:
      - DRONE_DATABASE_DRIVER=postgres
      - DRONE_DATABASE_DATASOURCE=postgres://user:password@postgres:5432/drone?sslmode=disable
      - DRONE_REGISTRATION_CLOSED=true
      - DRONE_USER_FILTER=user
      - DRONE_GITHUB_CLIENT_ID=secret
      - DRONE_GITHUB_CLIENT_SECRET=secret
      - DRONE_RPC_SECRET=secret
      - DRONE_SERVER_HOST=drone.domain
      - DRONE_SERVER_PROTO=https
      - DRONE_USER_CREATE=username:user,admin:true
      - TZ=America/Sao_Paulo
    volumes:
      - ./drone-data:/data
    networks:
      external_network:
      internal_network:

networks:
  external_network:
  internal_network:
    internal: true

Using this Caddyfile

drone.domain {
    reverse_proxy drone
}

On the runner side, it is docker Runner:

version: "3"

services:
  drone-agent:
    image: drone/agent:1
    command: agent
    restart: unless-stopped
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "./drone-agent/:/data"
    environment:
      - "DRONE_RPC_SERVER=https://drone.domain"
      - "DRONE_RPC_SECRET=secret"
      - "DRONE_RUNNER_CAPACITY=1"
4

drone/agent was deprecated in favor of drone/drone-runner-docker. The drone/agent had some issues with large log streams that were fixed in drone/drone-runner-docker. For installation instructions see docs.drone.io/runner/docker/overview.

1 Like
5

Thanks Ash, although this was embarrassing, I guess I have followed an old guide.
That was it, logs are blazing fast now!