Introduction
This article will walk you through the troubleshooting process of ECS delegate servers filling up space for accommodating logs and older versions.
Here are the few steps for debugging if you run into the issue:
1. Logrotate utility -
---------- On Debian and Ubuntu ----------
sudo apt-get update && apt-get install logrotate -y
---------- On CentOS, RHEL and Fedora ----------
yum update && yum install logrotate
The general logs which will be filling up are: /var/lib/docker/containers/[CONTAINER ID]/[CONTAINER_ID]-json.log
therefore:
Create a new logrotate file /etc/logrotate.d/docker-container and a conf file at /etc/logrotate.conf add same line to both as below:
/var/lib/docker/containers/*/*.log { rotate 3 daily compress missingok delaycompress copytruncate}
This will rotate the files daily and keep a max of 3 files in the directory
/var/lib/docker/containers/*
and run
cat /dev/null > /var/lib/docker/containers/CONTAINER_ID/CONTAINER_ID-json.log #this is to clear out the existing logfile
2. Adding logging rule in docker run:
As you will be using the docker image delegate what you need to do is:
a) stop the docker container i.e. sudo docker stop <container-id> && sudo docker rm <container-id>.
b) inside the harness-delegate-docker/ folder modify our launch script i.e. launch-harness-delegate.sh by adding following lines to it:
--log-driver json-file --log-opt max-size=100m --log-opt max-file=5
the docker run should look something like:
sudo docker run -d --restart unless-stopped --log-driver json-file --log-opt max-size=100m --log-opt max-file=5 --hostname=$(hostname -f)
This would keep a maximum of 5 files with a max size of 100mb
c) then run
cat /dev/null > /var/lib/docker/containers/CONTAINER_ID/CONTAINER_ID-json.log #this is to clear out the existing logfile
d) Finally re-run the script i.e. ./launch-harness-delegate.sh
This should ideally resolve issue with the logs filling up the delegate space. You can find further details on the Harness Delegate Logging scheme here: