I thought I’d share this profile snippet in case it’s useful to others after the conversation in Harness Delegate Logs to Splunk.
It sends the delegate log file to Elasticsearch service (Humio for us) using filebeat. Meant to be run on the harness/delegate Docker image.
curl -sSLO https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-oss-7.3.1-amd64.deb
dpkg -i filebeat-oss-7.3.1-amd64.deb
cat << EOF > /etc/filebeat/filebeat.yml
filebeat.inputs:
- paths:
- /opt/harness-delegate/delegate.log
encoding: utf-8
fields:
type: harness
delegate_group: '${DELEGATE_GROUP_NAME}'
multiline.pattern: '^\d{4}-\d{2}-\d{2}'
multiline.negate: true
multiline.match: after
queue.mem:
events: 8000
flush.min_events: 1000
flush.timeout: 1s
output:
elasticsearch:
hosts: ["YOUR_HOST"]
username: ...
password: ${secrets.getValue("token")}
compression_level: 5
bulk_max_size: 200
worker: 1
EOF
cat << EOF > /etc/default/filebeat
# Remove -e to quieten stderr on startup
TEST_ARGS="test config"
EOF
service filebeat restart
And for Humio users, here’s a working parser:
stripAnsiCodes(as=nonAnsi) | case {
nonAnsi = /(?<ts>[\d-]+ [\d:,]+) \[(?<version>[\d\.]+)\] (?<delegate>\d+)\s+\[(?<task>[\w-]+)\]\s+\[(?<thread>[\w-]+)\] (?<priority>[A-Z]+)\s+(?<source>[\w\.]+) - (?<msg>.*)/;
nonAnsi = /(?<ts>[\d-]+ [\d:,]+) \[(?<version>[\d\.]+)\] (?<delegate>\d+)\s+\[(?<thread>[^\]]+)\] (?<priority>[A-Z]+)\s+(?<source>[\w\.]+) - (?<msg>.*)/;
} | @timestamp := parseTimestamp("yyyy-MM-dd HH:mm:ss,SSS", field=ts, timezone="UTC") | drop([ts, nonAnsi])