Delegate shows as disconnected although delegate process is up and running

Running Delegates send heartbeats to the Harness Manager in 1 minute intervals. Sometime due to network issue we might see delegate is showing disconnected in Harness although delegate process is running and in the logs you can see below error:

Watcher log:

2023-03-09 14:16:22,325 [1.0.78300-000] 2016 [watcher-task-5] ERROR - Error [SocketTimeoutException: timeout], on attempt 1 timeout

Delegate log:

2023-03-09 14:17:12,606 [1.0.78500-000] 2210 [healthMonitor-1] WARN  io.harness.delegate.service.DelegateAgentServiceImpl - Timed out checking for profile java.util.concurrent.TimeoutException

2023-03-09 14:17:12,606 [1.0.78500-000] 2210 [healthMonitor-4] ERROR - Error [InterruptedIOException], on attempt 1 null

So if we see these errors that point towards network issue while connecting to Harness Manager and in these scenario watcher will try to restart the delegate and you will see heartbeatExpired:[true] in the logs:

2023-03-09 14:30:12,446 [1.0.78500-000] 2210 [healthMonitor-5] ERROR io.harness.delegate.service.DelegateAgentServiceImpl - Restarting delegate - variable values: restartNeeded:[false], frozen: [false], freezeIntervalExpired: [true],  heartbeatExpired:[true], lastHeartbeatReceivedAt:[1678371303463], lastHeartbeatSentAt:[1678372203311] 

So if this was temporary issue then delegate might connect back or else you will keep on seeing same socket connection error .

Further Reading:

1 Like