Running Delegates send heartbeats to the Harness Manager in 1 minute intervals. Sometime due to network issue we might see delegate is showing disconnected in Harness although delegate process is running and in the logs you can see below error:
Watcher log:
2023-03-09 14:16:22,325 [1.0.78300-000] 2016 [watcher-task-5] ERROR io.harness.network.FibonacciBackOff - Error [SocketTimeoutException: timeout], on attempt 1
java.net.SocketTimeoutException: timeout
Delegate log:
2023-03-09 14:17:12,606 [1.0.78500-000] 2210 [healthMonitor-1] WARN io.harness.delegate.service.DelegateAgentServiceImpl - Timed out checking for profile
com.google.common.util.concurrent.UncheckedTimeoutException: java.util.concurrent.TimeoutException
2023-03-09 14:17:12,606 [1.0.78500-000] 2210 [healthMonitor-4] ERROR io.harness.network.FibonacciBackOff - Error [InterruptedIOException], on attempt 1
java.io.InterruptedIOException: null
So if we see these errors that point towards network issue while connecting to Harness Manager and in these scenario watcher will try to restart the delegate and you will see heartbeatExpired:[true] in the logs:
2023-03-09 14:30:12,446 [1.0.78500-000] 2210 [healthMonitor-5] ERROR io.harness.delegate.service.DelegateAgentServiceImpl - Restarting delegate - variable values: restartNeeded:[false], frozen: [false], freezeIntervalExpired: [true], heartbeatExpired:[true], lastHeartbeatReceivedAt:[1678371303463], lastHeartbeatSentAt:[1678372203311]
So if this was temporary issue then delegate might connect back or else you will keep on seeing same socket connection error .
Further Reading: