RPC errors on the server

Our docker server and agents are deployed in a docker swarm setup
Server is on manager node, while the agents are on the worker nodes

I am mimicking this setup for the most part (esp about using endpoint mode as dnsrr) :

On the agent logs, we still see the following errors:
2018/04/27 18:04:46 grpc error: wait(): code: DeadlineExceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded

Any clue what might be causing the issue?