Getting 403 Forbidden error while adding Promethus connector

Harness Experts
,
1

While trying to add Prometheus connector sometime you may get below error

  1. 403 Forbidden error
  2. Could not connect: java.net.UnknownHostException:

As Prometheus endpoint can run on any cluster and its not necessary that all delegate has access to connect to those endpoint, so ideally a correct delegate should be picked by Harness automatically.
But in some cases it can go to wrong delegate and to understand the reason first we need to understand the working of test command, so while testing/submit we do APM_VALIDATE_CONNECTOR_TASK which consist of two task:

  1. Http connectivity check for the prometheus-host
  2. And than we run a query http://prometheus-host:port/api/v1/query?query=up)

So In some cases we have seen although there was connectivity from many delegate to promethus endpoint but very few or lets say only one delegate has access to run the actual query.
So in this case your connection test can fail as with http validity any delegate can be selected and if we try to run the query you might get 403 error.
As there is no option to provide delegate selector in APM connector, so we need to add Task Category type APM and need to select the correct delegate.

1 Like