we have drone setup on aws ecs cluster ( one server and 4 agents).
Due to some reason, from past one day, agents are unable to fetch aws secrets which are fetched by invoking an api-gw calling linked with a lambda function.
From drone-agent i can see the environment variable “DRONE_SECRET_ENDPOINT” is correctly defined. “https://abcfdef-api.eu-west-1.amazonaws.com/prod”. However now if i check api-gateway or from lambda function, they are not called/invoked.
Agent-logs give following message when build fails:
2021-08-02T12:25:25.852+03:00
{“error”:"{“message”:“Forbidden”}",“kind”:“secret”,“level”:“trace”,“msg”:“secret: external: cannot get secret”,“name”:“clientsecret”,“time”:“2021-08-02T09:25:25Z”}
Anyone knows how can i debug the agent, so it is able to call this “DRONE_SECRET_ENDPOINT”.
The error message would tell me that Drone is invoking the endpoint as expected, but the endpoint is returning a 403 Forbidden error message. You can see in the logs you posted here:
{“message”:“Forbidden”}
If Drone was working fine, and then all of a sudden stopped working, have you considered the possibility that something changed in Amazon that caused this to stop working? For example, a 403 Forbidden error could indicate something changed with your security settings, credentials, etc in Amazon.
Thanks @bradrydzewski yes it looks some issue when drone calls aws api-gateway which then calls lambda function. Need to check but we didn’t change any aws resource or setting
hmm … well we can see that Drone is making the API call to the endpoint and it is returning a 403 Forbidden. If nothing changed in Drone, and the AWS gateway started returning 403 Forbidden errors, it would stand to reason that something in AWS changed.
@bradrydzewski we enabled access logs on our api-gateway to which drone-agent is calling. From access logs we don’t see any invocation from drone-agent or 403 errors. I guess the above error message “{“message”:“Forbidden”}” comes from drone-agent.