Drone checks not found in GitHub PR

apoorva-aradhya · January 3, 2022, 8:51pm

Hi,

Drone checks seem to have disappeared from PRs in just one particular repo, although the build works just fine, it’s not updating back in the github. Anyone else have come across this issue? Will really appreciate some here, thanks in advance.

csgitharness · January 4, 2022, 12:16pm

Hello @apoorva-aradhya

Thank you for reaching out to Harness Support. This seems to be an odd occurence, we tested this on our end as well and could not reproduce it. Since this is occurring primarily for a single repo could we request you to please compare the “General Settings” for the Repos on the github end between a working and a non-working Repo and see if there are any differences between the two ?

Could you please provide some more clarity on what you mean by “it’s not updating back in the github” ? do you mean the status of the pipeline run does not reach back to github ?

Thanks,
Harness

bradrydzewski · January 4, 2022, 2:34pm

When interacting with a repository, most drone operations are read-only (get commit details, get yaml file, etc). Updating the pull request status, however, is the only write operation. The only root cause I can think of where this would stop working is if the repository owner (the individual that activated the repository) no longer has sufficient privileges (remember that Drone has to make API calls on behalf of an authorized user). This can be fixed by disabling the repository, and then having an account with sufficient repository permissions re-enable the repository, which will associate the new account with the repository.

Also note that if the API call to update the GitHub status fails, the error message is written to the logs with a warn-level. GitHub error messages are not incredibly detailed, however, I suspect it will provide some useful insight as to why the operation is failing.

github.com

harness/drone/blob/master/operator/manager/teardown.go#L204:L207

    
      
          		// this error is insufficient to fail the function,
          		// however, execution of the function should be halted
          		// to prevent a nil pointer in subsequent operations.
          		return nil
          	}
          
          
	req := &core.StatusInput{
          		Repo:  repo,
          		Build: build,
          	}
          	err = t.Status.Send(noContext, user, req)
          	if err != nil && err != scm.ErrNotSupported {
          		logger.WithError(err).
          			Warnln("manager: cannot publish status")
          	}
          	return nil
          }
          
          
// cancelDownstream is a helper function that tests for
          // downstream stages and cancels them based on the overall
          // pipeline state.

apoorva-aradhya · January 4, 2022, 6:21pm

Hi @bradrydzewski

We did disable and re-enable the repo using a user with admin privileges, it didn’t help though.
Also, was unable to find Drone in the Branch protection rule part of settings for that particular repo.

bradrydzewski · January 4, 2022, 6:31pm

hey @apoorva-aradhya in this case can you please retrieve the server log entry for updating the status and post to this thread? If the system is failing to create the status you would see a log entry that reads “manager: cannot publish status”.

edit: actually can you post all server logs associated with the build? It would be great to get a full picture of what is happening on the server, just in case there is an upstream issue I am not aware of / considering. No need for runner logs.

apoorva-aradhya · January 4, 2022, 9:55pm

Seeing a lot of these warning messages in server logs
{"build.id":888212,"build.number":122740,"error":"Validation Failed","level":"warning","msg":"manager: cannot publish status","repo.id":4803,"stage.id":1765188,"time":"2022-01-04T21:15:30Z"}

Will try to get the server logs associated with the build.

bradrydzewski · January 4, 2022, 10:47pm

I did a quick search and found this in the github documentation:

there is a limit of 1000 statuses per sha and context within a repository. Attempts to create more than 1000 statuses will result in a validation error [1]

I could see this happening when using a cron job to execute a build for the same commit over and over, or when using the API to execute a build with the same commit over. Perhaps there are other scenarios I am not considering. I do not have much context here, so perhaps you can add some color?

[1] https://developer.github.com/enterprise/2.13/v3/repos/statuses/#create-a-status
[2] https://github.com/renovatebot/renovate/issues/1779

cmckeen · January 7, 2022, 3:54pm

Hey @bradrydzewski, taking a look at the repo in question, it does not look like there are any cron jobs running, or anything in the history that looks like multiple builds on the same commit.

This is not a particularly high activity repo, in general only handful of commits a week. There is a 15 day gap between the commit where this started happening and the previous commit.

We are working on getting full server logs at the time of a build and scheduling a time next week. Some server logs that are available do not have information tying it directly to a build, so we are shooting for a time when there is low activity to grab logs without having other builds contaminating the results.

cmckeen · January 11, 2022, 3:07pm

After meeting with the team to grab applicable drone server logs, we discovered that this problem was no longer occurring, and Drone checks are now appearing normally on the repo. We recently upgraded from Drone v2.7.2 to v2.7.3, and the next PR in that repo following the upgrade showed normal checks.

I don’t see anything in Drone’s changelog that would indicate a bug resolution here, so I’m thinking that a redeploy/restart of the drone server itself resolved the problem. I’m not sure why it was only this particular repo that was affected, but I do not believe we need to pursue further troubleshooting at this point. If the problem reappears, we will capture drone server logs then, and provide another update. Thanks for your help!