Webhook 500 errors

Drone Support
1

We are getting 500 errors on github webhook payloads. What could be the reason? It is not consistent. That is the issue.

Here is log https://gist.github.com/naveensrinivasan/98f533aea8657f723aceb8cbbfeb6356

Any help would be appreciated.

Thanks

2

I don’t see /hook in your logs which means either a) these are not the full logs or b) hooks are not reaching the drone server.

When hooks are not able to reach the drone server it is generally indicates a network issue. This could be an issue with your host machine networking, firewall, proxy, etc.

3

How do I debug http 500 on webhooks from github?

Thanks

4

Drone usually writes the error message to the response. You can view by clicking the response tab in the above screenshot.

In the event the response is empty, you can check the server logs.

5

I don’t see it

746×426 39 KB

6

as mentioned above, drone writes to the response for most but not all failure conditions. if nothing is in the response, check the drone server logs.

7

And I have filtered out the errors

"time=\"2017-01-11T15:28:54Z\" level=error msg=\"failure to save commit for Revature/core. meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '425-2' for key 'build_number'\" \n"
"time=\"2017-01-11T15:28:54Z\" level=error msg=\"Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '425-2' for key 'build_number'\\n\" ip=10.138.0.10 latency=1.472476484s method=POST path=\"/hook\" status=500 time=\"2017-01-11T15:28:54Z\" user-agent=\"GitHub-Hookshot/31cb34a\" \n"
"time=\"2017-01-11T15:40:36Z\" level=error msg=\"failure to save commit for Revature/core. meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '429-2' for key 'build_number'\" \n"
"time=\"2017-01-11T15:40:36Z\" level=error msg=\"Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '429-2' for key 'build_number'\\n\" ip=10.8.0.1 latency=1.73831831s method=POST path=\"/hook\" status=500 time=\"2017-01-11T15:40:36Z\" user-agent=\"GitHub-Hookshot/31cb34a\" \n"
"time=\"2017-01-11T15:41:48Z\" level=error msg=\"failure to save commit for Revature/core. meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '430-2' for key 'build_number'\" \n"
"time=\"2017-01-11T15:41:48Z\" level=error msg=\"Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '430-2' for key 'build_number'\\n\" ip=10.138.0.10 latency=1.493025308s method=POST path=\"/hook\" status=500 time=\"2017-01-11T15:41:48Z\" user-agent=\"GitHub-Hookshot/31cb34a\" \n"
"time=\"2017-01-11T15:54:37Z\" level=error msg=\"Error #01: sql: no rows in result set\\n\" ip=10.8.0.1 latency=463.042715ms method=GET path=\"/api/repos/Revature/frontend/logs/209/1\" status=404 time=\"2017-01-11T15:54:37Z\" user-agent=\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36\" \n"
"time=\"2017-01-11T15:54:47Z\" level=error msg=\"Error #01: sql: no rows in result set\\n\" ip=10.8.0.1 latency=489.376739ms method=GET path=\"/api/repos/Revature/frontend/logs/208/1\" status=404 time=\"2017-01-11T15:54:47Z\" user-agent=\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36\" \n"
"time=\"2017-01-11T17:34:42Z\" level=error msg=\"failure to save commit for Revature/core. meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '435-2' for key 'build_number'\" \n"
"time=\"2017-01-11T17:34:42Z\" level=error msg=\"Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '435-2' for key 'build_number'\\n\" ip=10.138.0.10 latency=1.310611147s method=POST path=\"/hook\" status=500 time=\"2017-01-11T17:34:42Z\" user-agent=\"GitHub-Hookshot/31cb34a\" \n"
"time=\"2017-01-11T19:12:27Z\" level=error msg=\"failure to save commit for Revature/frontend. meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '212-3' for key 'build_number'\" \n"
"time=\"2017-01-11T19:12:27Z\" level=error msg=\"Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '212-3' for key 'build_number'\\n\" ip=10.138.0.10 latency=1.387678893s method=POST path=\"/hook\" status=500 time=\"2017-01-11T19:12:27Z\" user-agent=\"GitHub-Hookshot/31cb34a\" \n"
"time=\"2017-01-11T20:19:28Z\" level=error msg=\"failure to save commit for Revature/frontend. meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '213-3' for key 'build_number'\" \n"
"time=\"2017-01-11T20:19:28Z\" level=error msg=\"Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '213-3' for key 'build_number'\\n\" ip=10.8.0.1 latency=1.39787868s method=POST path=\"/hook\" status=500 time=\"2017-01-11T20:19:28Z\" user-agent=\"GitHub-Hookshot/31cb34a\" \n"
"time=\"2017-01-11T20:20:52Z\" level=error msg=\"failure to save commit for Revature/frontend. meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '216-3' for key 'build_number'\" \n"
"time=\"2017-01-11T20:20:52Z\" level=error msg=\"Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '216-3' for key 'build_number'\\n\" ip=10.138.0.10 latency=1.41621908s method=POST path=\"/hook\" status=500 time=\"2017-01-11T20:20:52Z\" user-agent=\"GitHub-Hookshot/31cb34a\" \n"
"time=\"2017-01-11T21:42:07Z\" level=error msg=\"failure to save commit for Revature/core. meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '436-2' for key 'build_number'\" \n"
"time=\"2017-01-11T21:42:07Z\" level=error msg=\"Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '436-2' for key 'build_number'\\n\" ip=10.138.0.10 latency=1.481351984s method=POST path=\"/hook\" status=500 time=\"2017-01-11T21:42:07Z\" user-agent=\"GitHub-Hookshot/6f33334\" \n"

The most of the http 500 are because of duplicate Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry. How is this possible and the funny thing is that If I manualy redeliver the webhook it does not fail and returns with http 200.

For example here is a github webhook which is failing with id 6f33334

1152×874 68.6 KB

and here is the corresponding error for that in the logs

"time=\"2017-01-11T21:42:07Z\" level=error msg=\"Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '436-2' for key 'build_number'\\n\" ip=10.138.0.10 latency=1.481351984s method=POST path=\"/hook\" status=500 time=\"2017-01-11T21:42:07Z\" user-agent=\"GitHub-Hookshot/6f33334\" \n"

What is the issue? How do we fix this? We are using mysql as our data store

Thanks

8

And we are also seeing this

1:M 12 Jan 13:38:16.509 * stomp: ack 1345462870451872970: message not found
1:M 12 Jan 13:38:16.509 # stomp: server panic: runtime error: invalid memory address or nil pointer dereference
time="2017-01-12T13:38:16Z" level=info ip=10.8.1.1 latency=15.529250115s method=GET path="/ws/broker" status=200 time="2017-01-12T13:38:16Z" user-agent=
9

@bradrydzewski Any update on this? Thanks

1 Like