No builds started drone 0.8 (latest) with gogs 0.11.29 hooks are send and delivered 200

Development
1

Hi trying drone 0.8 (latest) with gogs 0.11.29.

Before upgrading versions of drone from one of the 0.8 release candidates and gogs (0.11.19 probably) I had a working setup (except sometimes builds hanging). Tried switching gogs back to old container. But no improvement …

No build is started.
Drone logs suggest me that the hook request is received ok. Also the gogs webhook delivery status is ‘green’.

Drone log:

time="2017-10-17T17:47:17Z" level=info ip=x.x.x.x  latency=505.695µs method=GET path="/hook" status=200 time="2017-10-17T17:47:17Z" user-agent=GogsServer

I don’t see anything else in my drone logs, except browser and the agent reconnecting after container restart (which also didn’t help fire up the build process)

Probably I missing something that changed between the release candidates ?
What is the best way to troubleshoot this issue? I am quite new to drone as well as gogs.

Drone Debug mode is activated… but still I don’t see anything in the logs that point me in any direction

2

I run drone 0.8.1 with the latest version of gogs (set to auto-upgrade) and I can confirm there are no issues with either drone or gogs.

Drone logs suggest me that the hook request is received ok

I am not sure I agree, because the logs indicate it received a GET request, and gogs sends hooks via the POST request. There is no body attached to a GET request and as a result such a request would not include the hook payload from gogs.

time="2017-10-17T17:47:17Z" level=info ip=x.x.x.x  latency=505.695µs method=GET path="/hook"
3

good point GET/PUT …
Deleted the webhooks and reactivated the repo in drone.

Then i did a TEST webhook delivery in gogs.

Gogs tells me it uses POST:

Request URL: http://drone.../hook?access_token=xxx
Request method: POST
Content-Type: application/json
X-Gogs-Delivery: xxx
X-Gogs-Event: push
X-Gogs-Signature: xxx

Corresponding Drone logs … GET:

time="2017-10-17T20:56:54Z" level=info ip=x latency=298.553µs method=GET path="/hook" status=200 time="2017-10-17T20:56:54Z" user-agent=GogsServer

So maybe this is strange …

Response that gogs received: 200

Header:

Access-Control-Allow-Origin: *
Cache-Control: no-cache, no-store, max-age=0, must-revalidate, value
Content-Type: text/html; charset=UTF-8
Date: Tue, 17 Oct 2017 20:56:54 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Last-Modified: Tue, 17 Oct 2017 20:56:54 GMT
Server: Apache/2.4.10 (Debian)
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Drone-Version: 0.8.2+build.1334
X-Frame-Options: DENY
X-Xss-Protection: 1; mode=block

Body:

<!DOCTYPE html>
<html>
<head>
	<meta charset="utf-8">
	
	<meta name="version" content="0.8.2&#43;build.1334">

	


	


	


<link rel="shortcut icon" href="/favicon.png"></head>
<body>
<script type="text/javascript" src="/static/vendor.fc7f2b27186ea985e44c.js"></script><script type="text/javascript" src="/static/bundle.467f73292ec23d1702cb.js"></script></body>
</html>
4

do you have a reverse proxy or something in front of drone that auto-redirects or rewrites the path or method?

5

Drone and docker are running both as containers on the same host. Apache is running in front of it.
Can’t recall that I changed anything in the Apache config after I got Drone and Gogs finally work together the first time, before upgrading drone and gogs. But it doesn’t hurt to take a look at that.

It’s little frustrating to see that they both reach each other (gogs inventory synced to drone, drone can setup the webhook in gogs, gogs can send webhook to drone when I push code … but… but Drone does nothing after that and I see no messages in the logs (in debug mode) that give me a hint what is going wrong.

6

Based on the sample logs, it is not clear that hooks are being properly sent. The logs only showed GET /hook which is not sufficient to trigger a build. I can say with confidence that if you do not see POST /hook in your logs, it is an upstream problem, and not a drone problem.

The system cannot create logs for something that never happened. Unless the POST /hook endpoint is invoked, you should not expect any detailed webhook logging.

Gogs tells me it uses POST
Corresponding Drone logs … GET

If Gogs says it is sending a POST and Drone says it is receiving a GET, the most logical place to debug this would be your Apache configuration. It seems the most logical root cause, since Apache literally is a man-in-the-middle and alters the traffic between the source and destination.

7

Brad,

Tnx for your replies and hints and confirmation that it should work.
Such reactions help to find needed energy for troubleshooting missions like this.

RESTEasy plug-in in Firefox … helped me noticed that ALL methods (except POST) ended up in the Drone log correctly.
Only POST requests became a GET in drone.

Finally with curl -X POST and the webhook copied from gogs config, I got a clear hint what is wrong: responsecode 301 And a message that the Document was moved …

What Document has moved … Yes to httpS://
At that moment I notice the webhook URL… it starts with http, instead of httpS…

How that ended up in gogs is not yet clear for me.

And Gogs unfortunately didn’t give me respons 301, but response 200 which gave me the impression that everything is setup fine from gogs perspective.

I Changed webhook URL manually in gogs to https … and now I do see request coming into drone as POST requests.

You suggested to look at Apache Proxy Config. In my opinion Apache was doing Exactly what it is configured to do, namely
REDIRECTED http:// requests url to httpS://.

How POST requests after 301 from Apache get mangled into GET requests to drone is for me still a mystery.

I will try to find the energy and forward this strange behavior also to gogs developer.

Only the returned body (in gogs webhooks result) after the redirect and mangling POST into GET is maybe bit strange and might help others to find this post and help troubleshooting. I include it here.

<!DOCTYPE html>
<html>
<head>
	<meta charset="utf-8">
	<meta name="version" content="0.8.2&#43;build.1334">
<link rel="shortcut icon" href="/favicon.png"></head>
<body>
<script type="text/javascript" src="/static/vendor.fc7f2b27186ea985e44c.js"></script><script type="text/javascript" src="/static/bundle.467f73292ec23d1702cb.js"></script></body>
</html>

I should now be able to find a way to get things working again.

Thanks again for your replies and giving me the energy to continue troubleshooting.

Jean-Paul

8

You might also want to try setting the X-Forwarded-Proto parameter in your Apache configuration. Drone will use this parameter to create https hooks, which should help avoid the unnecessary redirect.

9

Tnx

Setting X-Forwarded-Proto helped to get the correct webhook in gogs

Updated my ansible playbook and now it works like a charm again.
Up to the next phase … automated testing of my ansible playbooks somewhere on a disposable cloud server to notice issues in config changes sooner.