Drone web page 404: Record not found, newly installed drone after drone server up for a while (couple hours)

kaijajan · September 12, 2019, 6:49am

Thanks to all drone developers.

I’m new in drone, and have my drone installed this week.
I encountered a strange issue and have no clue to make it right.
After installation everything went well at first.
Drone web server can be access on any links on it, and drone runner executed on every commit.
However, when I try to use it again after couple hrs of use.

I tried two version of drone, and agent. (latest, v1) All comes the same result.
I’m able to recover the server by restarting docker container, but the issue would hanpped again repeatly after couple hours of use.

Please help to figure the issue. Thanks

Here’s my setup:
OS: Linux Mint 19
gitea/gitea:latest
drone/drone:1
drone/agent:1

drone setup

docker run \
  --volume=/home/kaija/share/docker/drone_test/drone:/data \
  --env=DRONE_AGENTS_ENABLED=true \
  --env=DRONE_GITEA_SERVER=http://192.168.31.11:10080/ \
  --env=DRONE_GITEA_CLIENT_ID=c4505df4-aa91-49a1-9ba8-28f07b908a27 \
  --env=DRONE_GITEA_CLIENT_SECRET=ZQ4IGLsWe9Mv6-rVa3rbOO2oi33dzm2zEw3tMRImJBc= \
  --env=DRONE_RPC_SECRET=7e06f8dd3f7c0a957863b74fa98745d7 \
  --env=DRONE_SERVER_HOST=192.168.31.11:10180 \
  --env=DRONE_SERVER_PROTO=http \
  --env=DRONE_LOGS_DEBUG=true \
  --env=DRONE_LOGS_TEXT=true \
  --env=DRONE_LOGS_PRETTY=true \
  --env=DRONE_LOGS_COLOR=true \
  --publish=10180:80 \
  --publish=10443:443 \
  --detach=true \
  --name=drone \
  drone/drone:1

docker runner setup

docker run -d \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -e DRONE_RPC_PROTO=http \
  -e DRONE_RPC_HOST=192.168.31.11:10180 \
  -e DRONE_RPC_SECRET=7e06f8dd3f7c0a957863b74fa98745d7 \
  -e DRONE_RUNNER_CAPACITY=2 \
  -e DRONE_RUNNER_NAME=${HOSTNAME} \
  -e DRONE_DEBUG=true \
  -p 3000:3000 \
  --name runner \
  drone/agent:1

drone logs

DEBU[8114] events: stream cancelled                      request-id=1QizEHjpXYg7wTSMkn0zvi5Ns1o kaija.login=kaija1
DEBU[8114] events: stream closed                         request-id=1QizEHjpXYg7wTSMkn0zvi5Ns1o kaija.login=kaija1
DEBU[8114]                                               fields.time="2019-09-12T06:17:22Z" latency=15m50.711325145s method=GET remote="192.168.31.10:60984" request=/api/stream request-id=1QizEHjpXYg7wTSMkn0zvi5Ns1o
DEBU[8120]                                               fields.time="2019-09-12T06:17:28Z" latency="137.212µs" method=GET remote="192.168.31.10:52307" request=/ request-id=1Qj1AYEOquAaOsODgX12vJ8gZTx
DEBU[8120]                                               fields.time="2019-09-12T06:17:29Z" latency="157.195µs" method=GET remote="192.168.31.10:52307" request=/api/kaija request-id=1Qj1AhLTAyNlablFQVD12unJRSm
DEBU[8120]                                               fields.time="2019-09-12T06:17:29Z" latency="514.193µs" method=GET remote="192.168.31.10:52307" request="/api/kaija/repos?latest=true" request-id=1Qj1AigPK4mhp9BoMyQN3cFqsHn
DEBU[8120]                                               fields.time="2019-09-12T06:17:29Z" latency="596.452µs" method=GET remote="192.168.31.10:52307" request=/api/kaija/builds/recent request-id=1Qj1AmkFhKdifBGV6IRGOP8wqFP
DEBU[8120]                                               fields.time="2019-09-12T06:17:29Z" latency="512.529µs" method=GET remote="192.168.31.10:52308" request="/api/kaija/repos?latest=true" request-id=1Qj1Aknx1OZl6zW8LuzAlp6vdZM
DEBU[8121] events: stream opened                         request-id=1Qj1Ajlg6FhkVotLShHpmXZ9CCa kaija.login=kaija1
DEBU[8123] api: sync repository permissions              admin=true name=ats2 namespace=kaija1 read=true request-id=1Qj1AyuALmRHYJdjxRPjN4OdgHl kaija.login=kaija1 write=true
WARN[8123] api: cannot sync repository permissions       admin=true error="invalid character '<' looking for beginning of value" name=ats2 namespace=kaija1 read=true request-id=1Qj1AyuALmRHYJdjxRPjN4OdgHl kaija.login=kaija1 write=true
DEBU[8123]                                               fields.time="2019-09-12T06:17:31Z" latency=2.790022ms method=GET remote="192.168.31.10:52308" request=/api/repos/kaija1/ats2 request-id=1Qj1AyuALmRHYJdjxRPjN4OdgHl
DEBU[8123] api: sync repository permissions              admin=true name=ats2 namespace=kaija read=true request-id=1Qj1AyBW5fQQpfkbOz5rQvHgEoP user.login=kaija write=true
WARN[8123] api: cannot sync repository permissions       admin=true error="invalid character '<' looking for beginning of value" name=ats2 namespace=kaija read=true request-id=1Qj1AyBW5fQQpfkbOz5rQvHgEoP user.login=kaija write=true
DEBU[8123]                                               fields.time="2019-09-12T06:17:31Z" latency=2.092096ms method=GET remote="192.168.31.10:52308" request="/api/repos/kaija/ats2/builds?page=1" request-id=1Qj1AyBW5fQQpfkbOz5rQvHgEoP
DEBU[8130] manager: context canceled                     arch=amd64 kernel= kind=pipeline os=linux type=docker variant=
DEBU[8130] manager: context canceled                     arch=amd64 kernel= kind=pipeline os=linux type=docker variant=
DEBU[8131] manager: request queue item                   arch=amd64 kernel= kind=pipeline os=linux type=docker variant=
DEBU[8131] manager: request queue item                   arch=amd64 kernel= kind=pipeline os=linux type=docker variant=

no new runner logs since the drone is unable to use

web 404

dev tool issue

I also observed drone web page using dev tools,
the following two links are 404 not found when the issue occurrs.

http://192.168.31.11:10180/api/repos/kaija/ats2
http://192.168.31.11:10180/api/repos/kaija/ats2/builds?page=1

kaijajan · September 15, 2019, 1:24pm

Anyone has any ideas of this issue?
I’ve tried reinstall drone/agent too many times and all came out the same situation.
Can anyone help this please?

tboerger · September 18, 2019, 9:04am

This log line indicates to me that Drone is not able to talk properly to the Gitea API. Maybe it’s trying to redirect the request to HTTPS?

kaijajan · September 18, 2019, 9:41am

I didn’t enable HTTPS for gitea

Drone is able to talk to Gitea API(HTTP) at the first few hrs. The issue does not occur at beginning.
It’s very odd drone became unavailable for some reason.
And drone service could be recovered by restarting drone docker container. (only drone is ok)

Anyway, I will try only HTTPS again.

tboerger · September 18, 2019, 9:53am

I can only tell you what I can see. This is a typical log message if the API is not available as configured by the admin. Generally I would always suggest to use a reverse proxy and proper domain names.

kaijajan · September 19, 2019, 8:30am

I tried use HTTPS and got the same result.
Also I tried put gitea and drone in separate server, same result.

If use a reverse proxy or dns is required, I think this is really weird.

I will try previous version with gitea.
Perhaps will try Gitlab.

BTW, I use a very simple golang git repo with drone.yml for testing

kaijajan · September 20, 2019, 7:45am

I found the cause.

Drone sends incorrect access_token url to Gitea which is unrecognized by Gitea after hrs of serving.
Also Drone send access_token using HTTP GET method which is denied by Gitea sometimes. (should be POST)

/login/oauth/access_token (the correct url and use HTTP POST method)
//login/oauth/access_token (incorrect url with extra slash at the beginning)

normal request

[Macaron] 2019-09-20 05:47:19: Started POST /login/oauth/access_token for 172.17.0.1
[Macaron] 2019-09-20 05:47:19: Completed POST /login/oauth/access_token 200 OK in 79.17641ms

incorrect url request

[Macaron] 2019-09-20 07:04:51: Started POST //login/oauth/access_token for 172.17.0.1
[Macaron] 2019-09-20 07:04:51: Completed POST //login/oauth/access_token 404 Not Found in 1.326638ms
[Macaron] 2019-09-20 07:04:51: Started POST //login/oauth/access_token for 172.17.0.1
[Macaron] 2019-09-20 07:04:51: Completed POST //login/oauth/access_token 404 Not Found in 1.401705ms

incorrect HTTP method used

[Macaron] 2019-09-20 07:08:50: Started GET /login/oauth/access_token for 192.168.31.10
[Macaron] 2019-09-20 07:08:50: Completed GET /login/oauth/access_token 404 Not Found in 1.96634ms

The extra slash sent by drone is definitely a bug.
And unacceptable HTTP GET method may be disabled recently by Gitea dev team.

Please help to fix the above issue.

GregJPreece · September 24, 2019, 3:29am

Thanks @kaijajan, I just ran into this exact issue. Got my Drone setup activated the other day, sat down today to do more with it and got the 404 error. Have you opened a ticket for this on the issue tracker yet?

bradrydzewski · September 24, 2019, 3:37am

If the HTTP method is changing, that usually tells me Drone is being supplied with the wrong URL and is being automatically redirected. An auto-redirect changes a POST to a GET when the redirect takes place.

You can audit the code making the request to verify it is using a POST, see https://github.com/drone/go-login/blob/master/login/internal/oauth2/config.go#L106

invalid character ‘<’ looking for beginning of value"

This also points to an automatic redirect or an invalid url. The < means that the API call to Gitea is returning an html page that starts with <html.

Have you opened a ticket for this on the issue tracker yet?

The issue tracker is reserved for confirmed bugs and feature requests. This forum is the preferred location for end-user support.

GregJPreece · September 24, 2019, 4:07am

If the wrong URL for Gitea is supplied to Drone, how is it able to initially authenticate? I’ve also noticed that restarting the Docker container fixes the issue temporarily. It seems that sometimes Drone can authenticate and sometimes it cannot, but I only specify the Gitea URL once in my configuration.

I might be getting the requests out of sequence here, but might the extra slash in the request from Drone not lead to a redirect?

bradrydzewski · September 24, 2019, 4:13am

I am unable to reproduce any issues. I just set up Drone (latest) and Gitea (1.10.0+dev-318-g730065a3d) and the system is working as expected. I would encourage kaijajan to send a patch if certain this is a bug.

edit: here is the configuration I used when testing

DRONE_GITEA_CLIENT_ID=xxxxx-xxxx-xxxx-xxxx-xxxxxxxx
DRONE_GITEA_CLIENT_SECRET=xxxxxxxxx-xxxxxxxxxxxx-xxxx=
DRONE_GITEA_SERVER=https://try.gitea.io
DRONE_RPC_SECRET=xxxxxxxxxxxxxxxxxxxxxxxxxxx
DRONE_SERVER_HOST=localhost:8080
DRONE_SERVER_PROTO=http

GregJPreece · September 24, 2019, 4:19am

The error does not occur on initial setup. It appears to occur at some future point I am yet to nail down, when Drone checks in with Gitea. I’ve cloned the code repo to have a nose around as soon as I get chance.

bradrydzewski · September 24, 2019, 4:45am

since I cannot reproduce, and since this does not appear to be impacting most installations, there is just not enough data for me to conclude there is a bug with drone. If you are able to provide steps to reproduce, I would be happy to investigate further.

tboerger · September 27, 2019, 4:19pm

I can confirm that gitea is working fine in combination with drone if it’s setup correctly. I’m using that Stack daily.

Aang23 · September 27, 2019, 7:43pm

That’s a bug I do encounter on Gitlab too.

bradrydzewski · September 27, 2019, 7:52pm

There are valid reasons one may receive a 404 (detailed above).

This thread is related to problems refreshing the Gitea oauth2 token due to including a trailing / in the Gitea server address (which can be resolved by removing the trailing slash). When the system cannot refresh the token, the system is no longer authorized to access the repository hence the 404 not found. Gitlab does not use refresh tokens, so any issues with Gitlab would be unrelated to what is being discussed in this thread.

tj8000rpm · March 23, 2020, 2:35am

I have a same issue with drone 1.6.5 and gitea 1.11 environment, and found a workaround.

The issue seems to occurring when DRONE_GITEA_SERVER environment variable have trailing /.

Like this,

- DRONE_GITEA_SERVER=https://try.gitea.io/
+ DRONE_GITEA_SERVER=https://try.gitea.io

I think, it is need to write in the document page about should remove a trailing `` / ‘’ at least.

Such as this page, https://docs.drone.io/server/reference/drone-gitea-server/

bradrydzewski · March 23, 2020, 2:45am

trimming the trailing slash for gitea was merged into master [1] and is available when using the latest docker tag.

[1] https://github.com/drone/drone/commit/3e74e30845d1d0d03df80cf6babe7a2bc9a5ad6a

tj8000rpm · March 23, 2020, 7:16am

Thank you for your quick reply.
I understood this issues was solved in latest version.