Panic or undefined error when create build from bitbucket repository

bakito · December 29, 2020, 4:32pm

Continuing the discussion from Panic: runtime error: index out of range [0] with length 0 with drone 1.9.1 and Bitbucket server v6.10.5:

The ticket above was not about syncing repositories from bitbucket. This error happens when creating a build from an already imported bitbucket repository.

As far as I can see, this is not related to Attempt to enable repo on older Bitbucket servers fails and
Support for Bitbucket Server (Stash) Pull Request Modified events

Correct me if I’m wrong.

When executing …

λ drone build create project/repo

drone <= 1.9.1 panics with the error below and drone 1.10.1 returns

client error 404: {"message":"bitbucket: undefined error"}

I can not see anything else in the drone logs, also not when debug and trace log enabled.

Panic: runtime error: index out of range [0] with length 0
goroutine 140 [running]:
runtime/debug.Stack(0x3b, 0x0, 0x0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
	/usr/local/go/src/runtime/debug/stack.go:16 +0x22
github.com/go-chi/chi/middleware.Recoverer.func1.1(0xc000a65500, 0x1eae6c0, 0xc00034c9a0)
	/go/pkg/mod/github.com/go-chi/chi@v3.3.3+incompatible/middleware/recoverer.go:28 +0x1b6
panic(0x19f6080, 0xc0003545a0)
	/usr/local/go/src/runtime/panic.go:969 +0x166
github.com/drone/go-scm/scm/driver/stash.(*Error).Error(0xc00033c0e0, 0x0, 0xc0008aafb8)
	/go/pkg/mod/github.com/drone/go-scm@v1.7.1-0.20200621203823-3731ec1f1136/scm/driver/stash/stash.go:133 +0x41
github.com/drone/drone/handler/api/render.ErrorCode(0x1eae6c0, 0xc00034c9a0, 0x1e75920, 0xc00033c0e0, 0x194)
	/drone/src/handler/api/render/render.go:55 +0x35
github.com/drone/drone/handler/api/render.NotFound(...)
	/drone/src/handler/api/render/render.go:79
github.com/drone/drone/handler/api/repos/builds.HandleCreate.func1(0x1eae6c0, 0xc00034c9a0, 0xc000a65d00)
	/drone/src/handler/api/repos/builds/create.go:73 +0xa95
net/http.HandlerFunc.ServeHTTP(0xc000231ae0, 0x1eae6c0, 0xc00034c9a0, 0xc000a65d00)
	/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/drone/drone/handler/api/acl.CheckAccess.func1.1(0x1eae6c0, 0xc00034c9a0, 0xc000a65c00)
	/drone/src/handler/api/acl/check.go:140 +0x91d
net/http.HandlerFunc.ServeHTTP(0xc000577290, 0x1eae6c0, 0xc00034c9a0, 0xc000a65c00)
	/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/go-chi/chi.(*ChainHandler).ServeHTTP(0xc0000e7c80, 0x1eae6c0, 0xc00034c9a0, 0xc000a65c00)
	/go/pkg/mod/github.com/go-chi/chi@v3.3.3+incompatible/chain.go:31 +0x52
github.com/go-chi/chi.(*Mux).routeHTTP(0xc0003b6300, 0x1eae6c0, 0xc00034c9a0, 0xc000a65c00)
	/go/pkg/mod/github.com/go-chi/chi@v3.3.3+incompatible/mux.go:424 +0x278
net/http.HandlerFunc.ServeHTTP(0xc00003e790, 0x1eae6c0, 0xc00034c9a0, 0xc000a65c00)
	/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/go-chi/chi.(*Mux).ServeHTTP(0xc0003b6300, 0x1eae6c0, 0xc00034c9a0, 0xc000a65c00)
	/go/pkg/mod/github.com/go-chi/chi@v3.3.3+incompatible/mux.go:69 +0x4f3
github.com/go-chi/chi.(*Mux).Mount.func1(0x1eae6c0, 0xc00034c9a0, 0xc000a65c00)
	/go/pkg/mod/github.com/go-chi/chi@v3.3.3+incompatible/mux.go:291 +0x11b
net/http.HandlerFunc.ServeHTTP(0xc00037dd00, 0x1eae6c0, 0xc00034c9a0, 0xc000a65c00)
	/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/go-chi/chi.(*Mux).routeHTTP(0xc0003b60c0, 0x1eae6c0, 0xc00034c9a0, 0xc000a65c00)
	/go/pkg/mod/github.com/go-chi/chi@v3.3.3+incompatible/mux.go:424 +0x278
net/http.HandlerFunc.ServeHTTP(0xc00003e710, 0x1eae6c0, 0xc00034c9a0, 0xc000a65c00)
	/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/drone/drone/handler/api/acl.CheckAccess.func1.1(0x1eae6c0, 0xc00034c9a0, 0xc000a65c00)
	/drone/src/handler/api/acl/check.go:99 +0x10c1
net/http.HandlerFunc.ServeHTTP(0xc000577020, 0x1eae6c0, 0xc00034c9a0, 0xc000a65c00)
	/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/drone/drone/handler/api/acl.InjectRepository.func1.1(0x1eae6c0, 0xc00034c9a0, 0xc000a65b00)
	/drone/src/handler/api/acl/repo.go:133 +0xf0a
net/http.HandlerFunc.ServeHTTP(0xc000231a40, 0x1eae6c0, 0xc00034c9a0, 0xc000a65b00)
	/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/go-chi/chi.(*Mux).ServeHTTP(0xc0003b60c0, 0x1eae6c0, 0xc00034c9a0, 0xc000a65b00)
	/go/pkg/mod/github.com/go-chi/chi@v3.3.3+incompatible/mux.go:69 +0x4f3
github.com/go-chi/chi.(*Mux).Mount.func1(0x1eae6c0, 0xc00034c9a0, 0xc000a65b00)
	/go/pkg/mod/github.com/go-chi/chi@v3.3.3+incompatible/mux.go:291 +0x11b
net/http.HandlerFunc.ServeHTTP(0xc0000c08e0, 0x1eae6c0, 0xc00034c9a0, 0xc000a65b00)
	/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/go-chi/chi.(*Mux).routeHTTP(0xc0003b6000, 0x1eae6c0, 0xc00034c9a0, 0xc000a65b00)
	/go/pkg/mod/github.com/go-chi/chi@v3.3.3+incompatible/mux.go:424 +0x278
net/http.HandlerFunc.ServeHTTP(0xc00003e6d0, 0x1eae6c0, 0xc00034c9a0, 0xc000a65b00)
	/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/go-chi/chi.(*Mux).ServeHTTP(0xc0003b6000, 0x1eae6c0, 0xc00034c9a0, 0xc000a65b00)
	/go/pkg/mod/github.com/go-chi/chi@v3.3.3+incompatible/mux.go:69 +0x4f3
github.com/go-chi/chi.(*Mux).Mount.func1(0x1eae6c0, 0xc00034c9a0, 0xc000a65b00)
	/go/pkg/mod/github.com/go-chi/chi@v3.3.3+incompatible/mux.go:291 +0x11b
net/http.HandlerFunc.ServeHTTP(0xc0000c0a20, 0x1eae6c0, 0xc00034c9a0, 0xc000a65b00)
	/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/go-chi/chi.(*Mux).routeHTTP(0xc00004bf80, 0x1eae6c0, 0xc00034c9a0, 0xc000a65b00)
	/go/pkg/mod/github.com/go-chi/chi@v3.3.3+incompatible/mux.go:424 +0x278
net/http.HandlerFunc.ServeHTTP(0xc00003eb30, 0x1eae6c0, 0xc00034c9a0, 0xc000a65b00)
	/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/go-chi/cors.(*Cors).Handler.func1(0x1eae6c0, 0xc00034c9a0, 0xc000a65b00)
	/go/pkg/mod/github.com/go-chi/cors@v1.0.0/cors.go:199 +0x183
net/http.HandlerFunc.ServeHTTP(0xc0000c0a40, 0x1eae6c0, 0xc00034c9a0, 0xc000a65b00)
	/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/drone/drone/handler/api/auth.HandleAuthentication.func1.1(0x1eae6c0, 0xc00034c9a0, 0xc000a65600)
	/drone/src/handler/api/auth/auth.go:51 +0x467
net/http.HandlerFunc.ServeHTTP(0xc0001a0c00, 0x1eae6c0, 0xc00034c9a0, 0xc000a65600)
	/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/drone/drone/logger.Middleware.func1(0x1eae6c0, 0xc00034c9a0, 0xc000a65500)
	/drone/src/logger/handler.go:36 +0x373
net/http.HandlerFunc.ServeHTTP(0xc0000c0a80, 0x1eae6c0, 0xc00034c9a0, 0xc000a65500)
	/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/go-chi/chi/middleware.NoCache.func1(0x1eae6c0, 0xc00034c9a0, 0xc000a65500)
	/go/pkg/mod/github.com/go-chi/chi@v3.3.3+incompatible/middleware/nocache.go:54 +0x2a0
net/http.HandlerFunc.ServeHTTP(0xc0000c0aa0, 0x1eae6c0, 0xc00034c9a0, 0xc000a65500)
	/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/go-chi/chi/middleware.Recoverer.func1(0x1eae6c0, 0xc00034c9a0, 0xc000a65500)
	/go/pkg/mod/github.com/go-chi/chi@v3.3.3+incompatible/middleware/recoverer.go:35 +0x83
net/http.HandlerFunc.ServeHTTP(0xc0000c0ac0, 0x1eae6c0, 0xc00034c9a0, 0xc000a65500)
	/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/go-chi/chi.(*Mux).ServeHTTP(0xc00004bf80, 0x1eae6c0, 0xc00034c9a0, 0xc000a65500)
	/go/pkg/mod/github.com/go-chi/chi@v3.3.3+incompatible/mux.go:69 +0x4f3
github.com/go-chi/chi.(*Mux).Mount.func1(0x1eae6c0, 0xc00034c9a0, 0xc000a65500)
	/go/pkg/mod/github.com/go-chi/chi@v3.3.3+incompatible/mux.go:291 +0x11b
net/http.HandlerFunc.ServeHTTP(0xc0000c1c40, 0x1eae6c0, 0xc00034c9a0, 0xc000a65500)
	/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/go-chi/chi.(*Mux).routeHTTP(0xc00004bf20, 0x1eae6c0, 0xc00034c9a0, 0xc000a65500)
	/go/pkg/mod/github.com/go-chi/chi@v3.3.3+incompatible/mux.go:424 +0x278
net/http.HandlerFunc.ServeHTTP(0xc00003e640, 0x1eae6c0, 0xc00034c9a0, 0xc000a65500)
	/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/go-chi/chi.(*Mux).ServeHTTP(0xc00004bf20, 0x1eae6c0, 0xc00034c9a0, 0xc000a65400)
	/go/pkg/mod/github.com/go-chi/chi@v3.3.3+incompatible/mux.go:81 +0x2b2
net/http.serverHandler.ServeHTTP(0xc00017c380, 0x1eae6c0, 0xc00034c9a0, 0xc000a65400)
	/usr/local/go/src/net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc000a76640, 0x1eb3680, 0xc00002c480)
	/usr/local/go/src/net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2933 +0x35c

Kind Regards,
Marc

bradrydzewski · December 29, 2020, 5:05pm

The panic was previously reported and fixed by this commit.

The undefined error is a result of Bitbucket returning a response with an empty errors section. According to the Bitbucket API documentation an error response should contain an error message, as shown in this sample payload:

    {
        "errors": [
            {
                "context": "name",
                "message": "The name should be between 1 and 255 characters.",
                "exceptionName": null
            },
            {
                "context": "email",
                "message": "The email should be a valid email address.",
                "exceptionName": null
            }
        ]
    }

Prior to being fixed, the source of the panic was that we assumed there would always be at least 1 error message in the error payload, resulting in an index out of bounds exception. In order to prevent the panic, when no error messages exist in the payload (e.g. errors.length === 0) we return an “undefined error” message. When you receive an undefined error, it means Bitbucket is not providing us any useful details as to why the http request failed.

It might be helpful to engage Bitbucket Server support, since according to the Bitbucket API documentation this could indicate an unexpected server error:

A 500 (Server Error) HTTP code indicates an incorrect resource url or an unexpected server error. Double-check the URL you are trying to access, then report the issue to your server administrator or Atlassian Support if problems persist.

You may also want to check your Bitbucket Server logs to see if they provide any useful details. If you can find a way to provide me with a sample http request and response from your Bitbucket server that reproduces the issue, we can try to use this to triage and improve the code.

bakito · December 30, 2020, 12:38pm

Hi after patching the go-scm and building a patched drone server, I got this response.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<status>
    <status-code>404</status-code>
    <message>null for uri:
        https://bitbucket.local/rest/api/1.0/projects/project/repos/repo/commits/refs/heads/master
    </message>
</status>

The request was done to this URL which does not exist.
GET rest/api/1.0/projects/project/repos/repo/commits/refs/heads/master

According to the Bitbucket spec, requesting a branch via refs/heads/master seems not to be valid.

" From 2.11, ref names like “refs/heads/master” are no longer accepted by this resource."

After this insight I could successfully trigger a new build by providing the a commit hash.

λ drone build create project/repo --commit 4ee0731b8f793e5ff134943779818c38ca0c3cd7
Number: 2
Status: pending
Event: custom
Commit: 4ee0731b8f793e5ff134943779818c38ca0c3cd7
Branch: master
Ref: refs/heads/master
Author: xxx
Message: initial commit

bakito · December 30, 2020, 3:20pm

I’ve created a PR https://github.com/drone/go-scm/pull/89 to extend the error message parsing for bitbucket server.

In the PR I also added the “Accept: application/json” header, since go-scm got the error message in xml as posted in the comment above.

bradrydzewski · December 30, 2020, 3:55pm

thanks for researching this further. It sounds like they made a change to their API which is unfortunate . Is there an alternate API endpoint we can use? I don’t have a 6.x install to test against.

bradrydzewski · December 30, 2020, 4:03pm

Actually, we may already handle a similar situation for Bitbucket Cloud, which does not have an endpoint to get the latest commit by reference. We could perhaps use the same logic for Bitbucket server with this patch [1]

	switch s.client.Driver {
-	case scm.DriverBitbucket:
+	case scm.DriverBitbucket, scm.DriverStash:
		ref = scm.TrimRef(ref)
		branch, _, err := s.client.Git.FindBranch(ctx, repo, ref) // wont work for a Tag
		if err != nil {
			return nil, err
		}
		ref = branch.Sha
	}

It sounds like you have already compiled Drone from source to test. Would you be able to test the above change and let me know if this solves the problem for you?

[1] https://github.com/drone/drone/blob/master/service/commit/commit.go#L80

bakito · December 30, 2020, 4:39pm

I tested it wit my 6.5.1 bitbucket an it works.
Already created a PR for that: https://github.com/drone/drone/pull/3050