Drone-vault plugin - Infinite token refresh loop in Kubernetes

Drone Support
1

I was planning to create a github issue in the drone-vault repository but it appears issues are disabled.

We had this plugin running fine until we updated images over the weekend. In the recent update, it appears the Drone-vault plugin is falling into an infinite loop and constantly requesting auth tokens from Vault. Here is a visual of logs from our Vault cluster:

vault-logs
vault-logs1295×221 9.41 KB

Those big spikes are hundreds of thousands of Auth Update requests coming from the drone-vault plugin. The gap in the middle is when it actually took down both Vault and Consul.

Our solution was to revert to a previous drone-vault image, but it appears the drone/vault Dockerhub doesn’t have any images previous to the last update. We actually had to salvage the old image from one of our Kubernetes nodes and save it off in our own ECR repository.

I haven’t dug into the code at all, but I’m wondering if it was something to do with this: drone-vault/kube.go at master · drone/drone-vault · GitHub
Which was pushed 2 weeks ago as a part of this commit: refresh kubernetes token · drone/drone-vault@fd3ba39 · GitHub
I don’t see protection around an infinite loop there, but I also don’t know the code at all.

If nothing else, I have to suggest you keep old images around in Dockerhub. If we weren’t able to salvage the old image from weeks ago then we’d be forced to build our own image from a previous state in the repository.

2

I haven’t dug into the code at all, but I’m wondering if it was something to do with this

the referenced changes impact installations that are using kubernetes authentication and have set VAULT_AUTH_TYPE=kubernetes. This change supports renewal of the kubernetes jwt token which expires every 32 days.

I don’t see protection around an infinite loop there, but I also don’t know the code at all.

this is meant to be an infinite loop. It refreshes the auth token at a specified interval based on renewal period. It sounds like you need to configure a renewal period, assuming you are using kubernetes authentication, for example:

VAULT_TOKEN_RENEWAL=24h
3

I just tried that VAULT_TOKEN_RENEWAL ENV and it fixed the issue in the new image. Thanks!

I did a search and I don’t see that ENV variable specified in any of the drone-vault guides. I recommend it gets added in the documentation somewhere as some implementations (like ours) will be unstable without it.