I recently upgraded kubernetes and as such my drone pods were redeployed.
I’m unable to retrieve a secret from vault which I was previously able to retrieve for quite a long while as you can see from the creation date on the vault secret:
VAULT_TOKEN='REDACTED' VAULT_SKIP_VERIFY=true vault read secret/data/mysql/loaf101-com
Handling connection for 8200
Key Value
--- -----
data map[name:REDACTED pass:REDACTED rootpw:REDACTED user:REDACTED]
metadata map[created_time:2020-12-08T16:26:32.190072521Z deletion_time: destroyed:false version:1]
drone plugins secret get secret/data/mysql/loaf101-com name --repo=hostkraken/loaf101.com
Handling connection for 3000
secret not found
The config for my drone-vault deployment is as follows (i’ve verified that the vault token is still valid):
apiVersion: apps/v1
kind: Deployment
[...]
spec:
containers:
- env:
- name: DRONE_SECRET
value: REDACTED
- name: VAULT_ADDR
value: https://vaultino.vault-infra.svc.cluster.local:8200
- name: VAULT_TOKEN
value: REDACTED
- name: VAULT_CACERT
value: /vault/tls/ca.crt
image: drone/vault:latest
imagePullPolicy: IfNotPresent
[...]
volumeMounts:
- mountPath: /vault/tls/ca.crt
name: vault-tls
subPath: ca.crt
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: drone-vault
serviceAccountName: drone-vault
terminationGracePeriodSeconds: 30
volumes:
- name: vault-tls
secret:
defaultMode: 420
secretName: vault-tls
The logs from the drone-vault pod show the following:
time="2021-03-15T16:36:05Z" level=debug msg="secrets: cannot find secret name: secret not found"
I’ve validated that the drone-vault pod is able to reach Vault:
/ # curl --insecure -I https://vaultino.vault-infra.svc.cluster.local:8200/ui/
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: no-store
Content-Length: 4860
Content-Security-Policy: default-src 'none'; connect-src 'self'; img-src 'self' data:; script-src 'self'; style-src 'unsafe-inline' 'self'; form-action 'none'; frame-ancestors 'none'
Content-Type: text/html; charset=utf-8
Last-Modified: Mon, 15 Mar 2021 04:23:56 GMT
Service-Worker-Allowed: /
Vary: Accept-Encoding
Date: Mon, 15 Mar 2021 16:46:04 GMT
What could have possibly broken that would prevent drone from being able to retrieve secrets?