Drone on k8s with k8s secrets plugin

Drone Support
1

Hello, drone is awesome.

I’m attempting to use drone on k8s, and use k8s secrets in my build.

I have used this yaml to deploy drone and drone/kubernetes-secrets:

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: drone-data
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: drone
  labels:
    app: drone
spec:
  replicas: 1
  selector:
    matchLabels:
      app: drone
  template:
    metadata:
      labels:
        app: drone
    spec:
      containers:
      - name: drone
        image: drone/drone:1.0.0-rc.4
        livenessProbe:
          httpGet:
            path: /healthz
            port: 80
          initialDelaySeconds: 3
          periodSeconds: 3
        env:
        - name: DRONE_LOGS_DEBUG
          value: "true"
        - name: DRONE_KUBERNETES_ENABLED
          value: "true"
        - name: DRONE_KUBERNETES_NAMESPACE
          value: default
        - name: DRONE_GITEA_SERVER
          value: https://git.example.com
        - name: DRONE_RPC_SECRET
          valueFrom:
            secretKeyRef:
              name: drone-secret
              key: rpc
        - name: DRONE_SERVER_HOST
          value: drone.example.com
        - name: DRONE_SERVER_PROTO
          value: https
        - name: DRONE_SECRET_SECRET
          valueFrom:
            secretKeyRef:
              name: drone-secret
              key: secret
        - name: DRONE_SECRET_ENDPOINT
          value: http://localhost:3000
        ports:
        - containerPort: 80
          name: drone
        volumeMounts:
        - mountPath: /data
          name: drone-data
      - name: drone-secrets
        image: drone/kubernetes-secrets
        env:
        - name: DEBUG
          value: "true"
        - name: SECRET_KEY
          valueFrom:
            secretKeyRef:
              name: drone-secret
              key: secret
      volumes:
      - name: drone-data
        persistentVolumeClaim:
          claimName: drone-data
---
kind: Service
apiVersion: v1
metadata:
  name: drone
spec:
  selector:
    app: drone
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
  name: drone
spec:
  rules:
  - host: drone.example.com
    http:
      paths:
      - backend:
          serviceName: drone
          servicePort: 80
  tls:
  - hosts:
    - drone.example.com
    secretName: drone-cert

A secret I want to use:

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: drone-example-registry
  namespace: default
data:
  password: xxxxx
  username: xxxxx

And then this .drone.yml in my build:

kind: pipeline
name: default

steps:
- name: build-image
  image: banzaicloud/drone-kaniko
  settings:
    registry: registry.example.net
    repo: registry.example.net/hello-k8s
    tags:
    - ${DRONE_COMMIT_SHA}
    - latest
    username:
      from_secret: username
    password:
      from_secret: password
  when:
    branch:
    - master

---
kind: secret

external_data:
  username:
    path: drone-example-registry
    name: username
  password:
    path: drone-example-registry
    name: password

As you can see I’ve turned on the debug logging on both containers, but the drone/kubernetes-secrets show no attempts by the other container to access it (I tested it was logging requests that I made with curl) and the drone logs don’t show any errors.

I spend a few minutes looking at the drone source and couldn’t figure out how the env var DRONE_SECRET_ENDPOINT did anything. It seemed like perhaps that code had been removed?

Thanks!

1 Like
2

Hey there, sorry about that, the Kubernetes runtime does not yet support global secrets or registry credentials, but I’m actively working on this in the latest sprint. Kubernetes was not originally planned for 1.0, but I gave into scope creep and am doing my best to bring the implementation up to speed as quickly as possible before a 1.0 final :slight_smile:

I documented some of the known gaps in Kubernetes in the latest release notes, in the Roadmap section. Once implements, the Kubernetes runtime should have parity with the Docker runtime, or will at least be very close. See 1.0.0-rc.4 release notes

In the meantime, hopefully you can use the built-in secrets as a temporary workaround until the next release is available.

edit: just wanted to follow up that I hope to complete this sprint end of next week, at which point I will tag the next release.

2 Likes
3

@bradrydzewski thank you for that quick update. I’ll be watching for the next release.

4

Met same issue, looking forward to the k8s global secret feature release

5

Hey @bradrydzewski I tried this again, but with drone/drone:1.0.0-rc.5 and it’s still not working. Any thoughts on what I might be doing wrong or ideas on where to look?

I’m guessing the k8s global registry stuff works the same way with a microservice answering requests? I’m basing that off of https://github.com/drone/drone/issues/1998

I’m a developer working on my homelab, not making any money. Are there going to be features like this locked off in the final release?

Thanks!

6

as same as @myers, I have same problem:

Am I missing something or this feature not release?

7

hey there, sorry, I am unable to reproduce any issues. I will be publishing the full source code later next week so that everyone has access to the code and can self-troubleshoot.

1 Like
8

Looking forward to it.

9

I have also deployed the drone/kubernetes-secrets plugin according to documentation and I see zero requests to it. I was expecting the plugins/docker to push the built image to the repository with credentials out of the k8s secret. Doesn’t seem to work yet in rc-5.

Docker push log tail:

unauthorized: The client does not have permission to push to the repository.
time="2019-02-19T15:44:02Z" level=fatal msg="exit status 1"

kubernetes-secrets:

$ kubectl logs -f -n drone drone-drone-server-5b576884bc-l9smf -c secrets
time="2019-02-19T15:35:49Z" level=info msg="server listening on address :3000"
10

@rafi I provided some notes [1] for troubleshooting the Vault plugin. You should be able to use a similar approach to troubleshoot the Kubernetes secret plugin. I also recommend inspecting the source code to get more details.

[1] Drone-vault plugin not quite working

11

Hi there,

I’m also having issues getting kubernetes secrets to work with Drone deployed on k8s. This is the output I get when I run drone plugins secret get:

Post : unsupported protocol scheme ""

Any ideas what might be producing the above output? I have DRONE_LOGS_TRACE enabled, but am not seeing any relevant output :frowning:

12

see Drone-cli: plugins "unsupported protocol scheme"

1 Like
13

Hi there,

I’m trying to use the Kubernetes secrets from the Drone jobs without success. I have deployed the plugin with a pod running the Docker image drone/kubernetes-secrets:latest.

The issue here is that the kubernetes-secrets is trying to get the secrets from the default namespace, and I don’t find anyway to specify the namespace where the secret object is and for which have permissions with the current ServiceAccount.

Here is the log from drone-runner-kube when the Drone job is running and request the secret:

time=“2020-05-12T13:57:21Z” level=debug msg=“secret: external: cannot get secret” error=“kubernetes api: Failure 403 secrets “my-secret” is forbidden: User “system:serviceaccount:toolkit:drone-runner” cannot get resource “secrets” in API group “” in the namespace “default”\n” kind=secret name=my-secret thread=2

Any clue of how to fix this via configuration or something?

Greetings

14

I figured it out that with the environment variable KUBERNETES_NAMESPACE in the pod of the drone/kubernetes-secrets works as spected.