We’d like to use dynamic node pool provisioning, but to do so, we need to be able to interpolate $DRONE_COMMIT into nodeSelector. This currently isn’t possible.
Not sure I entirely follow, but you can interpolate drone variables in any section of the yaml. It is a pure string find / replace.
node:
commit: ${DRONE_COMMIT}
When doing this, the server rejects the interpolation since it doesn’t match a regex, a regex which isn’t matched because it appears the commit isn’t expanded. I’ll dig further for logs to confirm but using node.group: “drone-${DRONE_COMMIT}” not only failed but didn’t propagate the error up to the UI, causing a silent fail.
You can find the relevant source code here which might help you debug further. I do not personally use kubernetes so I’m not going to be much help.
[drone-server-7cd4994d7f-j66tl] {"commit":"*****","error":"Job.batch \"drone-job-68-whrzoufutqjpxqrw2\" is invalid: spec.template.spec.nodeSelector: Invalid value: \"drone-${DRONE_BUILD_EVENT}-${DRONE_BUILD_NUMBER}\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')","event":"pull_request","level":"error","msg":"trigger: cannot enqueue build","ref":"refs/pull/*****/head","repo":"*****","time":"2019-06-12T00:55:21Z"}
Looks like it isn’t being interpolated, I’ve dug around drone-yaml/runtime quite a bit with no explanation. The pertinent yaml block goes
node.group: drone-${DRONE_BUILD_EVENT}-${DRONE_BUILD_NUMBER}
node.group: <value> is not a valid yaml syntax, although perhaps I am misunderstanding. I therefore recommend, if reporting issues with yaml or interpolation, that you post your full yaml configuration file.
edit I should also point out that the native kubernetes runtime is experimental and is not recommended for production (per the docs). So you may find yourself in a tight spot if you run into kubernetes-related issues right now since it is not our immediate focus. Just a polite heads up.
According to pipeline.go, the Node field of a Pipeline is a map of strings which is usable in this fashion. Like Kubernetes nodeSelectors (and potentially the Agent’s implementation), the left and right and side of the assignment is effectively a freeform pair of strings. It looks like interpolations aren’t happening to map types, it applies to the key and not the value, or a flag on that field is missing. To be clear, the group part of node.group is the key of the node pool keypair, where the interpolated string is the value.
you are right, there are a few values that are parsed and stored in the database (by the server) before the yaml is interpolated (by the agent or controller) which happens way later in the process. These are the following fields that are read prior to interpolation:
kind: <value>
name: <value>
platform:
os: <value>
arch: <value>
node:
<key>: <value>
Would it be possible to have interpolation of these values supported? While I know native Kubernetes support is experimental, that interpolation provides node pool auto-provisioning for free, an important feature for our use case on GKE.
there is a bit of a chicken and egg issue that we face. The yaml is parsed before the build object has been created (which is not something that can be changed) which means certain variables will be empty, like build number.
Also, if we interpolate the yaml without expected values it can create an invalid yaml file. If we try to interpolate this yaml before we have a build number:
tag: [ ${DRONE_BUILD_NUMBER}, 'latest'
the result would be an invalid yaml that cannot be parsed:
tag: [ , 'latest' ]
one option you might consider is a custom yaml plugin, which can be used to generate a custom yaml on the fly. Not sure if this could work for you, but it is a new feature that is described here https://github.com/drone/drone/issues/1021#issuecomment-417819643
other options may be using something like Starlark instead of yaml, which is planned for a future release and allows more dynamic configuration. Build number would not be available, but some other fields like commit sha would. https://github.com/drone/drone/issues/2585
Is there a way to configure following on server side instead of .drone.yml file?
node:
<key>: <value>
something like default we can set on server for .drone.yml file?
you will need to create a plugin for this. In this particular case you can create a conversion plugin to dynamically modify the yaml configuration file, and automatically add node values based on custom logic that you define. See https://github.com/drone/boilr-convert
Using node: is only working for initial job created by drone, but not for services and different steps,
resources: is being respected on both service and steps, but neither node: nor nodeSelector: works on both.
Is there some different way to configure that?
@tarunbhardwaj please see our official documentation for node selection:
https://docs.drone.io/pipeline/kubernetes/syntax/nodes/
How can I see which version this was added? I am using drone version 1.6.1
How can I see which version this was added? I am using drone version 1.6.1
It is part of the drone-kubernetes-runner. If you are running Drone on kubernetes you should be using this kubernetes runner. If you installed Drone using helm you should be using gtaylor/drone-charts.