[K8s runner] resource requests failing to work

Drone Support
1

I can confirm that resource limits work as described.
However requests seem not to.

An example:

  1. the runner deployment has following env variables set to initialize containers with default values:
DRONE_RESOURCE_LIMIT_CPU = 250
DRONE_RESOURCE_LIMIT_MEMORY = "256MiB"
DRONE_RESOURCE_REQUEST_CPU = 50
DRONE_RESOURCE_REQUEST_MEMORY = "64MiB"
  1. I have a step with overriden requests and limits:
resources:
  limits:
    cpu: 4000
    memory: 6GiB
  requests:
    cpu: 2000
    memory: 4GiB
  1. in a pipeline with 2 steps (1 default settings, 1 overriden) we can check that limits are both ok but requests both wrong:
kubectl -n tools-drone get pods drone-crc8zqp3p03tgy3t910x -o=jsonpath='{.spec.containers[0].resources}'
{"limits":{"cpu":"4","memory":"6Gi"},"requests":{"cpu":"48m","memory":"56Mi"}}

kubectl -n tools-drone get pods drone-crc8zqp3p03tgy3t910x -o=jsonpath='{.spec.containers[1].resources}'
{"limits":{"cpu":"250m","memory":"256Mi"},"requests":{"cpu":"1m","memory":"4Mi"}}

Is this really a bug or just my misconfiguration? In the bug case should I file a ticket somewhere else?

2

Hi @gzatko, from what I understand this is now expected behaviour from the recent runner changes (although not desired in our case).

We hit the same issue and my colleague documented this with a few different scenarios and settings: [drone-runner-kube] Not correctly setting resource requests for Steps - #5 by onur

3

Thanks for the thread - I overlooked it somehow in search.
For me it’s important that Docker image 1.0.0-beta.6 enabled to set request per pod.

4

Hi,

We have recently updated our runner to version 1.0.0-beta.11

We still seem to be experiencing the same issue as stated before.

There appears to be no way to set the resource requests in the drone file. When working with a drone.yaml example at: Resources | Drone

When I check the pods each container will have an equal share of

 requests:
        cpu: 100m
        memory: 100Mi

This allocation will be divided by:

no. of steps +1 (assuming for the clone step)

When using a .drone.star file we are able to set the request if the steps request is less than the default values that appear to be above.

However if we attempt to set the value to anything higher in say a 1 step drone file, it will accept the limit but the the maximum request allowance is blocked.

Here is the drone.star:

def main(ctx):
  return {
    "name": "build",
    "kind": "pipeline",
    "type": "kubernetes",
    "steps": [
      {
        # Step 2, requests are lower than a MIN request amount
        "name": "stage-1",
        "image": "alpine",
        "resources": {
            "requests": {
                "cpu": 1000,
                "memory": "2000Mib"
            },
            "limits": {
                "cpu": 1000,
                "memory": "2000Mib"
            }
        },
        "commands": [
            "sleep 60"
        ]
      }
    ]
  }

Here is the resources in the stage 1 container:

resources:
      limits:
        cpu: "1"
        memory: 2000Mi
      requests:
        cpu: 50m
        memory: 48Mi

Is this still a known problem or can I be advised if there is anything we can change in the configuration to allocate a higher request in our drone steps?

5

Resource requests are defined at pipeline level, not at the step level. See http://discuss.harness.io/t/kubernetes-runner-latest-ignoring-step-resource-requests/9392/2 and http://discuss.harness.io/t/strategy-for-pod-cpu-settings-on-kubernetes-runner/7813/4

6

Even setting at pipeline does not make a difference:

kind: pipeline
type: kubernetes

resources:
  request:
    cpu: 2000
    memory: 2000MiB

steps:
  - name: en
    image: alpine
    resources:
      limit:
        cpu: 1000
        memory: 1000MiB
    commands:
      - echo hello

  - name: es
    image: alpine
    commands:
      - echo hola

  - name: fr
    image: alpine
    commands:
      - echo bonjour

container values:

--
    resources:
      requests:
        cpu: 25m
        memory: 28Mi
    securityContext:
--
    resources:
      requests:
        cpu: 25m
        memory: 24Mi
    securityContext:
--
    resources:
      requests:
        cpu: 25m
        memory: 24Mi
    securityContext:
--
    resources:
      requests:
        cpu: 25m
        memory: 24Mi
    securityContext:

The values should be 50 and the limit should have been set in the en container according to Strategy for pod/cpu settings on kubernetes runner - #3 by trondhindenes

7

Hi @bradrydzewski , do you have any further input from my reply?

8

I think you need to use requests and limits (plural instead of singular)

I tested with the following yaml

kind: pipeline
type: kubernetes

resources:
   requests:
     cpu: 2000
     memory: 2000MiB

steps:
  - name: en
    image: alpine
    commands:
      - echo hello
    resources:
      limits:
        cpu: 1000
        memory: 1000MiB

  - name: es
    image: alpine
    commands:
    - echo hola

I then used the below command to generate a kubernetes spec file. Note that this is not meant to be a usable spec, it just shows us how data from the yaml might be mapped to Kubernetes to help debug.

drone-runner-kube compile --spec

We see that each step is assigned 668Mi of memory which is 2000MiB divided by 3 steps (2 steps defined in the yaml, as well as the implicit clone step)

    resources:
      requests:
        cpu: 667m
        memory: 668Mi
9

That works more as expected @bradrydzewski , thank you.

--
    resources:
      requests:
        cpu: 500m
        memory: 500Mi
--
    resources:
      limits:
        cpu: "1"
        memory: 1000Mi
      requests:
        cpu: 500m
        memory: 500Mi
--
    resources:
      requests:
        cpu: 500m
        memory: 500Mi

--
    resources:
      requests:
        cpu: 500m
        memory: 500Mi
10

I encountered the issue described above with limits. I have set default resources limits using environment variables. It seems that values set in each step it’s ignored. I was trying to set limits at the stage level, but I receive the following error: msg="cannot accept configuration" build.id=1152 build.number=404 error="linter: cpu limit cannot be applied at stage level".

I am running 1.0.0-beta12 with following environment variables:

    DRONE_RESOURCE_LIMIT_CPU: 500
    DRONE_RESOURCE_LIMIT_MEMORY: 512MiB

My pipeline:

kind: pipeline
name: publish
type: kubernetes

resources:
    limits:
        cpu: 2000
        memory: "1Gi"

steps:
  - name: install
    image: golang:1.16.2-buster
    resources:
      limits:
        cpu: 2000
        memory: "1Gi"
    commands:
      - go mod vendor
    depends_on:
      - clone

Removing the resources from the stage level, the pipeline manage to run but the containers have the values from below:

beniamin@macbook ~ % kubectl -n content-ci get pods drone-b8z6z7oo5qlqam9w4gc4 -o=jsonpath='{.spec.containers[0].resources}'
{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"20m","memory":"28Mi"}}%

beniamin.calota@Beniamins-MacBook ~ % kubectl -n content-ci get pods drone-b8z6z7oo5qlqam9w4gc4 -o=jsonpath='{.spec.containers[1].resources}'
{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"20m","memory":"28Mi"}}%

Is there anything I can do, besides setting a very high value from environment and overriding with whatever I need from the step?

I have to mention that the cluster administrator does not allow to run containers without limits set, so the the remove of the environment variable it’s not an option for me.

11

You need to define resource requests at the pipeline level and resource limits at the step level. This (along with using plural vs singular spelling) solved the issue for the previous individual that created this thread. You can see an example in this comment.

I also think the below section from the documentation may be relevant to your issue. The global resource limits defined by environment variable represent the maximum value and cannot be overridden in the yaml.

Note that unlike resource requests, a resource limit is set for each container individually. In addition […] limits defined in the yaml […] can’t be higher than the value set by environment variable or a policy.

Further reading about resource requests and limits:
https://docs.drone.io/runner/kubernetes/configuration/resources/

12

Thanks for the quick response. Somehow I missed that part of documentation.
Do you plan to add something like DRONE_RESOURCE_MIN_REQUEST_*, but for limits?