Parallel steps in a pipeline not executing

goelankitt · October 28, 2020, 6:59pm

I have a pipeline with 2 initial steps that need to run in parallel and then the next three steps also need to run in parallel. However, once the first 2 steps complete, none of the others steps are picked up. An example of drone file that I am using is here:

steps:   
  - name: step_1
    commands:
      - echo step_1
    depends_on:
      - clone
  - name: step_2
    commands:
      - echo step_2
    depends_on:
      - clone
  - name: step_3
    commands:
      - echo step_3
    depends_on:
      - step1
      - step2
  - name: step_4
    commands:
      - echo step_4
    depends_on:
      - step1
      - step2
  - name: step_5
    commands:
      - echo step_5
    depends_on:
      - step1
      - step2

The steps 3,4,5 never execute and are pipeline is just stuck forever. Has anyone encountered this before? I am using drone 1.9.0.

bradrydzewski · October 28, 2020, 7:17pm

step execution is handled by the runner. have you checked to ensure you are using the latest version of the runner? The latest stable version of the docker runner is drone/drone-runner-docker:1.5.3.

If the problem is not resolved by using the latest runner, we would ask you to provide a full yaml that can be used to reproduce. There is some important information missing from the yaml you provided (such as the pipeline type) and having a full yaml to reproduce can make things easier for us. Lastly, we would kindly ask for your full runner logs with trace logging enabled.

Also, if you are using the kubernetes runner, please keep in mind the kubernetes runner is in Beta and is still being actively developed and may not be suitable for production use (it seems to work better on some kubernetes distros / managed providers than others). For production use, we recommend the Docker runner.

goelankitt · October 30, 2020, 9:39am

Thanks for the quick response. We are indeed using drone-kube-runner and not the docker runner. I understand it’s beta version. I would still like to share the behavior that I am seeing and just get some feedback if I am doing something wrong.

After running more tests, I found that if there are multiple parallel steps with one step having a condition that is not fulfilled then none of the steps start executing.

The exact pipeline yaml is posted below. The issue I am seeing is that the steps jest_1, jest_2, cypress_1, cypress_2, cypress_3 never start executing. This happens when the triggering event is pull_request. Interestingly, if I allow the build step to run for the pull_request event then in that case these steps start executing. Maybe it has something to do with the fact that build step has the same depends_on configured like these other steps except it has a condition to only run when the event is tag.

kind: pipeline
type: kubernetes
name: test-build-deploy

trigger:
  event:
    - pull_request
    - tag

defaults: &defaults
  image: cypress/browsers:node13.8.0-chrome81-ff75
  failure: fast

services:
  - name: cypress-xvfb
    <<: *defaults
    commands:
      - Xvfb :99
    resources:
      requests:
        cpu: 1500
        memory: 4096Mi

steps:
  - name: submodule
    image: alpine/git
    environment:
      SSH_KEY:
        from_secret: SSH_KEY
      SSH_PUBLIC_KEY:
        from_secret: SSH_PUBLIC_KEY
    commands:
      - mkdir /root/.ssh
      - echo -n "$SSH_PUBLIC_KEY" > /root/.ssh/id_rsa.pub
      - chmod 600 /root/.ssh/id_rsa.pub
      - echo -n "$SSH_KEY" > /root/.ssh/id_rsa
      - chmod 600 /root/.ssh/id_rsa
      - touch /root/.ssh/known_hosts
      - chmod 600 /root/.ssh/known_hosts
      - ssh-keyscan -H github.com > /etc/ssh/ssh_known_hosts 2> /dev/null
      - git submodule update --init
      - git submodule update --remote --recursive
    depends_on:
      - clone

  - name: prepare
    <<: *defaults
    commands:
      - yarn install --frozen-lockfile
    depends_on:
      - clone

  - name: eslint
    <<: *defaults
    commands:
      - yarn lint:eslint .
    depends_on:
      - prepare
      - submodule

  - name: jest_1
    <<: *defaults
    commands:
      - yarn jest --testPathIgnorePatterns=app/containers --runInBand
    depends_on:
      - prepare
      - submodule

  - name: jest_2
    <<: *defaults
    commands:
      - yarn jest app/containers --runInBand
    depends_on:
      - prepare
      - submodule

  - name: cypress_1
    <<: *defaults
    commands:
      - export DISPLAY=:99
      - export PORT=3000
      - export CYPRESS_BASE_URL=http://localhost:3000
      - yarn ci:cypress_1
    resources:
      requests:
        cpu: 1500
        memory: 4096Mi
    depends_on:
      - prepare
      - submodule

  - name: cypress_2
    <<: *defaults
    commands:
      - export DISPLAY=:99
      - export PORT=3001
      - export CYPRESS_BASE_URL=http://localhost:3001
      - yarn ci:cypress_2
    resources:
      requests:
        cpu: 1500
        memory: 4096Mi
    depends_on:
      - prepare
      - submodule

  - name: cypress_3
    <<: *defaults
    commands:
      - export DISPLAY=:99
      - export PORT=3002
      - export CYPRESS_BASE_URL=http://localhost:3002
      - yarn ci:cypress_3
    resources:
      requests:
        cpu: 1000
        memory: 4096Mi
    depends_on:
      - prepare
      - submodule

  - name: build
    <<: *defaults
    commands:
      - yarn build
    when:
      event:
        include:
          - tag
    depends_on:
      - prepare
      - submodule

  - name: publish
    image: plugins/ecr
    environment:
      PLUGIN_ASSUME_ROLE: '$SOME_ROLE_HERE$'
      PLUGIN_REPO: 'something'
      PLUGIN_REGION: 'eu-west-1'
      PLUGIN_REGISTRY: 'registry'
    settings:
      auto_tag: true
    when:
      event:
        include:
          - tag
    depends_on:
      - eslint
      - jest_1
      - jest_2
      - cypress_1
      - cypress_2
      - cypress_3
      - build

  - name: release_notes
    image: ruby:2.4.2-alpine
    commands:
      - apk add --no-cache git
      - git clone --depth 1 --branch droneci $SOME_URL_HERE$ release_notes
      - cd release_notes && bundle install
      - rm -rf .git
      - ruby create_changelog_last_tag.rb
    environment:
      GITHUB_USER:
        from_secret: GITHUB_USER
      GITHUB_TOKEN:
        from_secret: GITHUB_TOKEN
      RELEASE_NOTES_NOTIFY:
        from_secret: RELEASE_NOTES_NOTIFY
    when:
      event:
        include:
          - tag
    depends_on:
      - publish

bradrydzewski · October 30, 2020, 12:23pm

can you confirm you are running drone/drone-runner-kube:v1.0.0-beta.5, the latest beta release for the kubernetes runner?

goelankitt · October 30, 2020, 12:39pm

No, I am running drone/drone-runner-kube:v1.0.0-beta.4

bradrydzewski · October 30, 2020, 12:44pm

beta.5 includes many code improvement and fixes since beta.4, so please test with this latest tagged release and let us know if the problem persists.

edit: also if the problem does persist, please enable trace logging on the runner and post the full trace logs for the pipeline execution and a sample yaml that can be used to reproduce.

goelankitt · October 30, 2020, 12:54pm

Thanks. Will upgrade and update here

goelankitt · November 3, 2020, 7:33am

After the upgrade the behavior is the same.

However, if I change the depends_on for the build step then the pipeline just endlessly keeps on running, the cypress-xvfb step which is a service is the one that keeps on running even when all the other steps have finished successfully. The correct behavior should be that the service stops automatically once all the other steps are completed. Is there a way to stop the service after the other steps are completed?

The change made to depends on for the build step, the rest of the steps remain the same.

  - name: build
    <<: *defaults
    commands:
      - yarn build
    when:
      event:
        include:
          - tag
    depends_on:
      - eslint
      - jest_1
      - jest_2
      - cypress_1
      - cypress_2
      - cypress_3

max-rocket-internet · November 20, 2020, 1:28pm

And update @bradrydzewski

bradrydzewski · November 20, 2020, 1:59pm

also if the problem does persist, please enable trace logging on the runner and post the full trace logs for the pipeline execution.

waiting to see trace logs from beta.5