Support disabling concurrent builds

Drone Support
1

The ability to disable concurrent builds is an important feature when doing continuous deployment and running automated integration tests.

Implementing this with a distributed lock plugin (which I am currently working on) has a few benefits such as the ability to acquire and release locks at any point during the pipeline.

However, a major disadvantage of this approach is that locks are held during the pipeline which means that a build agent is used to acquire the lock. For example, if there are 4 agents and 4 commits are made on master, then 1 agent would build while the 3 others would just be maintaining a lock and 0 agents would be available for other branches or other repositories.

I think the ability to disable concurrent builds should therefore be built into Drone, at the very least per branch.

2

How’s the plugin work going ? I would be able to leverage this if you have it!

I’m not sure if I’ll have time to contribute much but if you would like help, throw the git repo up here and I or others could help. :slight_smile:

3

@brettswift We have not yet made any advancement as we are still considering if Drone is the right CI/CD solution for us. The need to use agents just to hold locks is a pretty big issue for our use case. I will definitely update this thread if there is any advancement.

@bradrydzewski Do you think it would be possible to get this functionality in core? Maybe something simple that could use the conditions syntax. For example:

lock:
  when:
    branch: master

drone/drone could support this right away since there can only be a single server instance at a time, and it could use a configurable distributed lock backing store when/if Drone supports a highly available setup.

4

You do not need to lock for the duration of the pipeline. You could acquire the lock before the deployment / integration / whatever step and release after the step completes.

pipeline:
  lock:
    image: plugins/locker
    lock: true
    when:
      branch: master
      event: push

  deploy:
    image: plugins/heroku
    app: foo.com
    when:
      branch: master
      event: push

  unlock:
    image: plugins/locker
    unlock: true
    status: [ success, failure ]
    when:
      branch: master
      event: push

If your goal is to limit concurrency for entire pipeline executions (e.g. only execute 1 pipeline for master branch at a time) then yes, a plugin is not going to be optimal here. This logic would need to be implemented in our queue system, which handles distributing work to agents.

If the goal is to lock limit concurrency of specific steps, there is no practical difference between a locking plugin vs native yaml syntax. If the goal is to limit overall pipeline execution concurrency within the queue (as mentioned above) the ratelimit logic could be configured in the yaml.

5

But in order to do it without syntactic sugar, a step is required and thus a docker container and an agent to hold the lock.

If it would be part of the queue system, it would not require an agent to hold the lock. Or am I missing something in how Drone works?

6

There are two difference use cases, both with different solutions

  1. desire to prevent concurrent pipeline execution. For example, I do not want two separate pushes to master to execute pipelines at the same time. This would need to be implemented at the queue level.
  2. desire to prevent concurrent pipeline step execution. For example, you do not want multiple rsync steps running at the same time. In this case, a lock plugin is going to be optimal (example in my prior comment)
7

Gotcha. We could try to get away with #2, but definitely need #1 for some use cases.

8

Can you elaborate on what that ratelimit logic might look like? Thanks!