Control the concurrency of workflows and jobs

Run a single job at a time.

Overview

By default, the OpenTestFactory orchestrator allows multiple jobs within the same workflow, multiple workflow runs within the same namespace, and multiple workflow runs across different namespaces to run concurrently. This means that multiple instances of the same workflow or job can run at the same time, performing the same steps.

The OpenTestFactory orchestrator also allows you to disable concurrent execution. This can be useful for controlling your organization’s resources in situations where running multiple workflows or jobs at the same time could cause conflicts or consume more storage than expected.

For example, the ability to run workflows concurrently means that if multiple runs are started in quick successions, each run could trigger a separate workflow run, and these runs will execute concurrently.

Using concurrency in a workflow or a job

Use workflow.concurrency or job.<job_id>.concurrency to ensure that only a single workflow or job from the same concurrency group will run at a time.

concurrency:
    group: {group_name} # mandatory, string or expression
    cancel-in-progress: {true|false} # optional, boolean, defaults to `false`

A concurrency group can be any string or expression. The expression can only use opentf, inputs, and variables contexts. For more information about expressions, see “Expressions”.

When a concurrent job or workflow is queued, if another job or workflow run using the same concurrency group is in progress, the queued job or workflow run will be pending. Any pending job or workflow run in the same concurrency group will be canceled when a new job or workflow run is queued. To also cancel any currently running job or workflow run in the same concurrency group, specify cancel-in-progress: true.

Note that you cannot define same concurrency group for a parent and its children, i.e for a workflow and its jobs. In this case, the workflow will be canceled.

Note

The concurrency group name is case insensitive. For example, prod and Prod will be treated as the same concurrency group.

Note

Ordering is not guaranteed for jobs or workflow runs using concurrency groups. Jobs or workflow runs in the same concurrency group are handled in an arbitrary order.

Example: Using concurrency and the default behavior

The default behavior of the OpenTestFactory orchestrator is to allow multiple jobs or workflow runs to run concurrently. The concurrency keyword allows you to control the concurrency of workflow or job runs.

For example, you can use the concurrency keyword immediately after where trigger conditions are defined to limit the concurrency of entire workflow runs for a specific branch:

metadata:
  name: ...
concurrency:
  group: ${{ opentf.workflow }}-${{ variables.ref }}
  cancel-in-progress: true

You can also limit the concurrency of jobs within a workflow by using the concurrency keyword at the job level:

metadata:
  name: ...
jobs:
  job-1:
    runs-on: ubuntu-latest
    concurrency:
      group: example-group
      cancel-in-progress: true

Example: Concurrency groups

When the workflow or job is part of a concurrency group, and a workflow run or job starts, the orchestrator will cancel any workflow runs or jobs that are already in progress in the same concurrency group. This is useful in scenarios where you want to prevent parallel runs for a certain set of a workflows or jobs, such as the ones used for deployments to a staging environment, in order to prevent actions that could cause conflicts or consume more resources than necessary.

In this example, job-1 is part of a concurrency group named staging_environment. This means that if a new run of job-1 is triggered, any runs of the same job in the staging_environment concurrency group that are already in progress will be cancelled.

jobs:
  job-1:
    runs-on: ubuntu-latest
    concurrency:
      group: staging_environment
      cancel-in-progress: true

Alternatively, using a dynamic expression such as concurrency: ci-${{ variables.ref }} in your workflow means that the workflow or job would be part of a concurrency group named ci- followed by the reference of the branch or tag that triggered the workflow. In this example, if a new commit is pushed to the main branch while a previous run is still in progress, the previous run will be cancelled and the new one will start:

concurrency:
  group: ci-${{ variables.ref }}
  cancel-in-progress: true