Skip to content

Workflow syntax for OpenTestFactory orchestrator

Note

This workflow syntax closely matches GitHub Actions workflow syntax. If you are new to GitHub actions and want to learn more, see “Workflow syntax for GitHub Actions.”

A workflow is a configurable automated process made up of one or more jobs. You must create a YAML file to define your workflow configuration.

About YAML syntax for workflows

Workflow files must use YAML syntax, and may have a .yml or .yaml file extension. If you are new to YAML and want to learn more, see “Learn YAML in five minutes.”

You may store workflow files anywhere in your repository.

Usage limits

There may be sizing limits on OpenTestFactory orchestrator workflows. The following are the minimum that must be supported.

  • Jobs - There can be at least 1024 jobs per workflow (including generated jobs).
  • Steps - There can be at least 1024 steps per job (including generated steps).
  • Job matrix - A job matrix may generate at least 256 jobs per workflow run.

An implementation may provide lower minimums if it provides a way to extend its limits so that they reach or exceed the above minimums.

Mandatory and optional sections

A workflow must have at least a metadata section and a jobs section. It may have additional variables, resources, hooks, and defaults sections. Those additional sections may be empty or omitted if not used.

metadata:
  name: mona
variables:
  ...
resources:
  ...
hooks:
  ...
defaults:
  ...
jobs:
  ...

metadata.name

Required The name of the workflow. This name can be used by tools to provide a human-friendly reference.

metadata.namespace

The name of the namespace the workflow is part of. If unspecified, the workflow will be part of the default namespace.

variables

A map of environment variables that are available to all jobs and steps in the workflow. You can also set environment variables that are only available to a job or step. For more information, see jobs.<job_id>.variables and jobs.<job_id>.steps.variables.

When more than one environment variable is defined with the same name, the orchestrator uses the most specific environment variable. For example, an environment variable defined in a step will override job and workflow variables with the same name, while the step executes. A variable defined for a job will override a workflow variable with the same name, while the job executes.

Example

variables:
  SERVER: production

resources

A map of resources that are available to all jobs and steps in the workflow. You can define three kinds of resources: testmanagers, repositories, and files.

testmanagers and repositories resources are a way to reuse resources in your workflow. files allows to provide local or external files to your workflow.

resources.repositories

An array of source code repository definitions. Each repository definition is a map and must have the name, type, repository, and endpoint entries.

Example of a repository resource

This example defines a myrepo repository:

resources:
  repositories:
  - name: myrepo
    type: bitbucket
    repository: example/my-example-repo
    endpoint: https://bitbucket.org

resources.repositories.name

Required The name of the repository, which is used to refer to the definition. It must be unique among the repository definitions.

resources.repositories.type

Required The type of the repository. git, github, gitlab, and bitbucket must be supported.

resources.repositories.repository

Required The reference of the repository.

resources.repositories.endpoint

Required The endpoint to use to access the source code manager.

resources.files

An array of externally-provided files.

Example of a file resource

This example defines one file. This file will have to be joined when launching the workflow:

resources:
  files:
  - dataset

The following function will put a copy of the provided file in the current execution environment, so that the following steps can use it:

- uses: actions/put-file@v1
  with:
    file: dataset
    path: dataset.xml

hooks

An array of hook definitions that will apply to all jobs and steps in the workflow, including generated ones.

Each hook definition is a map. It specifies the events that trigger it, and the actions it performs when triggered. It may also contain an if conditional that further limit its scope.

Hooks specified in a workflow complement the hooks defined by installed provider plugins. Their before actions are performed before the ones defined by the provider plugins and their after actions are performed after the ones defined by the provider plugins.

before_steps from workflow-defined hook 1
  before_steps from workflow-defined hook 2
    before_steps from provider-defined hook
      steps from function
    after_steps from provider-defined hook
  after_steps from workflow-defined hooks 2
after_steps from workflow-defined hooks 1

hooks.name

Required The name of the hook, as displayed by the orchestrator.

Example

hooks:
- name: my-hook

hooks.events

Required A non-empty array of events that trigger the hook. Events are described by their categoryPrefix, category, and categoryVersion labels.

You can use the _ placeholder to match any category or categoryPrefix if you have not specified the other one. You can specify a given categoryVersion, but this must not be the only specifier.

Examples

This hook will be triggered by my-plugin/foo, my-plugin/bar,and my-other-plugin/baz:

hooks:
- name: my-fishy-hook
  events:
  - categoryPrefix: my-plugin
  - categoryPrefix: my-other-plugin
    category: _

All providers will trigger this hook whenever they are used (either explicitly in the workflow, or in generated jobs or steps):

hooks:
- name: my-global-hook
  events:
  - category: _

hooks.before

Required if hooks.after not specified The steps to include before performing the triggering task.

Example

hooks:
- name: foo
  events:
  - category: _
  before:
  - run: echo foo!

hooks.after

Required if hooks.before not specified The steps to include after performing the triggering task.

Example

hooks:
- name: my before/after hook
  events:
  - categoryPrefix: my-plugin
    category: my-task
  before:
  - run: echo before my-plugin/my-task
  after:
  - run: echo after my-plugin/my-task

hooks.if

You can use the if conditional to prevent a hook from being triggered unless a condition is met. You can use any supported context and expression to create a conditional.

When you use expressions in an if conditional, you may omit the expression syntax (${{ }}) because the orchestrator automatically evaluates the if conditional as an expression. For more information, see “Context and expression syntax for OpenTestFactory workflows.”

defaults

A map of default settings that will apply to all jobs in the workflow. You can also set default settings that are only available to a job. For more information, see jobs.<job_id>.defaults.

When more than one default setting is defined with the same name, the orchestrator uses the most specific default setting. For example, a default setting defined in a job will override a default setting that has the same name defined in a workflow.

defaults.run

You can provide default shell and working-directory options for all run steps in a workflow. You can also set default settings for run that are only available to a job. For more information, see jobs.<job_id>.defaults.run. You cannot use contexts or expressions in this keyword.

When more than one default setting is defined with the same name, the orchestrator uses the most specific default setting. For example, a default setting defined in a job will override a default setting that has the same name defined in a workflow.

Example

defaults:
  run:
    shell: bash
    working-directory: scripts

jobs

A workflow run is made up of one or more jobs. Jobs run in parallel by default. To run jobs sequentially, you can define dependencies on other jobs using the jobs.<job_id>.needs keyword.

Each job runs in an execution environment specified by runs-on.

You can run an unlimited number of jobs as long as you are within the workflow usage limits. For more information, see “Usage limits.”

jobs.<job_id>

Each job must have an id to associate with the job. The key job_id is a string and its value is a map of the job’s configuration data. You must replace <job_id> with a string that is unique to the jobs object. The <job_id> must start with a letter or _ and contain only alphanumeric characters, -, or_.

Example

jobs:
  my_first_job:
    name: My first job
  my_second_job:
    name: My second job

jobs.<job_id>.name

The name of the job displayed by the orchestrator.

jobs.<job_id>.needs

Identifies any jobs that must complete successfully before this job will run. It can be a string or array of strings. If a job fails, all jobs that need it are skipped unless the jobs use a conditional statement that causes the job to continue.

Example

jobs:
  job1:
  job2:
    needs: job1
  job3:
    needs: [job1, job2]

In this example, job1 must complete successfully before job2 begins, and job3 waits for both job1 and job2 to complete.

The jobs in this example run sequentially:

  1. job1
  2. job2
  3. job3

jobs.<job_id>.runs-on

Required The environment to run the job on.

Examples

runs-on: robotframework
or

runs-on: [linux, robotframework]

Each execution environment has one or more tags. So, you can prescribe where the job should be run by specifying a single tag or an array of tags: an environment will be selected for the execution only if it has (at least, it can have some other additional ones) these specified tags.

For more information, see Execution environments.

jobs.<jobs_id>.outputs

A map of outputs for a job. Job outputs are available to all downstream jobs that depend on this job. For more information on defining job dependencies, see jobs.<job_id>.needs.

Job outputs are strings, and job outputs containing expressions are evaluated at the end of each job.

To use job outputs in a dependent job, you can use the needs context. For more information, see “Context and expression syntax for OpenTestFactory workflows.”

Example

jobs:
  job1:
    runs-on: ubuntu-latest
    # Map a step output to a job output
    outputs:
      output1: ${{ steps.step1.outputs.test }}
      output2: ${{ steps.step2.outputs.test }}
    steps:
    - id: step1
      run: echo "::set-output name=test::hello"
    - id: step2
      run: echo "::set-output name=test::world"
  job2:
    runs-on: ubuntu-latest
    needs: job1
    steps:
    - run: echo ${{needs.job1.outputs.output1}} ${{needs.job1.outputs.output2}}

jobs.<job_id>.variables

A map of environment variables that are available to all steps in the job. You can also set environment variables for the entire workflow or an individual step. For more information, see variables and jobs.<job_id>.steps.variables.

When more than one environment variable is defined with the same name, the orchestrator uses the most specific environment variable. For example, an environment variable defined in a step will override job and workflow variables with the same name, while the step executes. A variable defined for a job will override a workflow variable with the same name, while the job executes.

Example

jobs:
  job1:
    variables:
      FIRST_NAME: Mona

jobs.<job_id>.defaults

Use jobs.<job_id>.defaults to create a map of default settings that will apply to all steps in the job. You can also set default settings for the entire workflow. For more information, see defaults.

When more than one default setting is defined with the same name, the orchestrator uses the most specific default setting. For example, a default setting defined in a job will override a default setting that has the same name defined in a workflow.

jobs.<job_id>.defaults.run

Provide default shell and working-directory to all run steps in the job. Context and expression are not allowed in this section.

You can provide default shell and working-directory options for all run steps in a job. You can also set default settings for run for the entire workflow. For more information, see defaults.run. You cannot use contexts or expressions in this keyword.

When more than one default setting is defined with the same name, the orchestrator uses the most specific default setting. For example, a default setting defined in a job will override a default setting that has the same name defined in a workflow.

Example

jobs:
  job1:
    runs-on: ubuntu-latest
    defaults:
      run:
        shell: bash
        working-directory: scripts

jobs.<job_id>.if

You can use the if conditional to prevent a job from running unless a condition is met. You can use any supported context and expression to create a conditional.

When you use expressions in an if conditional, you may omit the expression syntax (${{ }}) because the orchestrator automatically evaluates the if conditional as an expression. For more information, see “Context and expression syntax for OpenTestFactory workflows.”

jobs.<job_id>.generator

Selects a generator to run. A generator creates a series of jobs that replace the current job.

We strongly recommend that you include the version of the generator you are using. If you do not specify a version, it could break your workflow or cause unexpected behavior when the generator owner publishes an update.

  • Using the specific major generator version allows you to receive critical fixes and security patches while still maintaining compatibility. It also assures that your workflow should still work.

Generators may require inputs that you must set using the with keyword. Review the generator’s README file to determine the inputs required.

Example using versioned generators

jobs:
  my_job:
    # Reference the major version of a release
    generator: foobar/hello_world@v1
  my_second_job:
    # Reference a minor version of a release
    generator: foobar/hello_world@v1.2

jobs.<job_id>.with

A map of the input parameters defined by the generator plugin. Each input parameter is a key/value pair.

Example

Defines the three input parameters (first_name, middle_name, and last_name) defined by the hello_world generator.

jobs:
  generator: foobar/hello_world@v1
  with:
    first_name: Mona
    middle_name: The
    last_name: Octocat

jobs.<job_id>.steps

A job contains a sequence of tasks called steps. Steps can run commands or functions. Not all steps run functions, but all functions run as a step. Each step runs in its own process in the execution environment and has access to the workspace and filesystem. Because steps run in their own process, changes to environment variables are not preserved between steps. The orchestrator provides built-in steps to set up and complete a job.

You can run an unlimited number of steps as long as you are within the job usage limits. For more information, see “Usage limits.”

Example

metadata:
  name: Greeting from Mona

jobs:
  my-job:
    name: My Job
    runs-on: ubuntu-latest
    steps:
    - name: Print a greeting
      variables:
        MY_VAR: Hi there! My name is
        FIRST_NAME: Mona
        MIDDLE_NAME: The
        LAST_NAME: Octocat
      run: |
        echo $MY_VAR $FIRST_NAME $MIDDLE_NAME $LAST_NAME.

jobs.<job_id>.steps[*].id

A unique identifier for the step. You can use the id to reference the step in contexts. For more information, see “Context and expression syntax for OpenTestFactory workflows.”

jobs.<job_id>.steps[*].if

You can use the if conditional to prevent a step from running unless a condition is met. You can use any supported context and expression to create a conditional.

When you use expressions in an if conditional, you may omit the expression syntax (${{ }}) because the orchestrator automatically evaluates the if conditional as an expression. For more information, see “Context and expression syntax for OpenTestFactory workflows.”

Example using contexts

This step only runs when the runner OS is Windows.

steps:
 - name: My first step
   if: ${{ runner.os == 'windows' }}
   run: echo This step is running on windows.

Example using status check functions

The my backup step only runs when the previous step of a job fails. For more information, see “Context and expression syntax for OpenTestFactory workflows.”

steps:
  - name: My first step
    uses: monacorp/function-name@v1
  - name: My backup step
    if: ${{ failure() }}
    uses: actions/heroku@v2

jobs.<job_id>.steps[*].name

A name for your step to display on the orchestrator.

jobs.<job_id>.steps[*].uses

Selects a function to run as part of a step in your job. A function is a reusable unit of code.

We strongly recommend that you include the version of the function you are using. If you do not specify a version, it could break your workflows or cause unexpected behavior when the provider owner publishes an update.

  • Using the specific major function version allows you to receive critical fixes and security patches while still maintaining compatibility. It also assures that your workflow should still work.

Functions may require inputs that you must set using the with keyword. Review the provider’s README file to determine the inputs required.

Providers are plugins that add functions. For more details, see Providers chapter.

Example using versioned functions

steps:
  # Reference the major version of a release
  - uses: actions/setup-node@v1
  # Reference a minor version of a release
  - uses: actions/setup-node@v1.2

jobs.<job_id>.steps[*].run

Runs command-line programs using the execution environment’s shell. If you do not provide a name, the step name will default to the text specified in the run command.

Commands run using non-login shells by default. You can choose a different shell and customize the shell used to run commands. For more information, see “jobs.<job_id>.steps[*].shell.”

Each run keyword represents a new process and shell in the execution environment. When you provide multi-line commands, each line runs in the same shell. For example:

  • A single-line command:
- name: Install Dependencies
  run: npm install
  • A multi-line command:
- name: Clean install dependencies and build
  run: |
    npm ci
    npm run build

Using the working-directory keyword, you can specify the working directory of where to run the command.

- name: Clean temp directory
  run: rm -rf *
  working-directory: ./temp

jobs.<job_id>.steps[*].shell

You can override the default shell settings in the execution environment using the shell keyword. You can use built-in shell keywords, or you can define a custom set of shell options. The shell command that is run internally executes a temporary file that contains the commands specified in the run keyword.

Supported platform shell parameter Description Command run internally
All bash The default shell on non-Windows platforms. When specifying a bash shell on Windows, the bash shell included with Git for Windows is used. bash --noprofile --norc -eo pipefail {0}
All python Executes the python command. python {0}
All pwsh The PowerShell Core. The orchestrator appends the extension .ps1 to your script name. pwsh -command ". '{0}'"
Windows cmd The default shell on Windows. The orchestrator appends the extension .cmd to your script name and substitutes for {0}. %ComSpec% /D /E:ON /V:OFF /S /C "CALL "{0}""
Windows powershell The PowerShell Desktop. The orchestrator appends the extension .ps1 to your script name. powershell -command ". '{0}'"

For pwsh and python, the command used must be installed in the execution environment.

For powershell, the execution policy must allow script execution on the execution environment. See “about_Execution_Policies” for more information.

Example running a script using bash

steps:
  - name: Display the path
    run: echo $PATH
    shell: bash

Example running a script using Windows cmd

steps:
  - name: Display the path
    run: echo %PATH%
    shell: cmd

Example running a python script

steps:
  - name: Display the path
    run: |
      import os
      print(os.environ['PATH'])
    shell: python

Custom shell

You can set the shell value to a template string using

command [options...] {0} [more_options...]

The orchestrator interprets the first whitespace-delimited word of the string as the command and inserts the file name for the temporary script at {0}.

For example:

steps:
  - name: Display the environment variables and their values
    run: |
      print %ENV
    shell: perl {0}

The command used, perl in this example, must be installed in the execution environment.

Exit codes and error action preference

For built-in shell keywords, we provide the following defaults that are executed by the execution environments. You should use these guidelines when running shell scripts.

  • bash:

    • Fail-fast behavior using set -e o pipefail: Default for bash and built-in shell. It is also the default when you do not provide an option on non-Windows platforms.
    • You can opt out of fail-fast and take full control by providing a template string to the shell options. For example, bash {0}.
    • sh-like shells exit with the exit code of the last command executed in a script, which is also the default behavior for functions. The execution environment will report the status of the step as fail/succeed based on this exit code.
  • powershell/pwsh:

    • Fail-fast behavior when possible. For pwsh and powershell built-in shell, we will prepend $ErrorActionPreference = 'stop' to script contents.
    • We append if ((Test-Path -LiteralPath variable:\LASTEXITCODE)) { exit $LASTEXITCODE } to powershell scripts so action statuses reflect the script’s last exit code.
    • Users can always opt out by not using the built-in shell, and providing a custom shell option like: pwsh -File {0}, or powershell -Command "& '{0}'", depending on need.
  • cmd

    • There does not seem to be a way to fully opt into fail-fast behavior other than writing your script to check each error code and respond accordingly. Because we can’t actually provide that behavior by default, you need to write this behavior into your script.
    • cmd.exe will exit with the error level of the last program it executed, and it will return the error code to the execution environment.

jobs.<job_id>.steps[*].with

A map of the input parameters defined by the function. Each input parameter is a key/value pair. Input parameters are set as environment variables. The variable is prefixed with INPUT_ and converted to upper case.

Example

Defines the three input parameters (first_name, middle_name, and last_name) defined by the hello_world function. These input variables will be accessible to the hello-world function as INPUT_FIRST_NAME, INPUT_MIDDLE_NAME, and INPUT_LAST_NAME environment variables.

jobs:
  my_first_job:
    steps:
      - name: My first step
        uses: actions/hello_world@master
        with:
          first_name: Mona
          middle_name: The
          last_name: Octocat

jobs.<job_id>.steps[*].variables

Sets environment variables for steps to use in the execution environment. You can also set environment variables for the entire workflow or a job. For more information, see variables and jobs.<job_id>.variables.

When more than one environment variable is defined with the same name, the orchestrator uses the most specific environment variable. For example, an environment variable defined in a step will override job and workflow variables with the same name, while the step executes. A variable defined for a job will override a workflow variable with the same name, while the job executes.

Public functions may specify expected environment variables in the README file. If you are setting a secret in an environment variable, you must set secrets using the secrets context. For more information, see “Using environment variables” and “Context and expression syntax for OpenTestFactory workflows.”

Example

steps:
  - name: My first function
    variables:
      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      FIRST_NAME: Mona
      LAST_NAME: Octocat

jobs.<job_id>.steps[*].continue-on-error

Prevents a job from failing when a step fails. Set to true to allow a job to pass when this step fails.

jobs.<job_id>.steps[*].timeout-minutes

The maximum number of minutes to run the step before killing the process.

jobs.<job_id>.timeout-minutes

The maximum number of minutes to let a job run before the orchestrator automatically cancels it. Default: 360

jobs.<job_id>.continue-on-error

Prevents a workflow run from failing when a job fails. Set to true to allow a workflow run to pass when this job fails.