Defining quality gate

A quality gate is a set of rules a workflow execution must satisfy. You must create a YAML file to define your quality gate or gates.

Quality gate files must use YAML syntax and may have a .yml or .yaml file extension.

Quality gate structure

A quality gate has a mandatory qualitygates section, which must contain at least one quality gate.

qualitygates:
- name: daily.quality.gate
  rules:
  - name: IHM tests
    rule:
      scope: test.technology == 'junit'
      threshold: 55%
      failure-status: [failure, blocked]
  - name: API tests
      ...
- name: nightly.quality.gate
  rules:
    ...

qualitygates[*].name

Required The name of the quality gate. This name is used as the mode parameter value when querying the quality gate service endpoints.

qualitygates[*].rules

Required The quality gate rules list. This section must contain at least one rule that will be used to evaluate a workflow execution.

qualitygates[*].rules[*].name

The name of a quality gate rule. Even if optional, it is recommended to name the rule to have more readable quality gate results.

qualitygates[*].rules[*].rule

Required A quality gate rule definition, that contains rule scope, threshold and failure statuses. Rule scope and threshold are mandatory.

qualitygates[*].rules[*].rule.scope

Required A quality gate rule scope. Only the tests matching the scope will be evaluated.

The rule scope uses workflow context syntax and is based on the test context. It contains information about the executed test cases.

Property name	Type	Description
test.job	string	The name of a job containing the executed test case.
test.technology	string	The executed test case technology.
test.uses	string	The name of an action that executed the test case.
test.runs-on	object	The tags describing the test case execution environment.
test.managed	boolean	If true, selects only the test cases managed by a test referential, and vice versa.

Example usage of the quality gate rule scope

This example quality gate applies three rules for a workflow with several jobs containing JUnit and Cypress tests. The first rule uses the scope: 'true' statement, which matches all tests in a workflow, the second targets only the tests that were executed on Linux inside the job named API, and the third refers to JUnit tests that do not use junit/junit action.

qualitygates:
- name: example.quality.gate
  rules:
  - name: All tests
    rule:
      scope: 'true'
      threshold: 90%
  - name: API tests
    rule:
      scope: (contains(test.runs-on, 'linux')) && (test.job == 'API')
      threshold: 100%
  - name: Webapp tests
    rule:
      scope: (test.technology == 'junit') || (test.uses != 'junit/junit@v1')
      threshold: 75%

qualitygates[*].rules[*].rule.threshold

Required A minimum threshold of success (percentage). The quality gate will be considered as passed if the average success rate of the tests matching the rule scope is greater than or equal to the threshold.

qualitygates[*].rules[*].rule.failure-status

The statuses of the test for which it is considered as failed. If failure-status is not set, it will default to [error, failure, blocked]. The possible values for the failure-status are error, success, blocked, failure and skipped.