Quality gate¶

When a workflow is launched within a pipeline, you may need to fine-tune pipeline failing or succeeding conditions. For instance, you may want a pipeline to fail if there are less than 50% of successful Selenium tests, or to succeed even if all the API tests have failed. The quality gate service provides such a possibility.

This service exposes two endpoints, GET and POST, that allow to evaluate a workflow execution result via the service level quality gate definition or the definition file sent by the user.

To simplify the communication with the quality gate service, two opentf-ctl commands, get qualitygate and describe qualitygate, can be used to evaluate a workflow.

Tip

It is also possible to apply a quality gate to a workflow directly from the run workflow command, which may be useful in some contexts. The run workflow command supports the same options as the get qualitygate command. The --mode option is mandatory for the quality gate to be applied.

Defining quality gate¶

A quality gate is a set of rules. Its definition must be a .yaml or .yml file. The quality gate syntax is described in the Quality gate syntax chapter.

The definition file may contain several quality gates, but at least one quality gate with at least one rule is mandatory. The rule is defined by three parameters: scope, threshold and failure-status: two first are mandatory, failure-status defaults to [failure, blocked, error] if not set.

In this example, the definition file contains one quality gate with one rule limiting evaluated tests scope to the Cypress tests.

qualitygates:
- name: cypress.quality.gate
  rules:
  - name: Cypress tests
    rule:
      scope: test.technology == 'cypress'
      threshold: 75%
      failure-status: [failure, blocked, error]

Using quality gate definition¶

The quality gate definition may be implemented on the quality gate service level or sent to the POST endpoint.

Service level definition¶

To use a service level definition, you need to pass the path of a quality gate definition file to the environment variable QUALITYGATE_DEFINITIONS when launching the orchestrator or the quality gate service.

In this case, to apply a quality gate to a workflow, you have to use its name as the mode parameter value when communicating with the quality gate service.

You may use the opentf-ctl get qualitygate command with --mode (or -m) option:

opentf-ctl get qualitygate {workflow_id} --mode cypress.quality.gate

Or you may send a GET request to the endpoint if opentf-ctl is not available:

BashCMDPowerShell

curl http://orchestrator.example.com/workflows/{workflow_id}/qualitygate?mode=cypress.quality.gate \
     -H "Authorization: Bearer ${TOKEN}"

curl http://orchestrator.example.com/workflows/{workflow_id}/qualitygate?mode=cypress.quality.gate ^
     -H "Authorization: Bearer %TOKEN%"

curl.exe http://orchestrator.example.com/workflows/{workflow_id}/qualitygate?mode=cypress.quality.gate `
         -H "Authorization: Bearer $Env:TOKEN"

User definition¶

User-provided quality gate definition can be directly sent to the service. The mode parameter value must refer to a quality gate from the posted definition (service level definitions are not taken into account in this case).

You may use the opentf-ctl get qualitygate command and pass the quality gate definition path with the --using or -u option:

opentf-ctl get qualitygate {workflow_id} --using path/to/cypress_qualitygate.yaml --mode cypress.quality.gate

Or you can send to the endpoint a POST request with the file attached if opentf-ctl is not available:

BashCMDPowerShell

curl -X POST \
        -H "Authorization: Bearer ${TOKEN}" \
        -F qualitygates=@path/to/cypress_qualitygate.yaml \
        http://orchestrator.example.com/workflows/{workflow_id}/qualitygate?mode=cypress.quality.gate

curl -X POST ^
        -H "Authorization: Bearer %TOKEN%" ^
        -F qualitygates=@path/to/cypress_qualitygate.yaml ^
        http://orchestrator.example.com/workflows/{workflow_id}/qualitygate?mode=cypress.quality.gate

curl.exe -X POST `
        -H "Authorization: Bearer $Env:TOKEN" `
        -F qualitygates='@path/to/cypress_qualitygate.yaml' `
        http://orchestrator.example.com/workflows/{workflow_id}/qualitygate?mode=cypress.quality.gate

Workflow evaluation¶

Quality gates apply only to successfully completed workflows with test results.

The completed workflow evaluation is based on the quality gate rules: their scope, threshold and failure status list. First, for each rule, the success ratio of tests in scope is calculated and then compared to the rule threshold. If the success ratio is equal to or greater than the threshold, the rule status is set to SUCCESS, otherwise, it is set to FAILURE. If no test matches the rule scope, the rule status is set to NOTEST.

Second, the workflow status is evaluated. If there is at least one rule in FAILURE status, the workflow status is FAILURE. If there are no failed rules and at least one rule in SUCCESS, the workflow status is SUCCESS. If all rules evaluate to NOTEST, the workflow status is also NOTEST.

Example of a workflow evaluation¶

Assuming a successfully completed workflow containing Cypress and JUnit tests, we want to evaluate its results. All Cypress tests need to be successfully executed and 25% of JUnit tests may fail. Some JUnit tests fail due to technical errors and we don’t want to take them into account.

First, we need to create a quality gate evaluating this execution and save it to the cypress_junit_qualitygate.yaml file.

qualitygates:
- name: cypress.junit.quality.gate
  rules:
  - name: Cypress tests
    rule:
      scope: test.technology == 'cypress'
      threshold: 100%
      failure-status: [failure, blocked, error]
  - name: JUnit tests
    rule:
      scope: test.technology == 'junit'
      threshold: 75%
      failure-status: [failure]

Then, supposing the quality gate service is already running, we send this definition using get qualitygate command:

opentf-ctl get qualitygate a13f0572-b23b-40bc-a6eb-a12429f0143c --using cypress_junit_qualitygate.yaml --mode cypress.junit.quality.gate

We will get a response that contains test evaluation results and the general workflow execution status. The JUnit tests execution result is evaluated as SUCCESS even if some tests have failed, but the general execution status is FAILURE, because Cypress tests did not pass the threshold.

RULE,RESULT,TESTS_IN_SCOPE,TESTS_FAILED,TESTS_PASSED,SUCCESS_RATIO
Cypress tests,FAILURE,20,10,10,50.0%
JUnit tests,SUCCESS,50,10,40,80.0%
Workflow a13f0572-b23b-40bc-a6eb-a12429f0143c failed the quality gate using mode cypress.junit.quality.gate.