Quality Gate Commands¶

The opentf-ctl tool provides a set of commands to use quality gates. You can query a quality gate for a workflow, using either a pre-defined set of rules or providing your own.

For more information on quality gates, see “Quality gate.”

`get qualitygate {workflow_id}`¶

This command queries the quality gate status for the specified workflow.

opentf-ctl get qualitygate a13f0572-b23b-40bc-a6eb-a12429f0143c
opentf-ctl get qualitygate a13

You can specify a partial workflow ID, as long as it is unique (if it is not, a message is displayed listing the available workflows).

You can specify the quality gate you want using the --mode command-line option.

There are two predefined quality gates: strict and passing. The strict quality gate is the default.

The strict quality gate passes if all tests were successful during the workflow execution.
The passing quality gate always passes.

When get qualitygate is used with --mode option, the quality gate will be loaded from the file defined on the service level.

You may also provide a custom quality gate definition file with the --using command-line option. In this case, the --mode option must also be specified and should refer to a quality gate defined in the provided custom definition file.

Tip

It is also possible to apply a quality gate to a workflow directly from the run workflow command, which may be useful in some contexts. The run workflow command supports the same options as the get qualitygate command. The --mode option is mandatory for the quality gate to be applied.

The get qualitygate return code is:

0 if the quality gate passed for the specified workflow
101 if the specified workflow is still running
102 if the quality gate failed for the specified workflow

Other return codes indicate a non-functional problem with the command (unknown workflow, unknown quality gate, …).

A workflow that contains no test matching quality gate rules scope will return NOTEST status. The general status FAILURE is returned if the workflow has not satisfied at least one rule of the quality gate. If all rules are satisfied, the SUCCESS status is returned.

The get qualitygate command default output contains the rules list and their evaluation results:

opentf-ctl get qualitygate a13 -m this.quality.gate 
opentf-ctl get qualitygate a13 --mode=this.quality.gate

RULE                     RESULT   TESTS_IN_SCOPE  TESTS_FAILED  TESTS_PASSED  SUCCESS_RATIO
cypress and junit tests  FAILURE  8               5             3             37.5%
cypress tests            SUCCESS  2               1             1             50.0%
junit tests              FAILURE  6               4             2             33.33%
Workflow a13f0572-b23b-40bc-a6eb-a12429f0143c failed the quality gate using mode this.quality.gate.

It is also possible to display the rule scope using --output=wide option:

opentf-ctl get qualitygate a13 --mode this.quality.gate --output wide

RULE                     RESULT   TESTS_IN_SCOPE  TESTS_FAILED  TESTS_PASSED  SUCCESS_RATIO  SCOPE
cypress and junit tests  FAILURE  8               5             3             37.5%          (test.technology == 'cypress') || (test.technology == 'junit')
cypress tests            SUCCESS  2               1             1             50.0%          test.technology == 'cypress'
junit tests              FAILURE  6               4             2             33.33%         (test.technology != 'cypress')
Workflow a13f0572-b23b-40bc-a6eb-a12429f0143c failed the quality gate using mode this.quality.gate.

Optional Parameters¶

The get qualitygate command allows for additional optional parameters, in no specific order, but they must follow the command if used (but they can be specified before the workflow ID):

--mode={mode} or -m {mode}                      # apply the quality gate 
--using={file} or -u {file}                     # use custom quality gate definition file
--output=wide or -o wide                        # show additional information
--output=custom-columns= or -o custom-columns=  # show specified information
--output=json or -o json                        # show information in JSON format
--output=yaml or -o yaml                        # show information in YAML format
--plugin={prefix}:{parameter}=                  # specify plugin options
--timeout={x}                                   # set query timeout (in seconds)

You can only specify one output format at a time. Please refer to “Output formats” for more information.

Selecting a Specific Quality Gate¶

The --mode optional parameter allows you to specify the quality gate to use. If not specified, the strict quality gate is used.

When --using is specified, it references a YAML file that contains a series of quality gate definitions. --mode is then mandatory and must refer to a quality gate from the definition file:

opentf-ctl get qualitygate a13 -u my_qualitygate.yaml -m custom.quality.gate 
opentf-ctl get qualitygate a13 --using=my_qualitygate.yaml --mode=custom.quality.gate

RULE         RESULT   TESTS_IN_SCOPE  TESTS_FAILED  TESTS_PASSED  SUCCESS_RATIO
first rule   SUCCESS  21              5             16            76.19%
second rule  SUCCESS  13              0             13            100.0%
Workflow a13f0572-b23b-40bc-a6eb-a12429f0143c passed the quality gate using mode custom.quality.gate.

For more information on quality gate definitions, see “Quality gate syntax.”

Publishing Quality Gate Results¶

The --plugin option allows you to publish the quality gate results to an external target. Currently, only GitLab merge requests and issues are supported: the results are published as a merge request or issue note.

The GitLab plugin accepts the following parameters:

Parameter	Description	Required
`gitlab:server`	GitLab instance URL	If not executed from a GitLab MR pipeline.
`gitlab:project`	GitLab project ID	If not executed from a GitLab MR pipeline.
`gitlab:mr`	GitLab merge request IID	If not executed from a GitLab MR pipeline and `gitlab:issue` is not specified.
`gitlab:issue`	GitLab issue IID	If not executed from a GitLab MR pipeline and `gitlab:mr` is not specified.
`gitlab:keep-history`	Update or not previous quality gate results	Yes
`gitlab:token`	GitLab authentication token	No
`gitlab:label`	Label the specified target	No

The gitlab:server and gitlab:project parameters are mandatory when the command is not executed from a GitLab MR pipeline (see GitLab CI Integration Guide for details). In this case, one of the gitlab:mr or gitlab:issue parameters has also to be specified, and you may specify both of them.

The gitlab:keep-history parameter is also mandatory and can be true or false. When it is set to false, the most recent results of the quality gate defined by --mode option will be updated when found. Otherwise, all results are kept.

Assuming the successfully completed workflow a13, the following command will send the quality gate new.quality.gate results to the GitLab instance https://gitlab.example.com and publish them as a note in the project 4453421 merge request 90.

BashCMDPowerShell

opentf-ctl get qualitygate a13 -m new.quality.gate \ 
--plugin gitlab:server=https://gitlab.example.com \ 
--plugin gitlab:project=4453421 \ 
--plugin gitlab:mr=90 --plugin gitlab:keep-history=true

opentf-ctl get qualitygate a13 -m new.quality.gate ^ 
--plugin gitlab:server=https://gitlab.example.com ^ 
--plugin gitlab:project=4453421 ^ 
--plugin gitlab:mr=90 --plugin gitlab:keep-history=true

opentf-ctl get qualitygate a13 -m new.quality.gate ` 
--plugin gitlab:server=https://gitlab.example.com ` 
--plugin gitlab:project=4453421 ` 
--plugin gitlab:mr=90 --plugin gitlab:keep-history=true

You can use the gitlab:label parameter to add a label to the related merge request or issue.

The label will have the following form: {prefix}::{quality gate result}, where {prefix} is the gitlab:label parameter value and {quality gate result} is Passed, Failed or No test respectively for SUCCESS, FAILURE and NOTEST quality gate responses.

If gitlab:label value is default, the QualityGate prefix will be applied.

Assuming the quality gate response is SUCCESS, the following command will publish the quality gate results to the specified GitLab issue and label it with a DefaultQualityGate::Passed label:

BashCMDPowerShell

opentf-ctl get qualitygate a13 -m passing \
--plugin gitlab:label=DefaultQualityGate \
--plugin gitlab:server=https://gitlab.example.com \
--plugin gitlab:project=4453421 --plugin gitlab:issue=144

opentf-ctl get qualitygate a13 -m passing ^
--plugin gitlab:label=DefaultQualityGate ^
--plugin gitlab:server=https://gitlab.example.com ^
--plugin gitlab:project=4453421 --plugin gitlab:issue=144

opentf-ctl get qualitygate a13 -m passing `
--plugin gitlab:label=DefaultQualityGate `
--plugin gitlab:server=https://gitlab.example.com `
--plugin gitlab:project=4453421 --plugin gitlab:issue=144

See GitLab CI integration guide for use examples.

Timeout¶

The quality gate creates test cases cache for each workflow. For large workflows, containing thousands of test cases, this operation can take some time, so the request may time out. Default request timeout is 8 seconds, but if you need to raise it, you can use the --timeout= command-line option to define your own timeout (in seconds):

opentf-ctl get qualitygate a13 --mode=robot.quality.gate --timeout=300

Wide View¶

This option shows the default columns and the rule scope.

opentf-ctl get qualitygate a13 -m custom.quality.gate -o wide
opentf-ctl get qualitygate a13 --mode=custom.quality.gate --output=wide

RULE         RESULT   TESTS_IN_SCOPE  TESTS_FAILED  TESTS_PASSED  SUCCESS_RATIO  SCOPE
first rule   SUCCESS  21              5             16            76.19%         (test.uses == 'cypress/execute@v1')
second rule  SUCCESS  13              0             13            100.0%         contains(test.runs-on, 'robotframework')
Workflow a13f0572-b23b-40bc-a6eb-a12429f0143c passed the quality gate using mode custom.quality.gate.

Custom View¶

If this optional parameter is specified, you can choose what is displayed for each known execution environment.

The columns definition format is as follows: name:value[,name:value]*. name is the column name in the output (RULE_SCOPE or RATIO in the example above). value is the path to the information in the workflow schema.

The following values are supported:

.rule.name
.rule.result
.rule.tests_in_scope
.rule.tests_failed
.rule.tests_passed
.rule.success_ratio
.rule.scope

opentf-ctl get qualitygate a13 -m custom.quality.gate -o custom-columns=RULE_SCOPE:.rule.scope,RATIO:.rule.success_ratio
opentf-ctl get qualitygate a13 --mode=custom.quality.gate --output=custom-columns=RULE_SCOPE:.rule.scope,RATIO:.rule.success_ratio

RULE_SCOPE                                RATIO
(test.uses == 'cypress/execute@v1')       76.19%
contains(test.runs-on, 'robotframework')  100.0%

`describe qualitygate {workflow_id}`¶

This command queries the quality gate status for the specified workflow and returns pretty-formatted status descriptions.

opentf-ctl describe qualitygate a13f0572-b23b-40bc-a6eb-a12429f0143c
opentf-ctl describe qualitygate a13

You can specify a partial workflow ID, as long as it is unique (if it is not, a message is displayed listing the available workflows).

The command output format is the only difference between get qualitygate and describe qualitygate commands. Their operational mode, return codes and parameters (except --output=...) are the same:

opentf-ctl describe qualitygate a13f0572-b23b-40bc-a6eb-a12429f0143c -u my_qualitygates.yaml -m first.quality.gate 
opentf-ctl describe qualitygate a13 --using=my_qualitygates.yaml --mode=first.quality.gate

--------RESULTS: first.quality.gate, junit tests--------

Workflow a13f0572-b23b-40bc-a6eb-a12429f0143c failed the quality gate `first.quality.gate` applying the rule `junit tests`.

    --------STATISTICS--------

    Tests in scope: 6
    Tests failed:   4
    Tests passed:   2
    Success ratio:  33.33%

Workflow a13f0572-b23b-40bc-a6eb-a12429f0143c failed the quality gate using mode first.quality.gate.