Insights¶

Insights are documents about a workflow execution.

The Insight Collector service produces them and attaches them to the workflow.

While the default configuration produces two insights, an execution log and a summary report, you can decide to produce only one of these insights or to produce additional insights, based on your preferences and needs.

This guide intends to help you understand how to configure the Insight Collector service, should you want to produce additional insights.

For example, you can produce a detailed execution log in case of failure, but a more succinct one in case of success. Similarly, your workflow may launch hundreds or thousands of tests and you can produce a global report for each test suite, and a detailed report for each failing test.

Execution Logs

A text file that contains the execution log of the workflow.

Getting started
Summary Reports

HTML reports that contain information about the workflow execution.

Getting started
Testing your Definitions

You can dynamically test your insights definitions.

Getting started

Common Configuration¶

Insights have a name, which will be used to name the file produced, and a kind, which will be used to determine the kind of insight produced.

The name does not have to be unique, but using different names for different insights is a recommended practice, as only the first produced insight with a given name will be attached to the workflow.

Each insight also has an attached if conditional. If the conditional is not met, the insight is not produced.

By default, the conditional is always(), which means that the insight is always produced. You have access to the workflow context in your conditionals.

You can specify a spec part. This part is optional.

insights:
- name: my_insight  # required
  kind: MyInsight   # required
  if: success()     # optional ('always()' if not specified)
  spec:             # optional (default configuration for kind if not specified)
    ...

Tip

To apply your insights configuration, you need to pass the configuration file path to the variable INSIGHTCOLLECTOR_CONFIGURATION when launching the orchestrator or the Insight Collector service.

See “Reports Configuration File” for more information.

Execution Logs¶

Execution logs insights have the kind ExecutionLog. The attachment’s filename is the name suffixed by .txt.

Examples¶

The example below sets all execution log properties to 0 to ensure that the produced execution log have the maximum possible depth:

insights:
- name: executionlog
  kind: ExecutionLog
  spec:
    step-depth: 0          # defaults to 1 if not specified
    job-depth: 0           # defaults to 1 if not specified
    max-command-length: 0  # defaults to 15 if not specified

The following example produces a more succinct execution log if the workflow is in success:

insights:
- name: executionlog
  kind: ExecutionLog
  if: success()
  spec:
    step-depth: 1
    job-depth: 1
    max-command-length: 15
- name: executionlog
  kind: ExecutionLog
  if: '!success()'
  spec:
    step-depth: 0          # defaults to 1 if not specified
    job-depth: 0           # defaults to 1 if not specified
    max-command-length: 0  # defaults to 15 if not specified

Note

This is a rare case where it is legitimate to have two insights with the same name. The first insight is produced if the workflow is successful, and the second insight is produced if the workflow is not successful.

Workflow Labels¶

If you want to specify the execution log detail level for a single workflow, you must add some execution log specific labels under the metadata.labels section of this workflow. The following labels are recognized:

Label	Value	Required
`executionlog/step-depth`	Step display depth	No
`executionlog/job-depth`	Job display depth	No
`executionlog/max-command-length`	Run commands display length in characters	No

executionlog/step-depth defines the step display depth for the execution log of the workflow. If this value is set to 0, all steps are displayed, if it is set to 1, only the steps of the first level are displayed and the nested steps are hidden, if it is set to 2, the steps produced by providers are also displayed.

executionlog/job-depth defines the job display depth for the execution log of the workflow. If this value is set to 0, all jobs are displayed, if it is set to 1, only the jobs of the first level are displayed and the nested jobs are hidden, if it is set to 2, the jobs produced by generators are also displayed.

executionlog/max-command-length sets the maximum display length of run commands, measured in characters. 0 displays full commands, while any positive integer specifies a maximum length.

Info

The values set at the workflow level overwrite the values set in the service configuration file. If a workflow contains no execution log related labels and the service configuration file is not specified, the execution log parameters default to the following values: step-depth=1, job-depth=1, max-command-length=15.

Example¶

The following workflow contains all executionlog/... labels which are set to 0:

metadata:
  name: Robot test
  labels:
    executionlog/step-depth: '0'
    executionlog/job-depth: '0'
    executionlog/max-command-length: '0'
jobs:
 robot-execute-test:
    runs-on: [robotframework, linux]
    steps:
    - uses: robotframework/execute@v1
      with:
        test: robotframework/test_chars.robot
      working-directory: /home/user/directory

Warning

Workflow label values must be strings (and not integers): the numerical values must thus be quoted.

It will produce the following execution log, where all the steps are shown, and all commands are displayed at the full length:

[...]
[timestamp][job uuid] Running function robotframeworkexecutev1 
[timestamp][job uuid]  Running function actionsdelete-filev1 
[timestamp][job uuid]   Running command: rm -f log.html
[timestamp][job uuid]  Running command: robot $ROBOTFRAMEWORK_EXTRA_OPTIONS 
--nostatusrc --listener "allure_robotframework;." "robotframework/test_chars.robot"
[...]
[timestamp][job uuid]  Running function actionscreate-archivev1 
[timestamp][job uuid]   Running command: rm -f RobotFramework_reports.tar
[timestamp][job uuid]   Running command: rm -f tmp_tar_files
[timestamp][job uuid]   Running command: rm -f intentionally_empty_archive
[timestamp][job uuid]   Running command: rm -f list_tar_files
[timestamp][job uuid]   Running command: touch list_tar_files
[...]

Summary Reports¶

Summary reports are HTML reports that contain information about the workflow execution. The Insight Collector service produces them and attaches them to the workflow.

The definition of a SummaryReport insight is as follows:

- name: my_insight     # required
  kind: SummaryReport  # required
  if: success()        # optional ('always()' if not specified)
  title: "My Insight Report"
                       # optional
  spec:                # optional (default configuration if not specified)
    ...

As with all insights, if an if conditional is specified, the insight is only produced if the conditional is met.

The insight’s title, if specified, is used as the report’s title. It defaults to Test Execution Summary Report.

The summary report is generated from the execution log and the workflow metadata.

It may contain any combination of the following sections, in any order and in any number (but there must be at least one section):

SummaryBlocks
Clouds
Tables

The spec part is optional. If not specified, the default sections configuration is used. It is as follows:

insights:
- name: htmlreport
  kind: SummaryReport
  title: "Test Execution Summary Report"
  spec:
    scope:
    scripts:
    style:
    template:
    - name: Summary
      kind: SummaryBlock
    - name: Jobs
      kind: Cloud
      datasource: jobs
    - name: Tags
      kind: Table
      datasource: tags
    - name: Testcases
      kind: Table
      datasource: testcases

It summarizes all jobs and testcases and displays all tags.

The spec section, if specified, can contain the following elements:

spec:
  scope:                   # optional
  scripts:                 # optional
  style:                   # optional
  template:                # required
  - name: A Section        # required
    kind: SummaryBlock     # required
    scope:                 # optional
    skipIfNotFound:        # optional (false if unspecified)
  - name: Another Section  # required
    kind: Cloud            # required
    datasource: jobs       # required
    scope:                 # optional
    skipIfNotFound:        # optional (false if unspecified)
  - name: Another one      # required
    kind: Table            # required
    datasource: testcases  # required
    scope:                 # optional
    skipIfNotFound:        # optional (false if unspecified)
    columns:               # optional (default columns based on datasource if
                           #           not specified)
      name: expression     # required (if `columns` is specified, there must be
                           #           at least one column defined)
      ...
  - ...

(The template list must contain at least one section. It can be of any kind. The sections can be in any order, and it’s just fine to have multiple sections of any given kind.)

Scopes¶

By default, the reports summarize all jobs and testcases and display all tags. This may be too much information for a large workflow. You can use the scope part to filter the information displayed in a summary report.

The scope is an expression. Only testcases matching the specified expression are displayed in the summary report. You can use the test context in your scope expressions.

You can specify a scope at spec level. You can also specify it in the template elements.

Those specifications are cumulative. If a scope is specified at spec level and at template level, the scopes are combined.

For example, if spec.scope is test.technology == 'junit', and if the Testcases template has a scope status == 'failed', the testcases table will display only the failed JUnit testcases.

Scripts¶

By default, the summary reports make use of two open-source JavaScript libraries:

<script src="https://cdnjs.cloudflare.com/ajax/libs/list.js/1.5.0/list.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/chart.js"></script>

You can use the scripts element to redefine those, for example if you are restricted to an intranet.

The scripts element, if defined, replaces the above scripts.

Style¶

If specified, the style part complements the CSS used by the summary reports. You can use it to change the colors, add a company logo, …

If defined, it complements the default CSS, it does not replace it.

Datasources¶

There are three datasources:

datasource	description
`jobs`	The set of all jobs
`tags`	The set of all tags
`testcases`	The set of all testcases

If a scope is specified, the sets will be constrained by the scope.

skipIfNotFound¶

The template’s items all share a common skipIfNotFound element.

If it is set to true (a boolean), then the section is silently skipped if the datasource is empty.

For example, you may want to have a detailed section for failed test cases, but do not want to see it if there were no failed test case.

If you do not set this element, it defaults to false, and the section will be present in the report, with a line saying that there were no matching items.

Sections¶

SummaryBlocks¶

SummaryBlocks are sections of the summary report. They are defined by their name and kind properties. The name property is used to name the section in the report.

The sections have two or three parts. The first part is the test execution summary. The second part is the jobs summaries (if there is only one job in the workflow, it is not shown, as it would be just like the first part). The third part is the technologies summary.

A SummaryBlock is a set of doughnuts summarizing the testcases statuses, globally and per
technology

Clouds¶

Clouds blocks are sections of the summary report. They are defined by their name and kind properties. The name property is used to name the section in the report.

They have a mandatory datasource element, and they may have a scope element too.

The datasource element defines the items shown in the cloud block.

A Cloud is a set of cloud items summarizing the items' statuses

Note

The only cloud block available is the Jobs cloud, so you can only use jobs as a datasource.

Tables¶

Tables blocks are sections of the summary report. They are defined by their name and kind properties. The name property is used to name the section in the report.

They have a mandatory datasource element, and they may have a scope element too.

The datasource element defines the rows shown in the table. You can use the tags datasource to display the tags in a table, and the testcases datasource to display the testcases in a table.

There are default columns specifications, per datasources. You can provide your own columns specifications by using the columns element, which is a collection of name, expression pairs.

The name is the column header, and the expression is the cell content.

A tags table¶

A Table is a set of rows summarizing the items' statuses

The default columns are defined as such:

columns:
  Tag: name
  OK: success
  KO: failure
  Other: other
  Total: total

For the columns values, you can use any of the five values shown above. The numeric values are displayed right-aligned, the other is shown as-is.

A testcase table¶

A Table is a set of rows summarizing the items' statuses

The default columns are defined as such:

columns:
  Test suite: test.suite
  Testcase: name
  Dataset: test.data
  Status: status
  Test technology: test.technology
  Job: test.job

For the columns values, you can use elements included in the test context.

There are two additional elements, name and status, which are always available.

There are special renderings for the name, status, test.data, and test.suite values. The other values are just displayed as is.

Examples¶

The first example produces a summary report that contains only the summary block and the jobs cloud, for successful workflows:

insights:
- name: htmlreport
  kind: SummaryReport
  if: success()
  title: "Test Execution Summary Report"
  spec:
    template:
    - name: Summary
      kind: SummaryBlock
    - name: Jobs
      kind: Cloud
      datasource: jobs

The following example produces a summary report that contains the summary block, the tags table, and a testcases table per test kinds (here, assuming the front-end tests were written in Robot Framework, the back-end tests in JUnit, and other tests in some other technologies), for all workflows:

insights:
- name: htmlreport
  kind: SummaryReport
  title: "Test Execution Summary Report"
  spec:
    template:
    - name: Summary
      kind: SummaryBlock
    - name: Tags
      kind: Table
      datasource: tags
    - name: Front-End Testcases
      kind: Table
      datasource: testcases
      scope: test.technology == 'robotframework'
      skipIfNotFound: true
    - name: Back-End Testcases
      kind: Table
      datasource: testcases
      scope: test.technology == 'junit'
      skipIfNotFound: true
    - name: Other Testcases
      kind: Table
      datasource: testcases
      scope: test.technology != 'robotframework' && test.technology != 'junit'
      skipIfNotFound: true

The columns in the testcases tables are the default ones.

The third example produces a summary report that contains the summary block, and a testcases table for all non-successful testcases. The columns are adjusted, so that they only contain the name of the testcase and the status of the testcase:

insights:
- name: htmlreport
  kind: SummaryReport
  title: "Failed Tests Execution Summary Report"
  spec:
    template:
    - name: Summary
      kind: SummaryBlock
    - name: Failed Testcases
      kind: Table
      datasource: testcases
      scope: status != 'success'
      skipIfNotFound: true
      columns:
        Test name: name
        Test status: status

The last example produces a summary report that only contains a testcases table for all non-successful test cases with default columns. The style of the summary report is adjusted using style element in order to add a logo on the top of the page, and the scripts element is used to override default libraries location. Please note that in the scripts element, opening and closing <script> tags are provided, whereas style element content is not surrounded by tags.

insights:
- name: htmlreport
  kind: SummaryReport
  title: "Failed Tests"
  spec:
    style: |
      body::before {
         content: '';
         background: url('path/to/logo.jpg');
         background-size: 100px 100px;
         display: inline-block;
         width: 100px;
         height: 100px;
      }
    scripts: |
      <script src="https://127.0.0.1/local_scripts/list.min.js"></script>
      <script src="https://127.0.0.1/local_scripts/chart.js"></script>
    template:
      - name: Failed Testcases
        kind: Table
        datasource: testcases
        scope: status == 'failure'

Testing your Definitions¶

You can use the opentf-ctl tool to test your insights definitions on a completed workflow.

For example, to test the definitions in the insights.yaml file, you can use the following command:

> opentf-ctl get workflows

WORKFLOW_ID                           STATUS  NAME
48744ed2-c7a3-4323-9b40-4e6d9b1faf8a  DONE    My Workflow

> opentf-ctl generate report 487 using insights.yaml

Next Steps¶

Here are some helpful resources for taking your next steps with insights

“Expressions” for an in-depth view of all possible things you can use in expressions
“Contexts” for an in-depth view of all possible contexts you can use in expressions
“Reports Configuration File” for more information on how to deploy your configuration.
“Dynamic Reports Generation” for more information on how to test your reports definitions.