Surefire parser service¶

This service parses the status of test cases.

A notification is published for all found test cases.

It exposes no user-facing endpoints.

Environment variable¶

Logging¶

You can set the DEBUG_LEVEL (all upper-cased) environment variable to DEBUG to add additional information in the console of the launched service. It defaults to INFO. (Please note that setting DEBUG_LEVEL to DEBUG will produce tons of logs.)

Tip

You can also set the DEBUG_LEVEL environment variable to TRACE to get even more logs. Do not do this except if you know you need it :)

Custom rules specification¶

The INTERPRETER_CUSTOM_RULES environment variable can be used to specify a custom configuration file. If this variable is not set, the default custom configuration location, /app/plugins/interpreter/interpreter.yaml, is used.

Default configuration¶

The Surefire parser service provides parsers covering six technologies by default:

Technology	Description
`cucumber`	Cucumber JVM ≤ 4
`cucumber5`	Cucumber JVM ≥ 5
`cypress`	Cypress
`junit`	JUnit
`soapui`	SoapUI
`skf`	Squash Keyword Framework

You can override those built-in parsers by configuring a parser that accepts one of these technologies.

Custom configuration file¶

The custom configuration file, if it exists, must be a YAML file. It specifies a list of parsers.

Its location is either /app/plugins/interpreter/interpreter.yaml or the one specified by the INTERPRETER_CUSTOM_RULES environment variable.

Here is an example of what a custom configuration file looks like:

- name: Cypress
  matchCases:
  - name: new syntax
    regex: ^(?<repositoryAndProject>[^#]+)#(?<testFile>[^#]+)$
    xpath: //testsuite[@file="{testFile}"]/following-sibling::testsuite/testcase
  - name: old syntax
    regex: ^(?<repositoryAndProject>[^/#]+)/(?<testFile>[^#]+)$
    xpath: //testsuite[@file="{testFile}"]/following-sibling::testsuite/testcase
  acceptedTechnologies:
  - cypress
  acceptedMediaTypes:
  - application/vnd.opentestfactory.cypress-surefire+xml
- name: SoapUI
  matchCases:
  - name: syntax full
    regex: ^(?<repository>[^#]+)/(?<project>[^#]+)#(?<suite>[^#]+)#(?<testCase>[^#]+)$
    xpath: //testsuite[@name[contains(., ".{suite}")]]/testcase[@name="{testCase}"]
  - name: syntax no test case
    regex: ^(?<repository>[^#]+)/(?<project>[^#]+)#(?<suite>[^#]+)$
    xpath: //testsuite[@name[contains(., ".{suite}")]]/testcase
  acceptedTechnologies:
  - soapui
  acceptedMediaTypes:
  - application/vnd.opentestfactory.soapui-surefire+xml

Info

If you change the content of this file, you must restart the service so that the changes are taken into account. It is not currently dynamically reloaded.

Parsers¶

A parser is defined by its name, technology, and media type. It handles attachments that are generated by one of its specified technologies and that are of one of its specified media types.

If more than one parser matches, the last seen one wins. That allows you to redefine a built-in parser.

Once technology and media type is verified, each case definition will be tried against the test reference in the same order they are declared. If more than one case match, then it will be the first in the list to be selected. If no regex matches a given test reference, an error status will be returned.

Structure¶

Parsers have the following form:

name: SoapUI
acceptedTechnologies:
- soapui
acceptedMediaTypes:
- application/vnd.opentestfactory.soapui-surefire+xml
matchCases:
- name: syntax full
  regex: ^(?<repository>[^#]+)/(?<project>[^#]+)#(?<suite>[^#]+)#(?<testCase>[^#]+)$
  xpath: //testsuite[@name="{suite}"]/testcase[@name="{testCase}"]
- ...

It is made of 4 required entries, name, acceptedTechnologies, acceptedMediaTypes and matchCases. It may have an optional entry, skippedStatus.

`name`¶

Required The parser name.

`acceptedTechnologies`¶

Required A non-empty list of accepted technologies. The technology name is given by the provider, technology name will be the key to mapping each configuration.

`acceptedMediaTypes`¶

Required A non-empty list of accepted media types: the media type is given by the provider, media type will be verified before parsing a result file.

`matchCases`¶

Required A non-empty list of case definitions.

Each case definition must contain:

a name entry
a regex entry that can match the test reference, you can use capturing groups in it
an xpath entry to find the result status in the surefire file when such regex matches the test reference. It must retrieve the nodes containing the status tag (<failure>, <error> or <skipped>). Captured groups declared in the regex can be used in xpath

Tip

Each reported group in xpath must be written between quotes and braces and only one group should appear between quotes. Use concat if you need two groups in the same attribute:

testcase[@name=concat("{part2}", "$", "{part3}")]

`skippedStatus`¶

Optional The output status when a test is skipped. Either success or error (success by default).

Parsing result computation¶

The parser will compute a status for each test reference. The computed status is one of:

success: the test passed
failure: the test failed
error: the test raised an error or no match was found for the test reference

If the test reference denotes a collection of test cases, the returned status is the aggregation of the individual statuses: if all test cases passed, the collection status is success. If at least one test case raised an error, the collection status is error. Finally, if no test case raised an error but at least one failed, the collection status is failure.

By default, skipped test cases in a collection are considered as in success. This can be modified by using the skippedStatus entry.

A collection that only contains skipped test cases has an error status.

Example¶

In this example you will configure a SoapUI parser that can retrieve the result status of a whole test suite with a test reference like MyRepository/path/to/file#mySuite, but also of a single test case corresponding to a test reference like MyRepository/path/to/file#mySuite#myTestCase.

Given this SoapUI execution report OpenWeatherTest.xml located in a repository Weather:

<testsuite name="OpenWeatherTest.ForecastSuite" 
           tests="2" 
           failures="1"
           errors="0"
           time="0.366">
  <testcase name="ForecastSuccess"
            time="0.31"/>
  <testcase name="MunchenForecastAssertError"
            time="0.056">
    <failure type="Cancelling due to failed test step"
             message="Cancelling due to failed test step">
      <![CDATA[<h3><b>GetMunchenForecast Failed</b></h3><pre>[CheckPopulation] Comparison failed for path [$..population], expecting [[9300]], actual was [[0]]
</pre><hr/>]]>
    </failure>
  </testcase>
</testsuite>

If you want to retrieve the result status of the whole suite ForecastSuite, your test reference will be like Weather/path/to/OpenWeatherTest.xml#ForecastSuite.

The XPath must retrieve the nodes which contain the status. Here it would be the <testcase> nodes. Since a test suite can contain several test cases, you have to retrieve all test case nodes in it. So the XPath should be:

//testsuite[@name[contains(., ".ForecastSuite")]]/testcase

If you want to retrieve the result status of the single test case MunchenForecastAssertError, your test reference will be like Weather/path/to/OpenWeatherTest.xml#ForecastSuite#MunchenForecastAssertError.

The corresponding XPath should then be:

//testsuite[@name[contains(., ".ForecastSuite")]]/testcase[@name="MunchenForecastAssertError"]

so it retrieves the single node <testcase name="MunchenForecastAssertError">...</testcase>.

In the general case, the XPath will be:

//testsuite[@name[contains(., ".mySuiteName")]]/testcase
//testsuite[@name[contains(., ".mySuiteName")]]/testcase[@name="MyTestCaseName"]

Since you need the suite name and the test case name in the XPath, you need to get them from the test reference using capturing groups in the regex regular expression.

^(?<repository>[^#]+)/(?<project>[^#]+)#(?<suite>[^#]+)$
^(?<repository>[^#]+)/(?<project>[^#]+)#(?<suite>[^#]+)#(?<testCase>[^#]+)$

The repository and project groups ensure the test reference is well-formed. The suite and testCase groups enable capturing to use them in the XPath expression.

You can now write the XPath using the captured groups using the {group} syntax:

//testsuite[@name[contains(., ".{suite}")]]/testcase
//testsuite[@name[contains(., ".{suite}")]]/testcase[@name="{testCase}"]

This gives the following match cases:

matchCases:
- name: reference with a test suite and no testcase
  regex: ^(?<repository>[^#]+)/(?<project>[^#]+)#(?<suite>[^#]+)$
  xpath: //testsuite[@name[contains(., ".{suite}")]]/testcase
- name: reference with a test suite and a testcase
  regex: ^(?<repository>[^#]+)/(?<project>[^#]+)#(?<suite>[^#]+)#(?<testCase>[^#]+)$
  xpath: //testsuite[@name[contains(., ".{suite}")]]/testcase[@name="{testCase}"]

Subscriptions¶

The parser service subscribes to the following events:

`kind`	`apiVersion`
`ExecutionResult`	Any

The parser service exposes a series of endpoints that are used by the event bus to post relevant events.

Launch command¶

If you want to manually start the Interpreter service, use the following command:

/bin/bash /app/plugins/interpreter/launch_interpreter.sh