Release schedule and history

Schedule

Starting with 2021-07, a new version of OpenTestFactory is released each month (except August).

Past releases

2023-07 (2023-07-28)

• Workflow retention policies can now be defined, each with a scope, a retention period, and a weight. Weights are used when the total number of workflows is getting larger than the maximum number of storable workflows, in which case the oldest workflows having the smallest weight will be deleted.

• To allow the scope of these policies to be based on the calendar, some functions related to date/time have been added.

• A new test.managed property can be used when defining the scope of a quality gate rule to test if a test is managed by a test referential.

• opentf-ctl get qualitygate and opentf-ctl describe qualitygate have been improved:

  • opentf-ctl get qualitygate reports a global status, thresholds are displayed for each rule, and rules resulting in NOTEST are reported as the other ones.
  • A WARNING message is logged in case of quality gate name or rule name duplication.
  • An ERROR message is logged in case of a quality gate rule having an invalid scope.
  • The help was incomplete, this has been fixed.
  • Error reporting has been homogenized

• Logs (of the Java microservices) have been further improved:

  • Some useless messages have been removed.
  • Some other messages have been downgraded from DEBUG to TRACE.
  • Some messages were missing timestamps, this has been fixed.

• Some other problems have been corrected:

  • opentf-ready was broken due to improper service name registration.
  • The quality gate evaluation was failing for rules with no name. When a rule is unnamed, a UUID is now used as its name.
  • The logs were sometimes erroneously reporting that the quality gate definition file had been modified.
  • opentf-ctl get workflow was sometime losing the last characters of timestamps.

The full list of modifications is available here.

2023-06 (2023-07-11)

• Quality gates can now be defined by providing a list of criteria, each criterion being a minimal success threshold that must be reached for a given test type. For example, a quality gate can require that Postman tests must be 100% successful and Cypress tests at least 90% successful.

• Hooks can now be defined to be applied just before or just after

  • the setup (i.e. the creation of the execution workspace)
  • the teardown (i.e. the deletion of the workspace)

  of a job.
  This allows, for example:

  • using a specific workspace, instead of creating a new one, by specifying this workspace in the before hook of a job setup;
  • preserving the workspace, instead of deleting it, by specifying to keep it in the before hook of a job teardown.

• A generic and configurable Surefire parser replaces the previous technology specific (JUnit, Cucumber, Cypress…) parsers. This parser is configured for the current test technologies, but that configuration can be easily extended for other new test technologies generating Surefire reports. Having a single parser also reduces memory consumption.

• Some new functions are available in expressions:

  • contains;
  • startsWith;
  • endsWith;
  • toJSON;
  • fromJSON.

• The result notification message now contains the list of all test attachments and, for each test, the ID of the attachment used to determine the test status.

• The log indicating that there is no SSH channel or no agent channel for the technology of the test being launched is now an INFO instead of a WARNING.

• Some bugs have been corrected:

  • If a property is undefined in a context, it will now be evaluated as an empty string instead of generating an error.
  • The microservices ported to Micronaut were not properly taking into account the configured debug level.
  • When a workflow was killed, the orchestrator sometimes remained BUSY because one of the microservices (the result aggregator) was improperly handling a workflow cancellation done quickly after its start.
  • Verbatim variables containing a backslash (\) were wrongly escaped on Windows.

The full list of modifications is available here.

2023-05 (2023-05-25)

• The memory consumption of the orchestrator has been reduced: the container should now work with 2GB of RAM instead of the previous 5GB.
  (This has been achieved by migrating some Java microservices to Micronaut.)

• A TAR file containing the screenshots is now created for Cypress tests executed with the cypress/execute@v1 action.
  This is done by using the --config screenshotsFolder=<directory> command line option to store the screenshots in a directory that will be TARed after the test end. Hooks of the cypress/execute@v1 action can get the name of that directory via the OPENTF_CYPRESS_SCREENSHOT_FOLDER environment variable.

• Log improvements:

  • SSH channel plugin logs are less verbose: information messages are no longer reported except if the log level (defined by the DEBUG_LEVEL or SSHCHANNEL_DEBUG_LEVEL environment variables) is set to DEBUG.
  • The orchestrator now assigns generated IDs to steps having no specified one.
  • A notification is generated for each step, it contains the ID, outcome, and conclusion of the step.

• Some bugs have been corrected:

  • Since 2023-04, running several Cucumber 4 tests was resulting in a workflow failure on the second test.
  • Some error conditions, such as trying to run a test that does not effectively exist, are better handled for Robot Framework, Cucumber, and JUnit: the test is reported as failed and the following tests are executed.
    There are still other similar conditions that will be improved in the future.
  • The incorrect subscription of some micro-services, which was the cause of some spurious error messages, has been fixed.
  • Using steps.<step id>.outcome to evaluate the result of a step completed successfully was generating an error.
  • Modifications of a pool definition, done while the SSH channel was busy, were ignored.

The full list of modifications is available here.

2023-04 (2023-04-25)

• The orchestrator now dynamically detects and applies a configuration change for hooks and pools, it is no more necessary to restart it.

• In order to facilitate the detection of an incorrect hook definition, an error message is now logged every time the (improperly) hooked function is executed.

• The orchestrator now supports empty targets and empty pools.

• The output of the /channels and /workflows/{workflow_id}/status endpoints can now be filtered by using a selector.
  The same filtering can be applied when using opentf-ctl get channels, opentf-ctl get workflow, and opentf-ctl run workflow.

• The orchestrator now logs all the environment variables used by the SSH channel plugin.

• When executing a JUnit test, a junit-run-log.txt file containing the complete output of the mvn command is generated. This will help analyse test environment problems resulting in a Maven failure.

• Trying to execute a JUnit test which does not effectively exist will no more interrupt the workflow, the remaining tests will be launched.

• Some bugs have been corrected:

  • Special characters are now disallowed in the tags of a ssh channel.
  • Starting with the 2023-02 delivery, the _and - characters are considered the same in function input parameters, but this should not have been done for patterns. This has been fixed.
  • Using the /agents endpoint to register an agent with no OS tag will now be rejected.
  • Spurious error messages when executing Maven (JUnit and Cucumber) tests, about the presence of target directory, have been removed.

The full list of modifications is available here.

2023-03 (2023-03-28)

• Some endpoints have been added or improved:

  • The new /namespaces endpoint lists the namespaces (accessible by the token used in the GET request).
  • A POST request on the /workflows endpoint can now have a namespace query parameter to specify in which namespace the workflow should run.
  • The new /version endpoint returns the versions of the components of the image.

  opentf-ctl has been updated to support these new features.

• The configuration of the orchestrator when using self-signed certificates has been simplified. It was too complex and not properly documented for the Java microservices.

• The actions/checkout@v3 function has been added: it supports a fetch-depth input defining the number of commits to retrieve when performing git clone, resulting in better cloning performance on large Git repositories. (The actions/checkout@v2 function is still supported.)

• opentf-ctl delete subscription {subscription_id} can now be used to cancel a subscription on the event bus.

• opentf-ctl now supports some short versions of the following options:

  • -w for --wait (or --watch)
  • -s for --step-depth
  • -j for --job-depth
  • -c for --max-command-depth
  • -a for --show-notifications
  • -m for --mode

• A provider will no longer stop if an external hook definition is invalid.

• Some corrections for Robot Framework tests have been done:

  • Problems with special characters in test names have been fixed (these corrections were already started in 2023-02).
  • The squash-tf-serviceslibrary has been rewritten.
    It no longer supports Python 2.
    A bug with improper retrieval of param values containing the % character has been fixed.

• Logs are still further improved :

  • SSH environments are now reported as user@host:port (instead of a hash).
  • Additional information is logged when using these environments:
    • files sent and awaited by an environment;
    • in debug mode, job assignation to/release from an environment.
  • These SSH logs were too verbose in debug mode. Some logs are now generated only if the SSHCHANNEL_DEBUG_LEVELenvironment variable is set to NOTSET or DEBUG.
  • When a PEaC uses an incorrect function input (e.g. a misspelled input name), an error is now logged.
  • Hook definitions are reported in the logs (both in case of correct and incorrect definitions).

The full list of modifications is available here.

2023-02 (2023-03-02)

Warning

If you upgrade the orchestrator to the opentestfactory/allinone 2023-02 image, then you must upgrade the Jenkins plugin to 2.1.0.

• The orchestrator now supports Cypress projects which are anywhere in a Git repository (previously, only Cypress projects at the root of the repository could be used).

• The extra-options input is now supported for all technologies: it has been added to the robotframework/robot, junit/mvntest, and cypress/cypress actions.

• The orchestrator now supports the definition of variables which are not interpreted by the shell (i.e. no expansion nor substitution is performed) by using the verbatim: true option.

• The characters - and _ can now be freely interchanged in the input names (e.g. extra-options and extra_options are the same). - should be preferred.

• In the case of communication failure, the EventBus will retry to publish the event (the maximum number of retries is defined by max_publication_attempts in the EventBus configuration file).

• The orchestrator now uses a WSGI server, Flask is no longer directly accessed (this was the source of the This is a development server. Do not use it in a production deployment. message in the logs).

• Logs are further improved:

  • Additional information is logged when using agents:
    • deregistration of an agent;
    • sent and awaited files by an agent;
    • in debug mode, job assignation to/release from an agent.
  • The EventBug logs are easier to read when the orchestrator in debug mode.

The full list of modifications is available here.

2022-12 (2023-01-03)

This is a maintenance release improving opentf-ctl and the documentation.

The full list of modifications is available here.

2022-10 (2022-11-10)

• In order to be usable with the new Jenkins releases, the Jenkins plugin has been ported to Java 11.

  Warning

  • if you use Jenkins version 2.357 or later, you must use the Plugin for Jenkins 2.0.0 or later;
  • otherwise, you must use a version 1.x.y of the Plugin for Jenkins (which is compatible with versions 2.164.1 up to 2.356 of Jenkins).

• The Jenkins plugin supports stepDepth and jobDepth set to 0, meaning that jobs / steps are displayed for any nested depth.

• The orchestrator now supports Cucumber and JUnit projects which are not at the root of the Git repository.
  The java-param-library has been completed to support such cases.

• The ~= operator has been added to test if a string matches a regular expression.

• The orchestrator supports any character in the value of environment variables, except for the newline character which is disallowed.
  Input parameters are available as environment variables (see the documentation) only if the conversion is possible: length shorter than 2048 characters and no newline character.

• Names of namespaces are trimmed (i.e. leading and trailing space characters are removed).

• Invalid opentf-ctl’s command line parameters are better handled: a meaningful error message is displayed.

The full list of modifications is available here.

2022-09 (2022-09-23)

• The orchestrator can now be installed behind a reverse proxy, see the documentation.

• Hooks can now access the inputs of the provider, either by using the ìnputs context or via INPUT_<VARIABLE_NAME> environment variables.

• PowerShell can now be used in PEaC files.

• Better support of some test technologies

  • SoapUI can now be used on Windows,
  • the inputs of the soapui/soapui@v1 action have been tidied up,
  • the postman/postman@v1 action now supports the extra-options input,
  • for JUnit, @RepeatedTest, @ParameterizedTest, @TestFactory, and @TestTemplate tests are now supported.

• opentf-ctl contains some improvements:

  • the lines of workflow logs are now truncated, the --max-command-length command line option can be used to define the truncation length,
  • some default values for --job-depth, --step-depth, and --max-command-length can now be configured,

  • the configuration file now supports

    • the definition of different endpoints for each service in case routes or ports have been customized, and
    • using force-base-url to force the URL in case of a misconfigured reverse proxy.

The full list of modifications is available here.

2022-07 (2022-07-25)

• This release focuses on improving the logs and the error messages to facilitate troubleshooting, for example when some components have been improperly set up during installation.
  Additionally, the Jenkins plugin reports the orchestrator logs, this reporting can be configured as described here.

• The java-param-library can now be used in Cucumber tests to retrieve the values of the parameters defined in a PEaC.

• Two Docker images are now provided to quickly set up execution environments:

  • the first for Robot Framework,
  • the other for Maven-driven tests (i.e. Cucumber and JUnit).

The full list of modifications is available here.

2022-05 (2022-06-17)

• This release brings a major feature: namespaces and access control.
  Namespaces are an optional mechanism to segregate the resources manipulated by the orchestrator: execution environments, API endpoints, workflows… Once the resources are dispatched on namespaces, it is possible to give permissions to access them: no access, read-only, or read/write. A typical use case is when the orchestrator is shared by several teams, but each one has its own test environments and does not want other teams to access them. It is now possible to limit the access of each team member to the environments, workflows… of her/his team, while the orchestrator administrators have access to everything.

• The new create-archive action can be used in a PEaC file to generate a tar file.

• The new /channels endpoint can be used to get the list of all execution environments (both the ones accessed via ssh or via the agent).

The full list of modifications is available here.

2022-04 (2022-04-22)

This release is a maintenance release containing small improvements:

• opentf-ctl is using much less memory when displaying workflow information (e.g. when using opentf-ctl workflow --wait or opentf-ctl get workflow).

• opentf-agent --version can now be used to get the version of the agent.

• A warning is now displayed when launching the agent if the specified encoding is UTF8 (or if no encoding is specified, as this is the default) while the execution environment is not set with this encoding.

The full list of modifications is available here.

2022-03 (2022-03-30)

This release adds the possibility to use OpenTestFactory as a quality gate in a CI/CD pipeline.

Some other, small, improvements have been done:

• We started to improve log management: some microservices (the ones used to drive tests) are reporting more information. This information is available in opentf-ctl get workflow and in the Jenkins plugin.

• The event retention period of the Observer can now be configured with retention_period_minutes (see the documentation).

• When running a Cucumber test, a report.json file was generated, but its content was not some JSON data, but, instead, the Cucumber terminal output. This file is no more generated.

• When running a Robot Framework test, a tar file containing the report.html and log.html files, and the screenshots generated during the test, is now generated.

• When investigating problems, the channel_id is now available in the logs.

• opentf-ctl view token TOKEN can be used to display the payload of a token. opentf-ctl check token TOKEN using PUBLIC_KEY_PATH can be used to validate a token.

• opentf-ctl get workflow has been extended with the --output=format option, so the returned information can be formatted in JSON or YAML.

• The agent now verifies that the provided OS tag corresponds to the operating system where it is running. Additionally, it validates the syntax of the provided orchestrator URL.

The full list of modifications is available here.

2022-02 (2022-02-25)

This is a maintenance release fixing some bugs and preparing the quality gate that will be added in 2022-03.

The full list of modifications is available here.

2022-01 (2022-01-24)

This is a maintenance release fixing some bugs and including some small improvements:

• more information and clearer messages in the logs
• agent
  • increased performance
  • possible disabling of certificate verification
  • support of private certificates
• opentf-ctl
  • opentf-ctl get workflows displays more columns (ID, status, first timestamp, name)
  • opentf-ctl now supports get agents and delete agent commands to list and kill agents

The full list of modifications is available here.

2021-12 (2021-12-23)

• The opentf-ctl tool has been enhanced with the following features:

  • the config command can be used to generate or modify a configuration file;
  • the version command displays the tools version;
  • the get agents command lists the registered agents;
  • environment variables with an OPENTF_RUN_ prefix are automatically provided to the workflow execution environment;
  • all workflow execution errors are now reported for get workflow and run workflow --wait;
  • better error messages for invalid config files.

• The Jenkins plugin has been open-sourced (it was previously part of the Squash code base): https://gitlab.com/henixdevelopment/open-source/opentestfactory/jenkins-plugin

The full list of modifications is available here.

2021-11 (2021-11-16)

• Added support of Postman.

• An Allure report is now produced for each executed PEaC.

2021-10 (2021-10-21)

• Added support for properly starting and stopping the orchestrator.

• Added tools to help manage the orchestrator:

  • opentf-ready to wait until the orchestrator is ready to accept workflows.

  • opentf-done to wait until the orchestrator can be safely stopped (i.e. it has no more pending tasks).

  • opentf-ctl that can be used to

    • start a workflow;
    • stop a workflow;
    • cancel a workflow;
    • generate a signed token;
    • list the subscriptions on the event bus.

• Added the put-file action to put a file in the execution environment (see the action description).

2021-09 (2021-09-09)

• Added the Java Param Library
  This library provides an API usable in Java tests to retrieve the values of the parameters defined in a PEaC.
• Added the support of hooks
  Hooks allow the definition of custom actions to be performed before and/or after a given action of the orchestrator: retrieval of non-standard test reports, starting/stopping a service required by the tests, logging the start and the end of the tests… These hooks can be defined either in a PEaC to be applied to the corresponding workflow or in a configuration file of a provider to be applied to all workflows.
  See the hooks guide for more information.

2021-07 (2021-07-16)

• Added support of SKF (Squash Keyword Framework)

1.1.0 (2021-06-04)

• Improved logs in Jenkins plugin

1.0.0 (2021-04-23)

Initial release

• basic orchestrator mechanisms: reception and execution of a PEaC, stopping a PEaC
• retrieval of tests in Git repositories
• launch of tests and parsing of results for JUnit, Robot Framework, Cucumber, Cypress, SoapUI
• generation of Allure reports (can be published only in Squash TM for the time being)
• Jenkins plugin