Release schedule and history

Schedule

A new version of OpenTestFactory is released each month (except August and, possibly, if the previous release was delivered very late).

Past releases

2024-11 (2024-12-03)

• opentf-ctl commands (and the related endpoints) accepting selectors (e.g., with the options --field-selector, --having, --selector) now accept colons (:) in the values of these ones.
• A new --having option has been added to the get workflows command, allowing users to filter the list based on some specific workflow events. More information can be found here.
• opentf-ctl now supports auto-completion for bash. This feature enables auto-completion for commands, options, and for dynamic parameters such as ids. More information can be found here.
• The SoapUI provider now supports parameters. Transmitted parameters are available through project properties (using the ${#Project#parameter_name} syntax). More information can be found here.
• Robot Framework will now stop generating Allure attachments if the environment variable OPENTF_ALLURE_ENABLED is not correctly set. This should improve the execution time in this specific case.
• A new action actions/download-artifact@v1 enables users to download any artifact defined at the workflow level (e.g., artifacts uploaded using the actions/upload-artifact@v1). This new action supports three optional parameters, name, pattern, and path, as described here.
• The behavior of OPENTF_VARIABLES has been improved to ensure that these variables are accessible in the variables context. This means that a variable set in a step can be used later in the workflow, for example for a condition. More information can be found here.
• Following the previous release, additional optimizations have been made to reduce the number of steps generated for some providers (Cucumber, Cypress, JUnit, Playwright, Postman, SKF, and SoapUI), potentially improving the overall execution time.
• Memory usage for the insight cache has been improved by applying filters to the cached data. This should improve the time necessary to generate insight reports, especially for longer workflows.
• It is now possible to customize the live execution-log level. This is done through three labels set in the workflow: live-executionlog/step-depth, live-executionlog/job-depth, and live-executionlog/max-command-length. These labels interact with the GET /workflows/${workflow_id}/logs endpoint. More information can be found here.

Several bugs have also been corrected:

• An agent executing long-running steps could appear as UNREACHABLE when the step execution exceeded the LIVENESS_PROBE parameter. The agent now remains as BUSY during such scenarios.
• Failure details in HTML reports were not correctly formatted for some technologies, and the output did not match what was shown in the console. This has been corrected to ensure consistent formatting.
• Previously, after manually canceling a workflow, the workflow continued executing steps from hooks before stopping correctly. The process has been improved, ensuring proper cancellation. More information can be found here.
• Previously, an insight report could be generated before all the results were available. This has now been fixed, the generation will now only start after all the workflow’s workers have completed.

The full list of modifications is available here.

2024-10 (2024-10-25)

• A new endpoint GET /workflows/{workflow_id}/logs is now available to allow the retrieval of the logs of a specific workflow. More information can be found here.
• A new action actions/upload-artifact@v1 is now available to allow the publication of a file at the workflow’s level. More information can be found here.
• The environment variable OPENTF_ALLURE_ENABLED is now available to enable the Allure report generation. By default, the Allure report is now disabled. More information can be found here.
• Some improvements have been made on the subscription of plugins, most notably to filter the messages to which a plugin can subscribe. This means that every plugin should only receive the messages it is interested in, leading to an improvement in performance.
• Timestamps are now in UTC with the timezone explicitly set.
• The actionprovider provider is now added to the aggregated plugins like the other providers. It is now launched through the allinone service.
• The notifications have been optimized. Some of them were not used anymore, therefore they have been removed.
• A new endpoint PATCH /agents is now available. This is a preparation step for a new feature that will be available later. More information can be found here.
• A new --as option is now available for the opentf-ctl generate report command to allow a user to specify a name for the generated report. More information can be found here.
• A new --report option is now available for the opentf-ctl run workflow, allowing users to save a local copy of the report after execution. More information can be found here.

Several bugs have also been corrected:

• With Playwright, having double quotes around the test case name in the test reference used to break the execution on Linux. This has now been fixed.
• If a workflow contained multiple jobs and a hook calling a function, the arranger could break with an internal error. This has now been fixed.
• A workflow waiting for a ProviderResult could be stuck without being cancellable. This has now been fixed.
• When the folder containing public keys had a file from another format, some services could be stuck with an internal error. This has now been fixed.
• In some cases, the filter in the test case section of the HTML report was not working correctly. This has now been fixed.
• After an opentf-ctl generate report command, the observer logged a few warnings about an unknown worker. This has been fixed.
• A workflow containing no test results previously logged a few errors from the InsightCollector. The log level has been adjusted to improve the logs.
• When calling the qualitygate with a workflow containing no test results, the qualitygate will now log a new message indicating that no test cases were found.
• While publishing results to GitLab in a non-MR context, the error message shown was unclear. This has been improved.

The full list of modifications is available here.

2024-09 (2024-09-23)

• Depending on the test technology, a workflow could be stopped if one of the tests had an incorrect test reference. The behavior is now consistent, it will continue on the remaining tests even if one is blocked for a technical reason.
• Context parameters usually defined in a service configuration file (e.g. max_jobs for the arranger) can now be set from an environment variable using the {SERVICE_NAME}_{PARAMETER_NAME} syntax. The available ones for each service are detailed in their own service configuration page. Services can also configure the waitress_threads_count via the {SERVICE_NAME}_WAITRESS_THREADS_COUNT environment variable. Default value is 4, except for the event bus and agent channel handler (10).
• The killswitch service has been updated to use the WorkflowCancellation message. This is used to differentiate a user-initiated cancellation from a technical cancellation (e.g. an error in the workflow). Additionally, it makes it possible to add a comment (reason and source) and supports a dryRun query parameter. If provided, the reason is also available in the WorkflowCancel message. More information can be found here. The kill workflow command has also been updated in consequence.
• The eventbus now quarantines per endpoint instead of per subscription. It no longer attempts to resend on 4xx errors (except 429) and uses a throttling delay instead of a final quarantine. Additionally, event re-serialization has been removed to reduce unnecessary processing.
• The agent protocol has been improved to reduce, in some cases, the commands necessary to execute an action. This means that there will be fewer requests between the orchestrator and the agent, potentially improving slightly the execution time. The agentchannel now also handles up to 10 requests at once and prevents ghost channels that could appear after a mass de-registration. This feature will be available for agents 1.8 or higher, with backward compatibility for older agents.
• Action provider and Robot Framework provider have been optimized by reducing the number of generated steps. This means that there will be fewer back and forth between the orchestrator and the execution environment, potentially improving the execution time.
• The publications now occur in their own threads. This improves the performance under load when using agents.
• The S3 publisher plugin now skips the validation of WorkflowResult and ExecutionCommand events that have no attachments. This implies a performance improvement.
• In some cases it may happens that the publication was not done correctly. The insight collector used to register at the end of a workflow. This is not the case anymore, it is now done at the beginning.
• Generators now fail specific jobs and propagate errors as needed, without canceling the entire workflow.
• Logs can now be redirected by setting the OPENTF_LOGGING_REDIRECT environment variable. This allows you to specify an alternative logging destination instead of the default sys.stderr.
• It is now possible to add an outputs section for provider and generator plugins. Outputs from previous jobs can now be referenced in workflows, and the WorkflowCompleted and WorkflowCanceled events include an outputs section if defined. More details can be found here.
• A job can now be ran without a runs-on entry. The job will then take any environment available.
• When executing on Windows, the execution log could contain a few unnecessary logs caused by certutil. They have been removed.
• An error message will now be shown if a job is referencing itself.
• A workflow without any test result used to raise a few errors in the insightcollector. This is not the case anymore.
• Plugin helpers now provide an asynchronous dispatch queue. It retries failed publications with increasing delays up to 60 seconds until successful.
• Channel handlers now filter out irrelevant ExecutionCommand messages. Additionally, generator handlers now log debug messages.
• The qualitygate now handles the new 202 response from the datasource endpoint.
• Removed the unused metadata.step_origin_status field from ExecutionCommand events for channel releases.
• Tokens can now be generated non-interactively using command-line options by specifying all parameters explicitly, with options for algorithm, issuer, subject, expiration, and output file. More information can be found here.
• The get qualitygate and get datasources commands have been updated to handle the new 202 response code. A new timeout option is also available on both commands. More information can be found here for get qualitygate and here for get datasource.
• A new timeout option is now available for the generate report command. More information can be found here.

Several bugs have also been corrected:

• The insightcollector could time out with larger workflows. A cache has been implemented to improve this.
• Channel setup hooks were executed after the job’s steps. This has now been fixed.
• Workflows referencing an unknown generator that previously remained stuck in RUNNING status until timeout can now be killed.
• The HTML report tables now handle cases where there are no matching test cases gracefully. Internal errors have been resolved, and additional debug information is provided if no data is found.
• The workflow is now cancelled if the number of jobs exceeds the max_jobs value specified in the arranger configuration file.
• Error handling for steps containing multiple ::attach:: workflow commands has been improved. Previously, if one of the commands failed, the error was not properly managed. This issue is now resolved.
• Dependency chains in generator jobs were not handled correctly. This could cause a workflow to be stuck as RUNNING. This is now fixed.
• A job is no longer allowed to include itself in its needs entry. This change prevents internal errors caused by such configurations.

The full list of modifications is available here.

2024-07 (2024-08-13)

• The orchestrator now allows parallel executions. This can be specified through the strategy.max-parallel keyword in a workflow. More information available here.
• New sections are now available for HTML reports to improve the exploitation of the execution time, such as BarCharts, ProgressBar, ScatterChart. The default one has also been updated with BarCharts and ProgressBar. More information available here.
• The orchestrator now allows jobs which do not have any dependencies to be selected as candidates for execution and no longer have a forced dependency with other jobs.
• To enhance compatibility with GitLab, testcase items now consist of unique names.
• A details.reason was added to WorkflowCanceled events to give more information about the origin of the cancellation.
• The qualitygate is now a plugin of the orchestrator to make use of the new datasource endpoint.
• Status functions (success(), failure(), cancelled(), and always()) are now only allowed in conditionals and insight model selection. More information available here.
• The workflow schema has been updated due to new/future components. It is now possible to set the timeout directly in the workflow through the timeout-minutes keyword. More information here.
• Service allinone logs now expose their origin services to improve the readability.
• Jobs now have access to the status skipped, for example, if a job has an unmet conditional.
• Tools integrate now new options --warmup-delay, --polling-delay and --max-retry to improve opentf-tools usage in various CI scenario. The option --tags has also been added to the run workflow command to add additional tags to a workflow execution. More information here and here.
• A new option --dry-run has been added to the opentf-ctl run workflow command on the receptionist endpoint. This helps to check if the workflow is valid without creating a new run workflow. More information is available here.

Several bugs have also been corrected:

• Multi-line variables are now correctly handled.
• For Postman some special characters broke the command line, such as double quotes for Linux and percentage signs for Windows. This is now secured.
• A step with a missing required input will now be considered as a step failure rather than an error that cancels the workflow.
• The orchestrator now correctly handles expression syntax on runs-on in a workflow.
• The command opentf-ctl get workflows encountered some performance issues when there was a certain quantity of workflows. The command is now faster to respond.
• Some services were missing in the command opentf-ctl config generate. This is now fixed.
• The JSON/YAML outputs now have a proper format, and when a workflow without any metadata.creationTimestamp entry is running, it no longer generates an error.
• The start of a generator will now be logged as a generator job rather than a function.

The full list of modifications is available here.

2024-05 (2024-06-19)

• The orchestrator now supports the Playwright technology, including new params, execute, and npx actions. This new implementation integrates the parsing of Playwright’s Surefire reports. More information is available here.
• The HTML report has been enhanced by incorporating detailed failure information.
• A new endpoint on the observer, GET /workflows/{workflow_id}/datasources/{datasource_name}, has been added. This feature displays information related to the user-provided data sources: testcases, jobs, and tags.
• The command opentf-ctl get datasource --kind {kind} has been added, this feature introduces the capability to retrieve data sources from a specific workflow.
• The Insight Collector service now produces an XML report, which can be downloaded as an artifact from a GitLab pipeline to generate a test results block in a merge request.
• A new status, UNREACHABLE, has been implemented for both SSH and agent environments. The channels service now checks the availability of these environments at regular intervals, which can be modified by adjusting the availability_check_delay property. An environment marked as UNREACHABLE will not be considered a candidate when requesting an environment.
• The launcher manifest is now configurable via the OPENTF_LAUNCHERMANIFEST variable making it possible to enable or disable chosen plugins.
• For enhanced security, the default user of the orchestrator image has been changed from root to squash.

A bug has also been corrected:

• The term “test cases” in the HTML report has been corrected.

The full list of modifications is available here.

2024-03 (2024-04-25)

• Compatibility issues in Orchestrator version 2023-12, which previously caused an error message stating “The current machine does not support all of the required CPU features by the image”, have now been resolved.
• The orchestrator can now generate an HTML report summarizing the test execution outcomes. This feature is configurable through the Insight Collector Configuration file. Multiple reports can be set up. More information can be found in the Summary Reports documentation.
• The opentf-ctl generate report {workflow_id} using {definition_file} command has been added, allowing the generation of HTML reports based on a user-provided local configuration file.
• Insight Collector has a new user-facing endpoint: POST /workflows/<workflow_id>/insights. This endpoint is used by previous opentf-ctl command.
• The test result notifications in workflows have been updated to include duration and failure details.
• The Robot Framework parser now supports “Skipped” and “Not run” step statuses.
• The attachments produced by workflows such as the execution log or Allure report now have their UUIDs which are displayed in the attachments list and can be used to retrieve these attachments. The UUID of the Allure report is the same as the workflow UUID to simplify attachment retrieval.

Several bugs have also been corrected:

• If different provider hooks containing OTF actions are specified, the aggregator service no longer merges them.
• The parameters containing special characters used to break the Cypress provider script. They have now been secured.
• If the Interpreter or Allure collector receive an incorrect document, they handle the error without generating a stack trace with DocumentBuildingException/SAXParseException.
• Error handling has been implemented to prevent the generation of a stack trace when the receptionist receives a malformed YAML file.
• Fix Postman parser error when test reference contains a single quote.

The full list of modifications is available here.

2024-01 (2024-02-14)

• A new insightcollector service is available to allow users to configure the execution log automatically generated after every workflow execution. This is done through a configuration file set by the INSIGHTCOLLECTOR_CONFIGURATION environment variable.
• The receptionist service can now be pinged or requested in dry-run mode on the POST /workflows endpoint. This can be used to test new deployments or check if a workflow is correct or not.
• JUnit @Nested tests are now supported. A new test reference syntax has been introduced to deal with them.
• A new allinone service is available to manage providers. This is used primarily to reduce the memory consumption.
• After a Robot Framework execution, if the Allure library did not generate *-attachment.html files, then, it may have resulted in an incorrect log that could confuse users. The get-files action has also been improved with a new skip-if-not-found option to remove the error message if no files were found.

Several bugs have also been corrected:

• Some Windows error codes could exceed the maximum value accepted by a 32-bits integer, causing an error in the Java services.
• A Cypress test without its own test suite was not interpreted correctly by the parser.
• Some logs (in TRACE) not related to a workflow were sent like they were, causing some workflows to appear again (with name and status set to <none>) after they were cleaned up in the workflow list.

The full list of modifications is available here.

2023-12 (2023-12-18)

• The opentf-ctl run workflow command now has access to most parameters of opentf-ctl get qualitygate. This allows the user to execute directly the quality gate when the workflow is done.
• A new command opentf-ctl get attachments is now available to print a list of every attachment available for a specific workflow.
• The command opentf-ctl cp has been improved:
  • The attachment UUID can now be shortened, it does not have to be complete anymore.
  • Patterns can now be used to retrieve multiple files.
  • A type can now be specified to retrieve only the files matching it.
• For Robot Framework, Cucumber, Cypress, Postman, and JUnit, additional options can now be added to the command line launching the tests, when using the executeaction, via {PROVIDER}_EXTRA_OPTIONS environment variables.
• A new endpoint HEAD /workflows/{workflow_id}/files/{attachment_id} has been added to the localstore to check if a file exists or not.
  The localstore now also checks if a file exists first to prevent some cases where it tried to delete non-existing ones.
• The docker images for executing Cucumber JVM and JUnit and Robot Framework tests have been updated.
• The opentestfactory/allinone image now uses Java 17 (previously, it used Java 11).

Some bugs have been corrected:

• An execution environment could wrongly reject a workflow if it had one namespace where the token used to launch the workflow was not authorized.
• The Cypress interpreter had issues to detect the correct test case when the test was executed on Windows.

The full list of modifications is available here.

2023-10 (2023-11-13)

• A new plugin tracker-publisher has been added. Its purpose is to publish the quality gate results in a CI/CD pipeline (currently only the publication in GitLab issues and merge requests is supported).
  opentf-ctl has been completed to enable this publication.
• A new endpoint GET /workflows/{workflow_id}/files/{attachment_id} has been added. It allows the retrieval of attachments generated during a workflow execution.
  opentf-ctl has been completed accordingly: opentf-ctl cp copies a workflow attachment to the specified destination.
  This endpoint is handled by a new localstore service (replacing the localcleaner service).
• The inception mode now supports defining attachment using wildcards (*, ?…).
• The environment variables are now logged in alphabetical order at the startup.
• Events (ExecutionCommand and ExecutionResult) signaling a channel release now include a new step_count attribute in the metadata section, providing the total executed steps in a job.

Several bugs have been corrected:

• The error message displayed when the prepare-inception function references an unknown resource has been clarified.
• Accessing a non-existent object in an expression could generate a Python stacktrace. This case is now handled correctly, a more explicit error message is printed.
• The startup script now waits that the EventBus is up and running before launching the plugins.

The full list of modifications is available here.

2023-09 (2023-09-27)

• Endpoints have been improved:

  • GET /workflows supports filtering on workflows statuses: RUNNING, FAILED, or DONE.
  • GET /subscriptions, GET /channels, GET /workflows, and GET /agents allow field and label selectors.
    (In consequence, the get …, kill workflow, delete subscription, and delete agent sub-commands of opentf-ctl now support the --selector/-l and --field-selector flags, the last three ones support additionally the --all flag.)
  • Endpoints supporting query parameters return the names of these ones in the X-Processed-Query HTTP header.

• opentf-ctl’s sub-commands have been improved:

  • opentf-ctl generate token using allows specifying an expiration date for the generated token and will ask for a passphrase if the specified private key is encrypted.
  • get workflows, get channels, get agents, get qualitygate, and get subscriptionsnow return fixed-width columns (instead of comma-separated values).
    They can also generate YAML (-o yaml) or JSON (-o json).
    They can report additional information (--output=wide) and specified data (--output=custom-columns=).
  • Options are specifiable in any order, allowing the UUID to be specified last.
  • UUIDs can be abbreviated.

• A dead letter queue has been implemented: some deadletter threads will try to redispatch publications whose first dispatch failed.
  It is possible to configure:

  • the delay before considering that a thread failed to respond,
  • the number of deadletter threads, and
  • the maximum number of retries.

• Hooks can now access the tags defined in the runs-on job property.

• The HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables are now taken into account by the Java micro-services.

• The command line used to launch JUnit tests has been modified to work properly with Maven daemon.

• Lines can now be commented out (using #) in a trusted key authorization file, a token authorization file, or an authorization policy file.

• In the Surefire parser configuration, the default value of the skippedStatus property is now success and an error is reported if that property has an invalid value.

• Messages and logs have been further improved:

  • The names of the Java microservices are now indicated in their log messages.
  • The help messages of opentf-ctl have been slightly completed and cleaned up.
  • More information is reported in case of quality gate usage errors.
  • A workflow completion, cancellation, or failure is logged as an INFO message, a timeout as a WARNING one.
  • The PYTHONPATH value and the image components are logged during the orchestrator startup.
  • A spurious message during a job teardown on Windows has been removed.
  • Missing timestamps have been added.
  • Some DEBUG logs have been moved to the TRACE level.
  • Some useless information has been removed.
  • …

Several bugs have been corrected:

• Some valid boolean expressions were generating errors.
• get-files issues:
  • It was not properly handling some cases where no files were matching the provided pattern.
  • When used with patterns such as **/*, it was improperly setting directories as attachments (it should only attach files).
  • It was failing for some patterns such as **/x*.ext.
• The orchestrator now prepends the name of each attachment with its UUID in order to mitigate file name collision.
• warn-if-not-found can now be used to specify an error message for the create-archive action, so it is now consistent with get-files action behavior.
• The create-archive action was generating an invalid empty tar file when there were no files to archive. It now generates a tar file containing an intentionally_empty_archive empty file.
• A possible race condition resulted in some attachments being never deleted.
• When using the GET /qualitygate endpoint, if mode is not present, strict is now assumed.
• The failure of a before step in a job hook was resulting in the workspace being cleaned up.
• Keys in the status part of subscription manifests were incorrectly set as integers; they are now strings.

The full list of modifications is available here.

2023-07 (2023-07-28)

• Workflow retention policies can now be defined, each with a scope, a retention period, and a weight. Weights are used when the total number of workflows is getting larger than the maximum number of storable workflows, in which case the oldest workflows having the smallest weight will be deleted.

• To allow the scope of these policies to be based on the calendar, some functions related to date/time have been added.

• A new test.managed property can be used when defining the scope of a quality gate rule to test if a test is managed by a test referential.

• opentf-ctl get qualitygate and opentf-ctl describe qualitygate have been improved:

  • opentf-ctl get qualitygate reports a global status, thresholds are displayed for each rule, and rules resulting in NOTEST are reported as the other ones.
  • A WARNING message is logged in case of quality gate name or rule name duplication.
  • An ERROR message is logged in case of a quality gate rule having an invalid scope.
  • The help was incomplete, this has been fixed.
  • Error reporting has been homogenized

• Logs (of the Java microservices) have been further improved:

  • Some useless messages have been removed.
  • Some other messages have been downgraded from DEBUG to TRACE.
  • Some messages were missing timestamps, this has been fixed.

• Some other problems have been corrected:

  • opentf-ready was broken due to improper service name registration.
  • The quality gate evaluation was failing for rules with no name. When a rule is unnamed, a UUID is now used as its name.
  • The logs were sometimes erroneously reporting that the quality gate definition file had been modified.
  • opentf-ctl get workflow was sometime losing the last characters of timestamps.

The full list of modifications is available here.

2023-06 (2023-07-11)

• Quality gates can now be defined by providing a list of criteria, each criterion being a minimal success threshold that must be reached for a given test type. For example, a quality gate can require that Postman tests must be 100% successful and Cypress tests at least 90% successful.

• Hooks can now be defined to be applied just before or just after

  • the setup (i.e. the creation of the execution workspace)
  • the teardown (i.e. the deletion of the workspace)

  of a job.
  This allows, for example:

  • using a specific workspace, instead of creating a new one, by specifying this workspace in the before hook of a job setup;
  • preserving the workspace, instead of deleting it, by specifying to keep it in the before hook of a job teardown.

• A generic and configurable Surefire parser replaces the previous technology specific (JUnit, Cucumber, Cypress…) parsers. This parser is configured for the current test technologies, but that configuration can be easily extended for other new test technologies generating Surefire reports. Having a single parser also reduces memory consumption.

• Some new functions are available in expressions:

  • contains;
  • startsWith;
  • endsWith;
  • toJSON;
  • fromJSON.

• The result notification message now contains the list of all test attachments and, for each test, the ID of the attachment used to determine the test status.

• The log indicating that there is no SSH channel or no agent channel for the technology of the test being launched is now an INFO instead of a WARNING.

• Some bugs have been corrected:

  • If a property is undefined in a context, it will now be evaluated as an empty string instead of generating an error.
  • The microservices ported to Micronaut were not properly taking into account the configured debug level.
  • When a workflow was killed, the orchestrator sometimes remained BUSY because one of the microservices (the result aggregator) was improperly handling a workflow cancellation done quickly after its start.
  • Verbatim variables containing a backslash (\) were wrongly escaped on Windows.

The full list of modifications is available here.

2023-05 (2023-05-25)

• The memory consumption of the orchestrator has been reduced: the container should now work with 2GB of RAM instead of the previous 5GB.
  (This has been achieved by migrating some Java microservices to Micronaut.)

• A TAR file containing the screenshots is now created for Cypress tests executed with the cypress/execute@v1 action.
  This is done by using the --config screenshotsFolder=<directory> command line option to store the screenshots in a directory that will be TARed after the test end. Hooks of the cypress/execute@v1 action can get the name of that directory via the OPENTF_CYPRESS_SCREENSHOT_FOLDER environment variable.

• Log improvements:

  • SSH channel plugin logs are less verbose: information messages are no longer reported except if the log level (defined by the DEBUG_LEVEL or SSHCHANNEL_DEBUG_LEVEL environment variables) is set to DEBUG.
  • The orchestrator now assigns generated IDs to steps having no specified one.
  • A notification is generated for each step, it contains the ID, outcome, and conclusion of the step.

• Some bugs have been corrected:

  • Since 2023-04, running several Cucumber 4 tests was resulting in a workflow failure on the second test.
  • Some error conditions, such as trying to run a test that does not effectively exist, are better handled for Robot Framework, Cucumber, and JUnit: the test is reported as failed and the following tests are executed.
    There are still other similar conditions that will be improved in the future.
  • The incorrect subscription of some micro-services, which was the cause of some spurious error messages, has been fixed.
  • Using steps.<step id>.outcome to evaluate the result of a step completed successfully was generating an error.
  • Modifications of a pool definition, done while the SSH channel was busy, were ignored.

The full list of modifications is available here.

2023-04 (2023-04-25)

• The orchestrator now dynamically detects and applies a configuration change for hooks and pools, it is no more necessary to restart it.

• In order to facilitate the detection of an incorrect hook definition, an error message is now logged every time the (improperly) hooked function is executed.

• The orchestrator now supports empty targets and empty pools.

• The output of the /channels and /workflows/{workflow_id}/status endpoints can now be filtered by using a selector.
  The same filtering can be applied when using opentf-ctl get channels, opentf-ctl get workflow, and opentf-ctl run workflow.

• The orchestrator now logs all the environment variables used by the SSH channel plugin.

• When executing a JUnit test, a junit-run-log.txt file containing the complete output of the mvn command is generated. This will help analyse test environment problems resulting in a Maven failure.

• Trying to execute a JUnit test which does not effectively exist will no more interrupt the workflow, the remaining tests will be launched.

• Some bugs have been corrected:

  • Special characters are now disallowed in the tags of a ssh channel.
  • Starting with the 2023-02 delivery, the _and - characters are considered the same in function input parameters, but this should not have been done for patterns. This has been fixed.
  • Using the /agents endpoint to register an agent with no OS tag will now be rejected.
  • Spurious error messages when executing Maven (JUnit and Cucumber) tests, about the presence of target directory, have been removed.

The full list of modifications is available here.

2023-03 (2023-03-28)

• Some endpoints have been added or improved:

  • The new /namespaces endpoint lists the namespaces (accessible by the token used in the GET request).
  • A POST request on the /workflows endpoint can now have a namespace query parameter to specify in which namespace the workflow should run.
  • The new /version endpoint returns the versions of the components of the image.

  opentf-ctl has been updated to support these new features.

• The configuration of the orchestrator when using self-signed certificates has been simplified. It was too complex and not properly documented for the Java microservices.

• The actions/checkout@v3 function has been added: it supports a fetch-depth input defining the number of commits to retrieve when performing git clone, resulting in better cloning performance on large Git repositories. (The actions/checkout@v2 function is still supported.)

• opentf-ctl delete subscription {subscription_id} can now be used to cancel a subscription on the event bus.

• opentf-ctl now supports some short versions of the following options:

  • -w for --wait (or --watch)
  • -s for --step-depth
  • -j for --job-depth
  • -c for --max-command-depth
  • -a for --show-notifications
  • -m for --mode

• A provider will no longer stop if an external hook definition is invalid.

• Some corrections for Robot Framework tests have been done:

  • Problems with special characters in test names have been fixed (these corrections were already started in 2023-02).
  • The squash-tf-serviceslibrary has been rewritten.
    It no longer supports Python 2.
    A bug with improper retrieval of param values containing the % character has been fixed.

• Logs are still further improved :

  • SSH environments are now reported as user@host:port (instead of a hash).
  • Additional information is logged when using these environments:
    • files sent and awaited by an environment;
    • in debug mode, job assignation to/release from an environment.
  • These SSH logs were too verbose in debug mode. Some logs are now generated only if the SSHCHANNEL_DEBUG_LEVELenvironment variable is set to NOTSET or DEBUG.
  • When a PEaC uses an incorrect function input (e.g. a misspelled input name), an error is now logged.
  • Hook definitions are reported in the logs (both in case of correct and incorrect definitions).

The full list of modifications is available here.

2023-02 (2023-03-02)

Warning

If you upgrade the orchestrator to the opentestfactory/allinone 2023-02 image, then you must upgrade the Jenkins plugin to 2.1.0.

• The orchestrator now supports Cypress projects which are anywhere in a Git repository (previously, only Cypress projects at the root of the repository could be used).

• The extra-options input is now supported for all technologies: it has been added to the robotframework/robot, junit/mvntest, and cypress/cypress actions.

• The orchestrator now supports the definition of variables which are not interpreted by the shell (i.e. no expansion nor substitution is performed) by using the verbatim: true option.

• The characters - and _ can now be freely interchanged in the input names (e.g. extra-options and extra_options are the same). - should be preferred.

• In the case of communication failure, the EventBus will retry to publish the event (the maximum number of retries is defined by max_publication_attempts in the EventBus configuration file).

• The orchestrator now uses a WSGI server, Flask is no longer directly accessed (this was the source of the This is a development server. Do not use it in a production deployment. message in the logs).

• Logs are further improved:

  • Additional information is logged when using agents:
    • deregistration of an agent;
    • sent and awaited files by an agent;
    • in debug mode, job assignation to/release from an agent.
  • The EventBug logs are easier to read when the orchestrator in debug mode.

The full list of modifications is available here.

2022-12 (2023-01-03)

This is a maintenance release improving opentf-ctl and the documentation.

The full list of modifications is available here.

2022-10 (2022-11-10)

• In order to be usable with the new Jenkins releases, the Jenkins plugin has been ported to Java 11.

  Warning

  • if you use Jenkins version 2.357 or later, you must use the Plugin for Jenkins 2.0.0 or later;
  • otherwise, you must use a version 1.x.y of the Plugin for Jenkins (which is compatible with versions 2.164.1 up to 2.356 of Jenkins).

• The Jenkins plugin supports stepDepth and jobDepth set to 0, meaning that jobs / steps are displayed for any nested depth.

• The orchestrator now supports Cucumber and JUnit projects which are not at the root of the Git repository.
  The java-param-library has been completed to support such cases.

• The ~= operator has been added to test if a string matches a regular expression.

• The orchestrator supports any character in the value of environment variables, except for the newline character which is disallowed.
  Input parameters are available as environment variables (see the documentation) only if the conversion is possible: length shorter than 2048 characters and no newline character.

• Names of namespaces are trimmed (i.e. leading and trailing space characters are removed).

• Invalid opentf-ctl’s command line parameters are better handled: a meaningful error message is displayed.

The full list of modifications is available here.

2022-09 (2022-09-23)

• The orchestrator can now be installed behind a reverse proxy, see the documentation.

• Hooks can now access the inputs of the provider, either by using the ìnputs context or via INPUT_<VARIABLE_NAME> environment variables.

• PowerShell can now be used in PEaC files.

• Better support of some test technologies

  • SoapUI can now be used on Windows,
  • the inputs of the soapui/soapui@v1 action have been tidied up,
  • the postman/postman@v1 action now supports the extra-options input,
  • for JUnit, @RepeatedTest, @ParameterizedTest, @TestFactory, and @TestTemplate tests are now supported.

• opentf-ctl contains some improvements:

  • the lines of workflow logs are now truncated, the --max-command-length command line option can be used to define the truncation length,
  • some default values for --job-depth, --step-depth, and --max-command-length can now be configured,

  • the configuration file now supports

    • the definition of different endpoints for each service in case routes or ports have been customized, and
    • using force-base-url to force the URL in case of a misconfigured reverse proxy.

The full list of modifications is available here.

2022-07 (2022-07-25)

• This release focuses on improving the logs and the error messages to facilitate troubleshooting, for example when some components have been improperly set up during installation.
  Additionally, the Jenkins plugin reports the orchestrator logs, this reporting can be configured as described here.

• The java-param-library can now be used in Cucumber tests to retrieve the values of the parameters defined in a PEaC.

• Two Docker images are now provided to quickly set up execution environments:

  • the first for Robot Framework,
  • the other for Maven-driven tests (i.e. Cucumber and JUnit).

The full list of modifications is available here.

2022-05 (2022-06-17)

• This release brings a major feature: namespaces and access control.
  Namespaces are an optional mechanism to segregate the resources manipulated by the orchestrator: execution environments, API endpoints, workflows… Once the resources are dispatched on namespaces, it is possible to give permissions to access them: no access, read-only, or read/write. A typical use case is when the orchestrator is shared by several teams, but each one has its own test environments and does not want other teams to access them. It is now possible to limit the access of each team member to the environments, workflows… of her/his team, while the orchestrator administrators have access to everything.

• The new create-archive action can be used in a PEaC file to generate a tar file.

• The new /channels endpoint can be used to get the list of all execution environments (both the ones accessed via ssh or via the agent).

The full list of modifications is available here.

2022-04 (2022-04-22)

This release is a maintenance release containing small improvements:

• opentf-ctl is using much less memory when displaying workflow information (e.g. when using opentf-ctl workflow --wait or opentf-ctl get workflow).

• opentf-agent --version can now be used to get the version of the agent.

• A warning is now displayed when launching the agent if the specified encoding is UTF8 (or if no encoding is specified, as this is the default) while the execution environment is not set with this encoding.

The full list of modifications is available here.

2022-03 (2022-03-30)

This release adds the possibility to use OpenTestFactory as a quality gate in a CI/CD pipeline.

Some other, small, improvements have been done:

• We started to improve log management: some microservices (the ones used to drive tests) are reporting more information. This information is available in opentf-ctl get workflow and in the Jenkins plugin.

• The event retention period of the Observer can now be configured with retention_period_minutes (see the documentation).

• When running a Cucumber test, a report.json file was generated, but its content was not some JSON data, but, instead, the Cucumber terminal output. This file is no more generated.

• When running a Robot Framework test, a tar file containing the report.html and log.html files, and the screenshots generated during the test, is now generated.

• When investigating problems, the channel_id is now available in the logs.

• opentf-ctl view token TOKEN can be used to display the payload of a token. opentf-ctl check token TOKEN using PUBLIC_KEY_PATH can be used to validate a token.

• opentf-ctl get workflow has been extended with the --output=format option, so the returned information can be formatted in JSON or YAML.

• The agent now verifies that the provided OS tag corresponds to the operating system where it is running. Additionally, it validates the syntax of the provided orchestrator URL.

The full list of modifications is available here.

2022-02 (2022-02-25)

This is a maintenance release fixing some bugs and preparing the quality gate that will be added in 2022-03.

The full list of modifications is available here.

2022-01 (2022-01-24)

This is a maintenance release fixing some bugs and including some small improvements:

• more information and clearer messages in the logs
• agent
  • increased performance
  • possible disabling of certificate verification
  • support of private certificates
• opentf-ctl
  • opentf-ctl get workflows displays more columns (ID, status, first timestamp, name)
  • opentf-ctl now supports get agents and delete agent commands to list and kill agents

The full list of modifications is available here.

2021-12 (2021-12-23)

• The opentf-ctl tool has been enhanced with the following features:

  • the config command can be used to generate or modify a configuration file;
  • the version command displays the tools version;
  • the get agents command lists the registered agents;
  • environment variables with an OPENTF_RUN_ prefix are automatically provided to the workflow execution environment;
  • all workflow execution errors are now reported for get workflow and run workflow --wait;
  • better error messages for invalid config files.

• The Jenkins plugin has been open-sourced (it was previously part of the Squash code base): https://gitlab.com/henixdevelopment/open-source/opentestfactory/jenkins-plugin

The full list of modifications is available here.

2021-11 (2021-11-16)

• Added support of Postman.

• An Allure report is now produced for each executed PEaC.

2021-10 (2021-10-21)

• Added support for properly starting and stopping the orchestrator.

• Added tools to help manage the orchestrator:

  • opentf-ready to wait until the orchestrator is ready to accept workflows.

  • opentf-done to wait until the orchestrator can be safely stopped (i.e. it has no more pending tasks).

  • opentf-ctl that can be used to

    • start a workflow;
    • stop a workflow;
    • cancel a workflow;
    • generate a signed token;
    • list the subscriptions on the event bus.

• Added the put-file action to put a file in the execution environment (see the action description).

2021-09 (2021-09-09)

• Added the Java Param Library
  This library provides an API usable in Java tests to retrieve the values of the parameters defined in a PEaC.
• Added the support of hooks
  Hooks allow the definition of custom actions to be performed before and/or after a given action of the orchestrator: retrieval of non-standard test reports, starting/stopping a service required by the tests, logging the start and the end of the tests… These hooks can be defined either in a PEaC to be applied to the corresponding workflow or in a configuration file of a provider to be applied to all workflows.
  See the hooks guide for more information.

2021-07 (2021-07-16)

• Added support of SKF (Squash Keyword Framework)

1.1.0 (2021-06-04)

• Improved logs in Jenkins plugin

1.0.0 (2021-04-23)

Initial release

• basic orchestrator mechanisms: reception and execution of a PEaC, stopping a PEaC
• retrieval of tests in Git repositories
• launch of tests and parsing of results for JUnit, Robot Framework, Cucumber, Cypress, SoapUI
• generation of Allure reports (can be published only in Squash TM for the time being)
• Jenkins plugin