Skip to content

Overview

Introduction

OpenTestFactory is an open initiative aiming for a single standard mechanism to plan tests, execute them, and publish their results. It can be used standalone or integrated in a CI/CD pipeline.

OpenTestFactory is

Description

OpenTestFactory allows developers of application code, QA specialists, and operation teams to write their own tests without binding to any testing frameworks or vendors. Every stakeholder can write and maintain their tests using tools and frameworks they are already familiar with, and other stakeholders can cherry-pick the test cases they are interested in their contexts.

OpenTestFactory offers the following standardization:

  • Standardized test orchestration: APIs to allow for test case selection and execution, that can be used in test management solutions or in existing CI/CD pipelines.
  • Standardized test execution: APIs to aid in the integration of testing frameworks, from parameter passing to test case execution.
  • Standardized test result publication: APIs to aid in the collection of test results.
  • Standardized quality gate: specification of the precise results formats as they head towards quality gate vendors.

See this page for some benefits of OpenTestFactory.

PEaC files

The tests to be executed are defined in a PEaC (Planned Execution as Code) file using either YAML or JSON format.

A PEaC contains a list of jobs. Each job can include:

  • commands to be executed in the test environment;
  • file operations to be performed in the test environment;
  • the retrieval of the test code from a code repository;
  • the execution of these tests.

The syntax and all available commands are detailed in the specification chapter.

OpenTestFactory architecture

OpenTestFactory is based on a microservice architecture containing the following services:

Microservice architecture

  • Receptionist service is the service receiving the PEaC file.
  • Arranger service ordonnances the tests defined in the PEaC.
  • Observer service can be used to track the progress of a PEaC being executed.
  • Killswitch service can be used to cancel a currently executing PEaC.
  • Generator services are connecting to test repositories to retrieve test plans (no such service is currently provided by the OpenTestFactory reference implementation).
  • Function providers are performing the base functions
    • cloning the Git repository containing the automated test code;
    • piloting the tests, each is specific to a given test technology.
  • Execution environment handlers manage the connections between OpenTestFactory and the test environments.
  • Results publishers push the test results to a given location.

Components of the reference implementation

The reference implementation contains the following components:

  • Orchestrator
    • reception of PEaCs;
    • ordonnancing the execution of the tests defined by a PEaC;
    • stopping the execution of a PEaC.
  • Plugins
    • interfacing with Git to retrieve the automated tests from a repository;
    • driving the tests via ssh or an agent installed in the execution environment;
    • managing the tests and parsing the results of
    • publishing results to
    • consolidating all test results in an Allure test report;
    • consolidating all test results in HTML and XML reports;
    • evaluating if the quality gate is successful or not.
  • Agent
    • enabling tests to be driven in an environment where ssh is not an option;
    • can be deployed anywhere Python is available.
  • Some parameter retrieval libraries enabling tests to get their parameter values from test data stored in a test repository
  • Some Docker images of execution environments:
  • Jenkins plugin
    • simplifying the integration to Jenkins.

Deploying the reference implementation

OpenTestFactory can be used standalone or it can be integrated in any CI/CD pipeline.

The installation chapter details how to install OpenTestFactory.

Contributing

OpenTestFactory is an open-source project. You can extend it if, currently, it does not fulfill your needs. You are welcome to contribute these enhancements to the project. How to contribute details on how to do so.

More information