Table of Contents

1. Introduction to Test Harness

2. Overview of Testing Webapp

3. Execution Components and Flow

4. Insight into Test Executors

5. Local Execution of Test Harness

6. Troubleshooting & Common Issues

7. Additional Resources

Introduction to the Test Harness

Purpose:
In BB ecosystem, the Test Harness aims to ensure that software components adhere to the standards and are compatible with the BB specification. By facilitating comprehensive testing against BB specifications, it ensures each application can smoothly fit into the broader BB framework.

For the time being each BB has it’s API Specification test suite implemented, in the future tests will cover also specific Test Cases as well as functional requirements.

Structure and Key Components:
The Test Harness is structured with thought of easy test execution for the candidate applications. A deeper look into its architecture reveals:

• GitHub BB Repository: The central hub for all applications and test suites. This repository dictates the workflow and serves as the primary source of data for the Test Harness.

  • Folder Structure: Test Harness setup relies on particular requirements that are defined by uniformed structure of each BB repository.

• CircleCI ORB: The ORB encapsulates a series of meticulous operations that are shared between all BBs.

  • Execution Matrix Generation: Through the Assembly script, the ORB evaluates the repository's structure to create an execution matrix. This matrix maps each application to its respective test suite, streamlining the process.

  • Test Execution: Leveraging the execution matrix, the ORB sequentially triggers tests, ensuring each application is vetted against its corresponding test suites.

• Testing Webapp: Beyond presenting results, the webapp offers a granular look into each test case.

  • Backend (Node.js): This backend handles incoming test results, processing the data for storage.

  • Database (MongoDB): All test results and associated metadata find their home here.

  • Frontend (React): Users interact with this React-powered UI. It displays test outcomes, provides filtering options, and allows for detailed dives into specific test cases.

Open

High Level Architecture of Test Harness

Operational Flow:

The Test Harness operates in a structured sequence:

1. Repository Input: Applications and test suites are sourced from the GitHub BB Repository.

2. Structure Analysis: The CircleCI ORB uses its assembly script to scan the repository's layout.

3. Matrix Creation: An execution matrix is formed, matching applications with appropriate test suites.

4. Test Environment Setup: Before tests start, a controlled environment is prepared.

5. Test Phase: Each application undergoes testing against its paired suite.

6. Result Compilation: Post-testing, results are forwarded to the Testing Webapp and stored in MongoDB.

7. Review: Developers access the Testing Webapp's React interface to view and analyze test outcomes.

Open

High Level Interaction

Overview of the Testing Webapp

Purpose:
The Testing Webapp captures and displays test results from the Test Harness. It aids in understanding the compatibility of applications with the BB standards.

Location:
The Testing Webapp is hosted https://testing.govstack.global/

Views:

• GovStack Building Block Compliance: This is the landing page. Here, users will see a summary of the most candidate applications in relevant building blocks with high level summary of compliance results.

  • NOTE: by default, this page only shows tests that are run against the main branch. During the development process, to see tests run against all branches, you can add a query parameter called ‘showAll’ to the URL: https://testing.govstack.global?showAll=true
    
    

• Application BB Compliance: A page with detailed information about test execution for specific candidate application in a building block (example)
  
  

Technical Underpinnings:

• Frontend: The Testing Webapp uses React, which ensures a responsive user interface.

• Backend: Node.js serves as the backend, handling data processing and communication with the database.

• Database: MongoDB stores all test results and related data.

Components and Test Execution Flow

BB Repository

Role of the Building Block (BB) Repository Structure

• The BB repository structure determines how the Test Harness identifies and interacts with test suites and applications.

• A clear directory hierarchy within the repository facilitates test execution process.

Attached screen presents usual structure of the BB Repository. Candidate applications with their respective setups are stored under examples/ directory. tests/ directory contain test executors for different test suites. Currently only openAPI compliance tests are provided.

Test Harness execution requires each candidate application to provide at least test_entrypoint.sh script which execution will result in API available on URL expected by the test executor (usually it’s localhost:3333).

See Test Utils ORB setup instruction for more details.

CircleCI

CircleCI Workflow functions as the driving mechanism for the entire testing process. Each BB should be activated in the CircleCI and have config file attached to it (example: https://github.com/GovStackWorkingGroup/bb-digital-registries/blob/main/.circleci/config.yml ). Config file for each BB can be the same. It is using predefined ORB to execute all operations, no custom setup is required, except for maintaining the used ORB version.

Upon CircleCI Job activation, workflow first navigates the BB repository structure. It creates an execution matrix based on available test suites and applications.

The matrix then ensures every possible combination of application and test suite undergoes validation. For instance if BB provides 2 applications:

and 2 test suites:

Tests will be executed for: (app1, suite1), (app1, suite2), (app2, suite1), (app2, suite2).

Candidate Application

Every application in the BB repository needs a test_entrypoint.sh. This script acts as the starting point for test execution. It's responsible for setting up the application and ensuring it's ready for testing. The script can be custom for each application, but in the result API accepting requests from the Test Suite client has to be exposed. How application is setup is up the the implementers. Example setups are usually using the dockerized application with it’s dependencies, however it wouldn’t be an issue to setup local api that would redirect requests to the external server where candidate application lives.

It is worth to mention that a lot of tests have some predefined requirements, like digital registry test suite expects registry named registryname with version number 111 and some data to be available. Uploading this seed data to the application should be part of the setup and can be executed within the test_entrypoint.sh.

Process of setting up Mockoon application: Setup Mockoon application (Mockoon applications were used in several BB during implementation of test cases to validate the test suite).

Process of setting up Camunda: Steps to check compliance against a GovStack API spec

Test Suite Execution and Results Compilation

After initializing the application, the Test Harness then navigates to the respective test suite and triggers its test_entrypoint.sh (note: even though the names of the scripts are the same the functionality is different). This script handles the actual testing process against the initialized application. At the end of the test suite's execution, a /results/result.message file should be present in the suite's directory. This file captures the outcomes of the tests. Besides result.message in message format, test executor can also generate second file result.xml that will be used by the CircleCI to display test execution result directly in the CircleCI web dashboard. This will be useful to check errors early and have additional information useful for the developers.

Communication with the Testing Webapp

• The Test Harness collects the result.message and some metadata (application, repository, branch).

• This data package is then sent to the Testing Webapp for storage and presentation.

• The Testing Webapp processes the data, making it viewable for developers and stakeholders.