Test Utils ORB setup instruction

The GovStack Test Utils Orb is a utility for managing alignment across Circle CI integration for the many different Building Block repositories in GovStack.

This document explains how to configure the environment in a Building Block repository to run tests for a real software and send the test results to the server.

Here is the repository for the ORB project: https://github.com/GovStackWorkingGroup/test-utils

Prerequisites

Ensure that your project has added configuration on the Circle CI project page.

Go to the Circle CI app
Set up circle configuration in the project (use “Set Up Project” button)

If you don't have rights to set up project please contact the administrator.

Up-to-date list of projects with corresponding configuration on the Circle CI project page:

bb-consent
bb-digital-registries
bb-identity
bb-information-mediator
bb-messaging
bb-payments
bb-registration
bb-schedular
bb-workflow

Step 1: Circle CI management setup

The GovStack Test Utils Orb simplifies the Circle CI configuration by providing pre-defined commands and workflows. Here's how the orb works in general:

It sets up the required environment for running tests.
It generates at runtime the necessary configurations and files for test execution.
It executes the tests for every candidate application configured in the repository and sends the test results to the the GovStack Testing App that visualizes the test [link].

To use the GovStack Test Utils Orb, follow these steps:

Make sure to add the necessary environment variables to the Project Settings on the CircleCI page. Note that there are 4 variables to define, 2 for production (testing.govstack.global) and 2 for staging (staging.testing.govstack.global). If any of these two sets of variables are missing, results with not be uploaded to that destination (for example if no staging URL defined, results will not be uploaded to staging - but they still might uploaded to production if the vars for production are set).
1. TEST_RESULTS_STORAGE_API_KEY: API key for test results storage.
2. TEST_RESULT_STORAGE_URL: expect the url to the testing.govstack.global app and the endpoint to upload the results. Example api.testing.govstack.globa/report/upload
3. TEST_RESULTS_STORAGE_API_KEY_STAGING: API key for test results storage for the staging server.
4. TEST_RESULT_STORAGE_URL_STAGING: expect the url to the staging.testing.govstack.global app and the endpoint to upload the results. Example api.staging.testing.govstack.globa/report/upload
Enable "Dynamic config using setup workflows" in the Project Settings on the Circle CI page. You can find this option in the "Advanced" tab at the bottom.

Step 2: Add Circle CI config file to the project folder

The Circle CI configuration file (config.yml) should define the jobs and workflows for running tests using the GovStack Test Utils Orb.

Here's how the configuration works:

The create-config job creates the generated.yml file, which is required to run the tests for all examples. This file contains the necessary configurations for the tests and is generated by the ORB for all candidate application that have their directories defined in the examples directory.
The execute-tests step runs the tests using the configurations from the generated.yml file.
The test artifacts, which include the test results, are stored on Circle CI and sent to the external server. The test results are stored in the .message format, while Circle CI uses the .xml format internally.

To set up the Circle CI configuration, follow these steps:

Create a .circleci folder in the root directory of your repository, if it doesn't exist already.
Add the config.yml file to the .circleci folder. You can use the following template:

version: 2.1
setup: true

orbs:
  test-harness: govstack-working-group/testutils@1.0.3

workflows:
  test_everything:
    jobs:
      - test-harness/create-config:
          post-steps: # Persist to workspace has to be defined in main workflow
            - persist_to_workspace:
                root: workspace
                paths:
                  - generated.yml
      - test-harness/execute-tests:
          requires:
            - test-harness/create-config

By adding this configuration file, the CircleCI pipeline will execute the tests and store the test results as artifacts.

Step 3: Required setup for application

First need to set up a directory with the name of the application in the /example folder
For example: /examples/<application_name>.

Then need to define the test_entrypoint.sh file in the /examples/<application_name> folder.

Here's what you need to do:

Provide the test_entrypoint.sh file that sets up the system for testing. This script prepares the environment and any dependencies required for running the tests.
The test_entrypoint.sh script should be responsible for tasks such as:
- Installing necessary dependencies or packages.
- Setting up databases or other required services.
- Configuring environment variables or system settings.

Make sure that the test_entrypoint.sh script is included in the directory and referenced correctly in the Circle CI configuration.

Ensure that the test_entrypoint.sh script finishes executing and sets up the environment correctly before the tests are executed.

Note: The specific setup steps in the test_entrypoint.sh file will depend on your product's requirements and dependencies.

Remember to adjust the contents of the test_entrypoint.sh script based on your project's needs.

Step 4: Required setup for test execution

After the systems are set up, the next step is to set up the test harness to execute tests on all of the candidate systems defined in the examples directory.

Add all the files needed for test execution to the test/openAPI/{testSuite} folder.
Example of file structure:
- /test - /openAPI - /docker - Dockerfile - entrypoint.sh - /features # here are stored all written tests - docker-compose.yaml - test_entrypoint.sh
Tests are executed using test/openAPI/test_entrypoint.sh. The result of the tests must be stored as a file at the following path: test/openAPI/{testSuite}/results/example.message in the message format. This file will be sent to the external server. We also expect a test/openAPI/{testSuite}/results/example.xml file in JUnit format, that will be visualized by the Circle CI GUI.

Example of setup using docker:

docker-compose.yaml - builds the gherkin-test-report container that will run the tests using it’s Dockerfile

version: '3.3'

networks:
  isolation-network:
    driver: bridge

services:
  # Local test execution waits for API to be available and create test harnes report
  gherkin-test-report:
    image: gherkin-test-execution
    build:
      context: .
      dockerfile: ./docker/Dockerfile
    command: run_tests
    environment:
      - RESULT_NAME=${RESULT_NAME:-example_result}
      - EXPORT_RESULTS=${EXPORT_SCRIPT:-docker/export_to_mongo.sh}
      - API_URL=${API_URL:-localhost:3003/healthcheck}
    volumes:
      - ./result/:/app/results
    network_mode: 'host'

volumes:
  test_result:

test_entrypoint.sh - the entrypoint to executing the tests - should be the first file called out, before Docker compose. This is the file that will be executed by Circle CI to run the tests against each candidate system.

add new folder docker and inside add
- Dockerfile - builds the image used for running the test (has node, does npm install). The entrypoint for the container is entrypoint.sh

- entrypoint.sh - the actual script that waits for the app and executes the tests inside the gherkin-test-report container

After adding all necessary files the Circle CI build should run on GitHub automatically after every repository update.

This example will run Cucumber JS tests.