Test Utils ORB setup instruction
The GovStack Test Utils Orb is a utility for managing alignment across Circle CI integration for the many different Building Block repositories in GovStack.
This document explains how to configure the environment in a Building Block repository to run tests for a real software and send the test results to the server.
Here is the repository for the ORB project: https://github.com/GovStackWorkingGroup/test-utils
Prerequisites
Ensure that your project has added configuration on the Circle CI project page.
Go to the Circle CI app
Set up circle configuration in the project (use “Set Up Project” button)
If you don't have rights to set up project please contact the administrator.
Up-to-date list of projects with corresponding configuration on the Circle CI project page:
bb-consent
bb-digital-registries
bb-identity
bb-information-mediator
bb-messaging
bb-payments
bb-registration
bb-schedular
bb-workflow
Step 1: Circle CI management setup
The GovStack Test Utils Orb simplifies the Circle CI configuration by providing pre-defined commands and workflows. Here's how the orb works in general:
It sets up the required environment for running tests.
It generates at runtime the necessary configurations and files for test execution.
It executes the tests for every candidate application configured in the repository and sends the test results to the the GovStack Testing App that visualizes the test [link].
To use the GovStack Test Utils Orb, follow these steps:
Make sure to add the necessary environment variables to the Project Settings on the CircleCI page. Note that there are 4 variables to define, 2 for production (testing.govstack.global) and 2 for staging (staging.testing.govstack.global). If any of these two sets of variables are missing, results with not be uploaded to that destination (for example if no staging URL defined, results will not be uploaded to staging - but they still might uploaded to production if the vars for production are set).
TEST_RESULTS_STORAGE_API_KEY
: API key for test results storage.TEST_RESULT_STORAGE_URL
: expect the url to the testing.govstack.global app and the endpoint to upload the results. Exampleapi.testing.govstack.globa/report/upload
TEST_RESULTS_STORAGE_API_KEY_STAGING
: API key for test results storage for the staging server.TEST_RESULT_STORAGE_URL_STAGING
: expect the url to the staging.testing.govstack.global app and the endpoint to upload the results. Exampleapi.staging.testing.govstack.globa/report/upload
Enable
"Dynamic config using setup workflows"
in the Project Settings on the Circle CI page. You can find this option in the "Advanced" tab at the bottom.
Step 2: Add Circle CI config file to the project folder
The Circle CI configuration file (config.yml
) should define the jobs and workflows for running tests using the GovStack Test Utils Orb.
Here's how the configuration works:
The
create-config
job creates thegenerated.yml
file, which is required to run the tests for all examples. This file contains the necessary configurations for the tests and is generated by the ORB for all candidate application that have their directories defined in theexamples
directory.The
execute-tests
step runs the tests using the configurations from thegenerated.yml
file.The test artifacts, which include the test results, are stored on Circle CI and sent to the external server. The test results are stored in the
.message
format, while Circle CI uses the.xml
format internally.
To set up the Circle CI configuration, follow these steps:
Create a
.circleci
folder in the root directory of your repository, if it doesn't exist already.Add the
config.yml
file to the.circleci
folder. You can use the following template:
version: 2.1
setup: true
orbs:
test-harness: govstack-working-group/testutils@1.0.3
workflows:
test_everything:
jobs:
- test-harness/create-config:
post-steps: # Persist to workspace has to be defined in main workflow
- persist_to_workspace:
root: workspace
paths:
- generated.yml
- test-harness/execute-tests:
requires:
- test-harness/create-config
By adding this configuration file, the CircleCI pipeline will execute the tests and store the test results as artifacts.
Step 3: Required setup for application
First need to set up a directory with the name of the application in the /example
folder
For example: /examples/<application_name>
.
Then need to define the test_entrypoint.sh
file in the /examples/<application_name>
folder.
Here's what you need to do:
Provide the
test_entrypoint.sh
file that sets up the system for testing. This script prepares the environment and any dependencies required for running the tests.The
test_entrypoint.sh
script should be responsible for tasks such as:Installing necessary dependencies or packages.
Setting up databases or other required services.
Configuring environment variables or system settings.
Make sure that the test_entrypoint.sh
script is included in the directory and referenced correctly in the Circle CI configuration.
Ensure that the test_entrypoint.sh
script finishes executing and sets up the environment correctly before the tests are executed.
Note: The specific setup steps in the test_entrypoint.sh
file will depend on your product's requirements and dependencies.
Remember to adjust the contents of the test_entrypoint.sh
script based on your project's needs.
Step 4: Required setup for test execution
After the systems are set up, the next step is to set up the test harness to execute tests on all of the candidate systems defined in the examples
directory.
Add all the files needed for test execution to the
test/openAPI/{testSuite}
folder.
Example of file structure:- /test - /openAPI - /docker - Dockerfile - entrypoint.sh - /features # here are stored all written tests - docker-compose.yaml - test_entrypoint.sh
Tests are executed using
test/openAPI/test_entrypoint.sh
. The result of the tests must be stored as a file at the following path:test/openAPI/{testSuite}/results/example.message
in the message format. This file will be sent to the external server. We also expect atest/openAPI/{testSuite}/results/example.xml
file in JUnit format, that will be visualized by the Circle CI GUI.
Example of setup using docker:
docker-compose.yaml
- builds the gherkin-test-report container that will run the tests using it’s Dockerfile
version: '3.3'
networks:
isolation-network:
driver: bridge
services:
# Local test execution waits for API to be available and create test harnes report
gherkin-test-report:
image: gherkin-test-execution
build:
context: .
dockerfile: ./docker/Dockerfile
command: run_tests
environment:
- RESULT_NAME=${RESULT_NAME:-example_result}
- EXPORT_RESULTS=${EXPORT_SCRIPT:-docker/export_to_mongo.sh}
- API_URL=${API_URL:-localhost:3003/healthcheck}
volumes:
- ./result/:/app/results
network_mode: 'host'
volumes:
test_result:
test_entrypoint.sh
- the entrypoint to executing the tests - should be the first file called out, before Docker compose. This is the file that will be executed by Circle CI to run the tests against each candidate system.
add new folder
docker
and inside add
-Dockerfile
- builds the image used for running the test (has node, does npm install). The entrypoint for the container isentrypoint.sh
- entrypoint.sh
- the actual script that waits for the app and executes the tests inside the gherkin-test-report container
After adding all necessary files the Circle CI build should run on GitHub automatically after every repository update.
This example will run Cucumber JS tests.