Different test suite versions will be determined through GitHub releases.

The structure of the repository remains the same

/examples
    /mock
    /registrationApp
    /…
/test
    /openAPI
        …
   /nextTestSuite
        …

Test suite versions are determined using release tags, which helps maintain a clean repository structure.

In CircleCI, we employ a dynamic matrix to identify the candidate application and the test suites provided in a given BB repository. This matrix can be expanded by using release git tags that match specific regex patterns. Semantic versioning is used to represent tags versions.

1. Semantic Versioning

In the GovStack project, Semantic Versioning (SemVer) is applied to provide clarity and predictability around the project's release cycles and compatibility. Below are guidelines tailored specifically to how GovStack interprets and utilizes Semantic Versioning:

Basic Structure

GovStack uses the standard SemVer format vX.Y.Z for version numbers, where X is the major version, Y is the minor version, and Z is the patch version. Version numbers do not contain leading zeroes, e.g., v1.9.0, v1.10.0.

Release Rules

Once a version is released, its contents must not be altered. Any change necessitates a new version release.

Major Version (X)

The major version X is incremented when changes are introduced to the GovStack API that would break backward compatibility, requiring clients or dependent services to adjust their implementations.
Incrementing the major version resets the minor and patch versions to 0 (e.g., v2.9.5 to v3.0.0).
Major version changes are likely to require users to change something about their setup.

Minor Version (Y)

The minor version number Y increases when new features are introduced that are compatible with the existing functions of the software.
It may also be incremented if any public API functionality is marked as deprecated, or if there are significant improvements within the private code that do not affect the public API.
Incrementing the minor version resets the patch version to 0 (e.g., v1.1.9 to v1.2.0).

Patch Version (Z)

The patch version number Z goes up by one when we make small fixes that don't change how things basically work. So, everything that used to work still will.
A bug fix is an internal change that corrects a fault, flaw, or error without affecting the public API or existing features (e.g., v1.0.0 to v1.0.1).

Example test suite releases for BB:

v1.0.0
v2.3.3
v3.0.0

The dynamic matrix will execute tests for each suite and the latest version of the repository (main branch), using a candidate app, test suite, and version:

suite × candidate × version

In the presence of multiple releases, the number of executed test harnesses may rise significantly. To manage this, employing dedicated tags for releases corresponding to changes in test suites is advisable.

2. Implementation

1. Enhancement to `src/templates/generate-config.py`

In the generate-config.py, a function has been introduced that fetches tags from GitHub. These tags are subsequently filtered using a regex pattern for semantic versioning. Moreover, the list_test_execution has undergone modifications to utilize a three-dimensional matrix. This matrix will comprehensively encompass all potential combinations of suite × candidate × version.

Note: The list of tags currently includes the 'main' branch in addition to version-specific tags.

2. Extension to `base-generated-config.yaml`

The base-generated-config.yaml now includes an additional step:

      - run:
          name: Checkout Test Suite from Specific Version
          command: |
            # Clone repository into a temporary directory and switch to the specified tag
            git clone . temp_repo
            cd temp_repo
            git checkout << parameters.bb-version >>

            if [ ! -d "<< parameters.test-suite-path >>" ]; then
              mkdir -p << parameters.test-suite-path >>
            fi

            cp -r test/* << parameters.test-suite-path >>
            cd .. && rm -rf temp_repo

The main functionality of this step is to copy the test suite corresponding to a particular version (based on the git tag or main branch). Following this, it runs the test suite against the most recent release of the candidate application.

3. Modifications to the Generated Config

Within the generated config, the following command is used:

cd << parameters.example-app-path >> && ./test_entrypoint.sh --version << parameters.bb-version >>

The newly introduced --version << parameters.bb-version >> parameter provides other developers with the flexibility to employ it within their candidate applications. This can be particularly beneficial for tasks like matching data specifically created for a certain version or any other related requirements.

4. Introduction of New Reporting Parameter

A new parameter has been integrated into the report dispatched to the database. This addition can be seen in the following curl command:

This new parameter provides a more detailed and refined representation of the data, enriching the report's comprehensiveness.

3. Candidate Applications Use

The test suite version is passed as an input to the candidate application's ./test_entrypoint.sh script using the --version flag (e.g., --version v1.0.3). This allows the setup to be customized for a specific release.

This feature can be utilized for various purposes, such as matching data seeds for tests or any other requirements that a developer may have to align with the specific version.

Test Suite Versioning