Steps to check compliance against a GovStack API spec

Taylor Downs, Apr 24, 2023

Important Pre-Reading

1. This document describes how a contributor can test the compliance of a candidate application against a particular test-suite for a GovStack building block specification.

   1. Please note that as of April 25th, 2023, the only test suites that have been developed are for checking API compliance. On Software Compliance Concept, you’ll find all other compliance categories which are to be assessed manually.

2. Every GovStack building block will have an API, defined in an OpenAPI Specification (usually swagger.yaml) defined in their /api/ directory.

3. Every GovStack building block will have several test suites (e.g., the API test suite /test/api-suite/) defined in their /test/ directory.

4. Every GovStack building block will have several candidate applications with deployment & configuration scripts defined in their /examples directory.

5. Whenever a commit or pull request is made to a building block repository, all of the test suites inside /test/ will be run against all of the candidate applications inside /examples .

   1. More specifically, compliance is checked by starting and configuring each example application by executing a /examples/yourApp/test_entrypoint.sh --config api-suite script provided by a contributor and then running all test suites against that running application.

6. Read on to learn more about how to add a new example application for compliance testing, as well as some tips and guidance for how to configure a test_entrypoint.sh file, an adaptor, and underlying deployment and configuration services that will enable yourApp to pass GovStack tests.

(1) Add your candidate to the repo

1. Clone the repo for the specification you’re checking:
   git clone git@github.com:GovStackWorkingGroup/bb-workflow.git
   
   

   Open

   

    

2. Add a folder for your candidate in the /examples directory.
   
   

   Open

   

    

3. Provide a way for the test runner to launch and configure your application via the execution of a test_entrypoint.sh file. (This will launch your candidate via docker-compose, provide an adaptor—maybe via Caddy, maybe via some other application—and configure it to pass a particular suite of tests.)
   
   

(2) Test your compliance locally, develop your adaptor

1. cd into your candidate’s directory and start your application with ./test_entrypoint.sh

2. cd into the /test/api-suite directory and install the testing dependencies:

   1. asdf install

   2. yarn install

3. Run the tests

   1. yarn run test

4. See which are passing and which are failing. Design and develop an adaptor that will enable your candidate application to achieve compliance with more of the tests detailed in this particular suite.

   1. Add link to adaptor design and development guidance

(3) Open a pull request against the official GovStack spec repo

1. Once you’re happy with your candidate application’s level of compliance, open a PR against the official GovStack spec.

2. When you open your PR, you’ll see that the test suite now runs against your candidate and reports back your compliance results on CircleCi.
   
   
   

   
   
   Here we can see that a workflow candidate “Camunda” has been properly configured and is currently passing 0 out of 5 workflow api specification tests.
   
   
   

3. Request a review on your PR from one of the repository admins.

4. Once your PR is merged, you’ll be able to see your compliance results on the public GovStack Building Block Compliance Platform.
   
   
   

   
   
   

Next Steps

Going forward, you and other contributors to the specification will be able to see how proposed changes to the API and its associated tests will impact compliance with all “checked in” candidates for a building block.