January 19, 2022

Data used in tests

If there is particular data that we need to pass the test then apps and candidate app authors should use the json blob data during test_entrypoint.sh execution. Data models can vary between candidates. App author is responsible for configuring the app in the way to pass the tests and check compliance. We can encourage them to use our json blob and make functionality creating schema based on our spec - therefore, during any data changes test result shouldn’t change. However, it’s not a mandatory approach.

Use cases - how do we want to approach tests to make sure that use cases can be passed.

At the moment we have following structure: 

bb-folder/

/test/<gherkin app>

/examples/

candidate-a/

test_entrypoint.sh

…

When we’ll start to execute tests against multiple test suites, we’ll need to change structure:

1. Of the /test/ subdirectory:

/test/

    test_suite_A/<gherkin app>

    test_suite_B/<gherkin app>

    …

With separate definition for all of the test suites (Post Partum, OpenAPI spec, ect.)

1. Of the /examples/ subdirectory:

/examples/

    /candidate-a/test_entrypoint.sh –config test_suite_A 

    /candidate-b/test_entrypoint.sh –config test_suite_B

Type of test suite would be determined by the execution attribute. 

Note: This has to be discussed with the committee as we come up with a solution during the meeting.  

Integration with Sandbox team work

The Sandbox team will be creating adaptors and creating setup and deployment scripts for particular use cases. We should have a conversation with the Sandbox team to decide when and how we can align our work to make sure we’re going in the right direction.

CircleCi configuration - test entry points

Test entry points should not run in the background! Taylor did the change on his PR. The test entry point has to be finished first and so does the whole setup before running the tests. Taylor removed the background process and it seems to be working well so we should align our approach with it.
@Damian Borowiecki to check https://app.circleci.com/pipelines/github/GovStackWorkingGroup/bb-workflow/88/workflows/5a5fc85e-f395-46ec-a0aa-871bdec0780a/jobs/157 and try to get the same solution to the one he was working on. How to interpret the results above?

CircleCi configuration - Security checks

How can we avoid contributors making changes? We need to think about it in the future. We need to be careful when we do PRs reviews.

CircleCi configuration - Backend

We need to slightly improve the way test execution results are passed to the database. In the current solution we’re executing the tests and then aggregate results, which are sent directly to the mongodb. This has several drawbacks, among others: 

• Exposed DB 

• MongoDB schema kept in several places 

• Credentials to mongoDB added to all building blocks

• Harder to maintain the solution across multiple bb

We decided to change the solution and make aggregations and insertions of test results on express.js backend site. This will allow us to keep better control over the structure and make it easier to test results.