Feature Request: Custom CircleCI Configuration Injection for Test Harness

Owned by Damian Borowiecki
Nov 20, 2023
2 min read

This feature request proposes the implementation of a capability within the Test Harness system to allow users to inject custom CircleCI configurations. This feature aims to accommodate specific requirements of certain applications that cannot be fully addressed by the current instance setup within the Test Harness that relies on the test_entrypoint.sh, such as those requiring pre-test execution steps like Helm or AWS configurations.

Objectives

  • To enable users to include additional CI steps either before or after the dynamically generated job.

  • To provide flexibility in the CI process to handle diverse software requirements that are not compatible with a standardized docker environment.

  • To maintain the security and integrity of the CI process while allowing for customizable workflows.

Scope of Customization

Customization will be allowed through a specific configuration file placed within the user's application folder. The file will dictate additional CI steps to be executed either pre or post the main test execution.

Configuration File

  • Name: circleci-custom-config.yml

  • Location: Within each application's respective folder under /examples.

  • Syntax: The file must follow YAML syntax rules and conform to CircleCI configuration standards.

 

Flow

Use Cases

  • Applications requiring Kubernetes deployment via Helm before test execution.

  • Services needing AWS resource configuration or any cloud service setup prior to testing.

  • Any other preconditions that need to be fulfilled on the CI environment not handled by test_entrypoint.sh.

Security and Compliance Considerations

  • All custom configurations will go through a standard review process to ensure they don't contain harmful operations (e.g., cryptocurrency mining, security vulnerabilities).

  • Execution of the custom steps must not allow elevation of privileges in the CI environment.

  • Access to sensitive environment variables or credentials within the CI process must be controlled and audited.

Validation and Error Handling

  • YAML validation will be performed to ensure the file is not corrupted.

  • If a step defined in circleci-custom-config.yml fails, the entire workflow will be halted, and an explicit error message will be provided.

  • This mechanism is to prevent partial execution which could result in unclear testing states or results.

Impact on Existing Workflows

  • This feature will be optional; existing workflows will function unchanged if the custom configuration file is not provided.

  • The Test Harness system will default to the standard process when the configuration file is absent.

User Feedback Loop

  • Users may utilize a dedicated Slack channel for real-time communication regarding feature implementation and troubleshooting.

  • Issues can also be reported and tracked via a JIRA board specific to the Test Harness project.

Implementation Steps

  1. Update Test Harness Architecture: Refactor the current setup to allow for the inclusion of custom configurations.

  2. Develop Configuration Parsing Mechanism: Implement a method to parse and validate circleci-custom-config.yml.

  3. Integrate into CircleCI Workflow: Adjust the CircleCI workflow to execute steps defined in the custom config.

  4. Security Review Process Definition: Establish a review process for the custom configuration submissions.

  5. Documentation: Update documentation with instructions on how to create and use circleci-custom-config.yml.