GovStack sandbox execution environment
Sandbox environment provides Kubernetes resources to run containers in shared environment.
Overview
Our implementation follows guidelines from AWS documentation, https://docs.aws.amazon.com/prescriptive-guidance/latest/containers-provision-eks-clusters-terraform/welcome.html
Sandbox environment will be installed to one region. We are going to use eu-west-1 ( Frankfurt ) region. Inside region we create 3 public and private subnets. Subnets are distributed within 3 availability zones.
Subnets
In subnets we are using 24 subnet mask, so there is about 250 ip address to use.
Private subnets contains building blocks and other required components ( Kubernetes infra components )
At this phase we are not going to install building blocks to public subnets. If some building blocks requires access to outside of private network we have installed NAT GW to provide internet access.
Managed node groups
Instead of fargate, we chose managed working node groups. Fargate did not support all the features we needed for the kubernetes so it was chosen instead.
High-level architecture
Provisioning
We are using Terraform an Terragrunt for installing/provisioning Sandbox environment.
Provisioning uses a layered approach on the env provisioning
Base infra for sandbox (VPC, subnets (3xpublic and 3xprivate), internet gw, nat gw, …)
modules/eks/vpc.tf
Environment for govstack instances (EKS, ECR) (depends on base layer)
modules/eks/eks-cluster.tf
CircleCI pipeline specific roles
modules/circleci
ECR repositories
modules/ecr
Environment for sandbox portal (ECS, ECR (should we share the ECR → basic infra), ALB?), depends on base layer. There is separate environment made for the portal
Found in separate repository https://github.com/GovStackWorkingGroup/sandbox-portal-infra
The stacks of circleci, eks and kube should be ran on that order. Provisionin order of the rest does not matter at this time.
Checkout latest IaC codes from repository: https://github.com/GovStackWorkingGroup/sandbox-infra
Update AWS login settings:
aws configure
NOTE: Instead of using aws own configure, it is recommended using https://github.com/99designs/aws-vault
Provisioning instructions for infrastructure can be referred with Infrastructure deployments
Example container installation
Add Helm repo
helm repo add bitnami https://charts.bitnami.com/bitnami
Install mysql
helm install bitnami/mysql --generate-name
Check installed containers
Example Rest application installation
Example app source code can be found from:
Create a Dockerfile, if is not exists
Run docker build
Login to AWS ECR
Using eu-central-1 as a region
And 814942682479 as a aws account id
Push image to ECR
Create Helm chart
Change correct values for values.yml
Install Helm chart
Verify that installation executed successfully
Test application
Create tunnel to application and after that open browser with http://localhost:18081/api/v1
Building block isolation
Building blocks inside Govstack environment are isolated by Kubernetes namespace. Every organisation and/or business-case get own namespace and related building blocks are installed “under” that namespace. This functionality ensures that building blocks can interact between other building blocks inside namespace but can’t directly interact with other namespace BB’s. Also capacity reservations can be easily adjusted if needed.
Building block CI/CD
NOTICE: This MVP solution for now and in the future there will be Portal/Automated functionality handle CI/CD process.
Every Building block should contains helm charts inside building block repository.
TODO: Add more information about helm charts
CI/CD pipelines are develop to together with BB developer and Sandbox admin.
Only Sandbox Admin are allowed to maintain/install BB:S to Sandbox environment.
CircleCI
OIDC
For general authentication to AWS Resources CircleCI build jobs uses OIDC (OpenID Connect indentity) tokens. So we are not storing AWS secrets to CircleCI.
For every pipeline, separate iam role is needed to be created in aws side for it to be authenticated to the kubernetes. This is done by adding it to env.hcl and then running terragrunt stacks circleci and eks in that order.
For information: https://circleci.com/blog/openid-connect-identity-tokens/ and https://circleci.com/docs/openid-connect-tokens/
Note!
In CircleCI jobs we need to use context, CircleCI stores OIDC token to context during workflow execution
Contexts provide a mechanism for securing and sharing environment variables across projects. The environment variables are defined as name/value pairs and are injected at runtime. This document describes creating and using contexts in CircleCI. CircleCI - context
CircleCI auth to aws
After OIDC is in place, pipelines needs an AWS IAM role that authenticates them to deploy to kubernetes. This is done with the circleci module in terraform. In the corresponding environment, you need to add the correct values to env.hcl
Example from development environment
Org id is the id of the circleci organisation, this can be found from the project settings
ssl thumbprint can be calculated following https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html or with the tool in infra repo tools/ssl_thumbprint.sh that basicly just implements that guide.
In projects project_id can be found from the pipelines configuration in circleci and name you can decide yourself.
After applying the changes to circleci stack, run also the eks stack so those roles are added to aws_auth list
Kubernetes auth
Build jobs also requires access to Kubernetes when installing new deployments. Kubernetes controls it’s own access, so access to AWS services is not enough.
Adding permission to specific user/role requires to modify aws-auth inside kube-system namespace. This can also be done with terraform, as is done with the circleci pipeline roles
Current settings can be check:
And settings can be add/edit using kubectl edit configmap
/aws-auth or using eksctl
For example, following command add required permissions to build job:
Quality ensure of BB
Before BB is installed to GovStack environment it must go thru API Testing process and results must to provide to Sandbox admin. When Image is pushed to ECR vulnerability scan is executed and BB must be at level Medium or lower, see below. ? Otherwise BB is not allowed to install Sandbox environment.
CVSS scores
Critical (9.0-10.0) score are not allowed
High scores ( 7.0 - 8.9 ) must be analysed before install and take mitigate actions if possible
Lower scores can be installed
TODO: specify which level vulnerability issues can be