GovStack sandbox execution environment

Sandbox environment provides Kubernetes resources to run containers in shared environment.

Overview

Our implementation follows guidelines from AWS documentation, https://docs.aws.amazon.com/prescriptive-guidance/latest/containers-provision-eks-clusters-terraform/welcome.html

Sandbox environment will be installed to one region. We are going to use eu-west-1 ( Frankfurt ) region. Inside region we create 3 public and private subnets. Subnets are distributed within 3 availability zones.

Subnets

In subnets we are using 24 subnet mask, so there is about 250 ip address to use.

Private subnets contains building blocks and other required components ( Kubernetes infra components )

At this phase we are not going to install building blocks to public subnets. If some building blocks requires access to outside of private network we have installed NAT GW to provide internet access.

Managed node groups

Instead of fargate, we chose managed working node groups. Fargate did not support all the features we needed for the kubernetes so it was chosen instead.

High-level architecture

Provisioning

We are using Terraform an Terragrunt for installing/provisioning Sandbox environment.

Provisioning uses a layered approach on the env provisioning

Base infra for sandbox (VPC, subnets (3xpublic and 3xprivate), internet gw, nat gw, …)
- modules/eks/vpc.tf
Environment for govstack instances (EKS, ECR) (depends on base layer)
- modules/eks/eks-cluster.tf
CircleCI pipeline specific roles
- modules/circleci
ECR repositories
- modules/ecr
Environment for sandbox portal (ECS, ECR (should we share the ECR → basic infra), ALB?), depends on base layer. There is separate environment made for the portal
- Found in separate repository GitHub - GovStackWorkingGroup/sandbox-portal-infra

The stacks of circleci, eks and kube should be ran on that order. Provisionin order of the rest does not matter at this time.

Checkout latest IaC codes from repository: GitHub - GovStackWorkingGroup/sandbox-infra

Update AWS login settings:

aws configure

NOTE: Instead of using aws own configure, it is recommended using GitHub - 99designs/aws-vault: A vault for securely storing and accessing AWS credentials in development environments

Provisioning instructions for infrastructure can be referred with Infrastructure deployments

Example container installation

Add Helm repo

helm repo add bitnami https://charts.bitnami.com/bitnami

Install mysql

helm install bitnami/mysql --generate-name

Check installed containers

Example Rest application installation

Example app source code can be found from:

Create a Dockerfile, if is not exists

Run docker build

Login to AWS ECR

Using eu-central-1 as a region

And 814942682479 as a aws account id

Push image to ECR

Create Helm chart

Change correct values for values.yml

Install Helm chart

Verify that installation executed successfully

Test application

Create tunnel to application and after that open browser with http://localhost:18081/api/v1

Building block isolation

Building blocks inside Govstack environment are isolated by Kubernetes namespace. Every organisation and/or business-case get own namespace and related building blocks are installed “under” that namespace. This functionality ensures that building blocks can interact between other building blocks inside namespace but can’t directly interact with other namespace BB’s. Also capacity reservations can be easily adjusted if needed.

Building block CI/CD

NOTICE: This MVP solution for now and in the future there will be Portal/Automated functionality handle CI/CD process.

Every Building block should contains helm charts inside building block repository.

TODO: Add more information about helm charts

CI/CD pipelines are develop to together with BB developer and Sandbox admin.

Only Sandbox Admin are allowed to maintain/install BB:S to Sandbox environment.

CircleCI

OIDC

For general authentication to AWS Resources CircleCI build jobs uses OIDC (OpenID Connect indentity) tokens. So we are not storing AWS secrets to CircleCI.

For every pipeline, separate iam role is needed to be created in aws side for it to be authenticated to the kubernetes. This is done by adding it to env.hcl and then running terragrunt stacks circleci and eks in that order.

For information: https://circleci.com/blog/openid-connect-identity-tokens/ and https://circleci.com/docs/openid-connect-tokens/

Note!

In CircleCI jobs we need to use context, CircleCI stores OIDC token to context during workflow execution

Contexts provide a mechanism for securing and sharing environment variables across projects. The environment variables are defined as name/value pairs and are injected at runtime. This document describes creating and using contexts in CircleCI. CircleCI - context

CircleCI auth to aws

After OIDC is in place, pipelines needs an AWS IAM role that authenticates them to deploy to kubernetes. This is done with the circleci module in terraform. In the corresponding environment, you need to add the correct values to env.hcl

Example from development environment

Org id is the id of the circleci organisation, this can be found from the project settings

ssl thumbprint can be calculated following Obtain the thumbprint for an OpenID Connect identity provider - AWS Identity and Access Management or with the tool in infra repo tools/ssl_thumbprint.sh that basicly just implements that guide.

In projects project_id can be found from the pipelines configuration in circleci and name you can decide yourself.

After applying the changes to circleci stack, run also the eks stack so those roles are added to aws_auth list

Kubernetes auth

Build jobs also requires access to Kubernetes when installing new deployments. Kubernetes controls it’s own access, so access to AWS services is not enough.

Adding permission to specific user/role requires to modify aws-auth inside kube-system namespace. This can also be done with terraform, as is done with the circleci pipeline roles

Current settings can be check:

Kubernetes auth

Kubernetes RBAC

And settings can be add/edit using kubectl edit configmap/aws-auth or using eksctl

For example, following command add required permissions to build job:

Quality ensure of BB

Before BB is installed to GovStack environment it must go thru API Testing process and results must to provide to Sandbox admin. When Image is pushed to ECR vulnerability scan is executed and BB must be at level Medium or lower, see below. ? Otherwise BB is not allowed to install Sandbox environment.

CVSS scores
- Critical (9.0-10.0) score are not allowed
- High scores ( 7.0 - 8.9 ) must be analysed before install and take mitigate actions if possible
- Lower scores can be installed

TODO: specify which level vulnerability issues can be