X-Road Installation in Sandbox

Installation is based on information found in https://github.com/nortal/GovStack-IM-BB-SandBox-Deployment. The procedure is defined as a GitLab CI pipeline, so some adaptation was needed.

Due to existing playground deployment, the custom images were already in ECR. The deployment uses a slightly modified X-Road, and building X-Road is needed without access to the images.

Deployment

git clone https://github.com/nortal/GovStack-IM-BB-SandBox-Deployment

Set up ENV variables:

K8S_NAMESPACE=im-xroad
X_ROAD_IMAGE_TAG="7.2.2-IAM"
K8S_CS_SS_DB_STORAGE_CLASS_NAME=gp3
K8S_TLD_NAME=im-xroad.playground.sandbox-playground.com
K8S_EXPOSE_SERVICES=false
AWS_ACCOUNT=`<account id>`
AWS_DEFAULT_REGION=eu-central-1
X_ROAD_METRICS_IMAGE_TAG=latest
PUBSUB_TAG=0.0.1-develop-c5e275ed
PUBSUB_MESSAGING_API_IMAGE_TAG=$PUBSUB_TAG
PUBSUB_MANAGEMENT_API_IMAGE_TAG=$PUBSUB_TAG
PUBSUB_MANAGEMENT_UI_IMAGE_TAG=$PUBSUB_TAG
PUBSUB_SUBSCRIBER_MOCK_IMAGE_TAG=$PUBSUB_TAG
PUBSUB_DB_SCHEMA_IMAGE_TAG=$PUBSUB_TAG
MANAGEMENT_API_XROAD_SECURITY_SERVER_BASE_URL=<https://sandbox-xroad-ss3.${K8S_NAMESPACE}.svc.cluster.local:4000/api/v1/>
MANAGEMENT_API_XROAD_PUBSUB_SUBSYSTEM_IDENTIFIER=SANDBOX:GOV:PROVIDER:TEST
K8S_SUBNET_ALLOW_LIST=""

Core X-Road Deployment

echo "--- xroad_deploy ---"
helm upgrade --install --atomic --debug \
--wait --timeout 60m \
--namespace "$K8S_NAMESPACE" \
--set apiService.create=true \
--set global.serviceExt.enabled=$K8S_EXPOSE_SERVICES \
--set-string global.storageClassName=${K8S_CS_SS_DB_STORAGE_CLASS_NAME} \
--set-string xroad-ss.tokenPin="1234" \
--set-string xroad-cs.tokenPin="1234" \
--set-string sandbox-im-xroad-cs.serverTag="${X_ROAD_IMAGE_TAG}-cs" \
--set-string sandbox-im-x-road-ss.servers.ss1="${X_ROAD_IMAGE_TAG}-ss1" \
--set-string sandbox-im-x-road-ss.servers.ss2="${X_ROAD_IMAGE_TAG}-ss2" \
--set-string sandbox-im-x-road-ss.servers.ss3="${X_ROAD_IMAGE_TAG}-ss3" \
--set-string sandbox-im-x-road-ss.iamIssuerUri="<https://iam-${K8S_TLD_NAME}/realms/pubsub-realm>" \
--set-string sandbox-im-x-road-ss.iamAuthorizationUri="<https://iam-${K8S_TLD_NAME}/realms/pubsub-realm/protocol/openid-connect/auth>" \
--set-string sandbox-im-x-road-ss.iamTokenUri="<https://iam-${K8S_TLD_NAME}/realms/pubsub-realm/protocol/openid-connect/token>" \
--set-string sandbox-im-x-road-ss.iamUserInfoUri="<https://iam-${K8S_TLD_NAME}/realms/pubsub-realm/protocol/openid-connect/userinfo>" \
--set-string global.registry="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com" \
sandbox-im-xroad ./x-road/sandbox-im-x-road

Keycloak is part of PubSub, but it is shared with X-Road to provide authentication. For keycloak deployment, create new pubsub/keycloak/config/pubsub-realm-sandbox.json file based on existing template and change the URLs to be consistent with the selected external domain (K8S_TLD_NAME)

echo "--- keycloak_deploy ---"
helm upgrade --install --atomic --debug \
--wait  --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set serviceExt.enabled=$K8S_EXPOSE_SERVICES \
--set-string config.realmConfigFile="config/pubsub-realm-sandbox.json" \
keycloak-chart ./pubsub/keycloak

echo "--- x_road_metrics_uninstall ---"
helm uninstall --debug  --wait  --timeout 60m \
--namespace "$K8S_NAMESPACE" \
xroad-metrics

echo "--- x_road_metrics_deploy ---"
helm upgrade --install --atomic --debug \
--wait  --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set-string database.initDb.image="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/xroad-metrics/init-db" \
--set-string database.initDb.image_tag=${X_ROAD_METRICS_IMAGE_TAG} \
--set-string collector.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/xroad-metrics/collector" \
--set-string collector.image.tag=${X_ROAD_METRICS_IMAGE_TAG} \
--set-string corrector.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/xroad-metrics/corrector" \
--set-string corrector.image.tag=${X_ROAD_METRICS_IMAGE_TAG} \
xroad-metrics ./x-road/x-road-metrics

PubSub

Set up limited access to X-Road Admin UIs (see https://github.com/GovStackWorkingGroup/sandbox-infra/blob/SND-651/live/playground/kube/im-xroad.tf for an example).
Create MANAGEMENT_API_XROAD_ADMIN_API_KEY on security server SS3

MANAGEMENT_API_XROAD_ADMIN_API_KEY=<API KEY>

echo "--- x_road_artemis_deploy ---"
helm upgrade --install --atomic --debug \
--wait  --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set serviceExt.enabled=${K8S_EXPOSE_SERVICES} \
--set-string artemis.topLevelDomainName=${K8S_TLD_NAME} \
--set-string artemis.subnetAllowList="${K8S_SUBNET_ALLOW_LIST}" \
artemis ./pubsub/artemis/

echo "--- x_road_im-msg-db_deploy ---"
helm upgrade --install --atomic --debug \
--wait  --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set-string topLevelDomainName=${K8S_TLD_NAME} \
--set-string subnetAllowList="${K8S_SUBNET_ALLOW_LIST}" \
im-msg-db oci://registry-1.docker.io/bitnamicharts/postgresql \
-f ./pubsub/im-msg-db/values.yaml

echo "--- im-msg-db-schema_uninstall ---"
helm uninstall --debug  --wait  --timeout 60m \
--namespace "$K8S_NAMESPACE" \
im-msg-db-schema

echo "--- x_road_im-msg-db-schema_deploy ---"
helm upgrade --install --atomic --debug \
--wait  --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set-string imMsgBbSchema.image.tag=${PUBSUB_DB_SCHEMA_IMAGE_TAG} \
--set-string imMsgBbSchema.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/pubsub/schema" \
im-msg-db-schema ./pubsub/im-msg-db-schema/

echo "--- x_road_messaging-api_deploy ---"
helm upgrade --install --atomic --debug \
--wait  --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set serviceExt.enabled=${K8S_EXPOSE_SERVICES} \
--set-string ingress.topLevelDomainName=${K8S_TLD_NAME} \
--set-string ingress.subnetAllowList="${K8S_SUBNET_ALLOW_LIST}" \
--set-string messagingApi.image.tag=${PUBSUB_MESSAGING_API_IMAGE_TAG} \
--set-string messagingApi.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/pubsub/messaging-api" \
messaging-api ./pubsub/messaging-api/

echo "--- x_road_management-api_deploy ---"
helm upgrade --install --atomic --debug \
--wait  --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set serviceExt.enabled=${K8S_EXPOSE_SERVICES} \
--set-string serviceExt.topLevelDomainName=${K8S_TLD_NAME} \
--set-string serviceExt.subnetAllowList="${K8S_SUBNET_ALLOW_LIST}" \
--set-string managementApi.image.tag=${PUBSUB_MANAGEMENT_API_IMAGE_TAG} \
--set-string managementApi.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/pubsub/management-api" \
--set-string managementApi.xroadAdminClient.securityServerBaseUrl="${MANAGEMENT_API_XROAD_SECURITY_SERVER_BASE_URL}" \
--set-string managementApi.xroadAdminClient.apiKey="${MANAGEMENT_API_XROAD_ADMIN_API_KEY}" \
--set-string managementApi.xroadAdminClient.pubsubSubsystemIdentifier="${MANAGEMENT_API_XROAD_PUBSUB_SUBSYSTEM_IDENTIFIER}" \
--set-string managementApi.oauth2.issuerUri="<https://iam-${K8S_TLD_NAME}/realms/pubsub-realm>" \
--set-string managementApi.cors.allowedOrigins="<https://management-ui-${K8S_TLD_NAME}>" \
management-api ./pubsub/management-api/

echo "--- x_road_management-ui_deploy ---"
helm upgrade --install --atomic --debug \
--wait  --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set serviceExt.enabled=${K8S_EXPOSE_SERVICES} \
--set-string ingress.topLevelDomainName=${K8S_TLD_NAME} \
--set-string ingress.subnetAllowList="${K8S_SUBNET_ALLOW_LIST}" \
--set-string managementUi.image.tag=${PUBSUB_MANAGEMENT_UI_IMAGE_TAG} \
--set-string managementUi.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/pubsub/management-ui" \
--set-string managementUi.iamIssuerUri="<https://iam-${K8S_TLD_NAME}/realms/pubsub-realm>" \
--set-string managementUi.managementApiUri="<https://management-ui-${K8S_TLD_NAME}>" \
management-ui ./pubsub/management-ui/

echo "--- x_road_subscriber-mock_deploy ---"
helm upgrade --install --atomic --debug \
--wait  --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set-string subscriberMock.image.tag="${PUBSUB_SUBSCRIBER_MOCK_IMAGE_TAG}" \
--set-string subscriberMock.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/pubsub/subscriber-mock" \
subscriber-mock ./pubsub/subscriber-mock/

Fixes

Most deployments do not specify resource requests and limits, which does not work well with Karpenter and autoscaling. One possibility is to add a default resource limit for the namespace (1Gi is probably too much for some services, and can be too little for others)
```
apiVersion: v1
kind: LimitRange
metadata:
  name: default-mem-limit
spec:
  limits:
  - default:
      memory: 1Gi
    defaultRequest:
      memory: 1Gi
    type: Container
```