...
| Code Block |
|---|
K8S_NAMESPACE=im-xroad
X_ROAD_IMAGE_TAG="7.2.2-IAM"
K8S_CS_SS_DB_STORAGE_CLASS_NAME=gp3
K8S_TLD_NAME=im-xroad.playground.sandbox-playground.com
K8S_EXPOSE_SERVICES=false
AWS_ACCOUNT=`<account id>`
AWS_DEFAULT_REGION=eu-central-1
X_ROAD_METRICS_IMAGE_TAG=latest
PUBSUB_TAG=0.0.1-develop-c5e275ed
PUBSUB_MESSAGING_API_IMAGE_TAG=$PUBSUB_TAG
PUBSUB_MANAGEMENT_API_IMAGE_TAG=$PUBSUB_TAG
PUBSUB_MANAGEMENT_UI_IMAGE_TAG=$PUBSUB_TAG
PUBSUB_SUBSCRIBER_MOCK_IMAGE_TAG=$PUBSUB_TAG
PUBSUB_DB_SCHEMA_IMAGE_TAG=$PUBSUB_TAG
MANAGEMENT_API_XROAD_SECURITY_SERVER_BASE_URL=<https://sandbox-xroad-ss3.${K8S_NAMESPACE}.svc.cluster.local:4000/api/v1/>
MANAGEMENT_API_XROAD_PUBSUB_SUBSYSTEM_IDENTIFIER=SANDBOX:GOV:PROVIDER:TEST
K8S_SUBNET_ALLOW_LIST="" |
Most deployments in the provided Helm charts do not specify resource requests and limits, which does not work well with Karpenter and autoscaling. One possibility is to add a default resource limit for the namespace (
1Gi is probably too much for some services):
| Code Block |
|---|
#limits.yml
apiVersion: v1
kind: LimitRange
metadata:
name: default-mem-limit
spec:
limits:
- default:
memory: 1Gi
defaultRequest:
memory: 1Gi
type: Container
#
kubectl apply -f limits.yml -n $K8S_NAMESPACE |
Core X-Road Deployment
| Code Block |
|---|
echo "--- xroad_deploy ---"
helm upgrade --install --atomic --debug \
--wait --timeout 60m \
--namespace "$K8S_NAMESPACE" \
--set apiService.create=true \
--set global.serviceExt.enabled=$K8S_EXPOSE_SERVICES \
--set-string global.storageClassName=${K8S_CS_SS_DB_STORAGE_CLASS_NAME} \
--set-string xroad-ss.tokenPin="1234" \
--set-string xroad-cs.tokenPin="1234" \
--set-string sandbox-im-xroad-cs.serverTag="${X_ROAD_IMAGE_TAG}-cs" \
--set-string sandbox-im-x-road-ss.servers.ss1="${X_ROAD_IMAGE_TAG}-ss1" \
--set-string sandbox-im-x-road-ss.servers.ss2="${X_ROAD_IMAGE_TAG}-ss2" \
--set-string sandbox-im-x-road-ss.servers.ss3="${X_ROAD_IMAGE_TAG}-ss3" \
--set-string sandbox-im-x-road-ss.iamIssuerUri="<https://iam-${K8S_TLD_NAME}/realms/pubsub-realm>" \
--set-string sandbox-im-x-road-ss.iamIssuerUriiamAuthorizationUri="<https://iam-${K8S_TLD_NAME}/realms/pubsub-realm>-realm/protocol/openid-connect/auth>" \
--set-string sandbox-im-x-road-ss.iamAuthorizationUriiamTokenUri="<https://iam-${K8S_TLD_NAME}/realms/pubsub-realm/protocol/openid-connect/auth>token>" \
--set-string sandbox-im-x-road-ss.iamTokenUriiamUserInfoUri="<https://iam-${K8S_TLD_NAME}/realms/pubsub-realm/protocol/openid-connect/token>/protocol/openid-connect/userinfo>" \
--set-string global.registry="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com" \
sandbox-im-set-string xroad ./x-road/sandbox-im-x-road-ss.iamUserInfoUri="<https://iam-${ |
Keycloak is part of PubSub, but it is shared with X-Road to provide authentication. For keycloak deployment, create new pubsub/keycloak/config/pubsub-realm-sandbox.json file based on existing template and change the URLs to be consistent with the selected external domain (K8S_TLD_NAME
...
Keycloak is part of PubSub, but it is shared with X-Road to provide authentication. For keycloak deployment, create new pubsub/keycloak/config/pubsub-realm-sandbox.json file based on existing template and change the URLs to be consistent with the selected external domain (K8S_TLD_NAME)). Also update `pubsub/keycloak/values.yml` with
| Code Block |
|---|
keycloak:
...
httpPort: 8080
...
args:
[
"start",
"--http-port=8080",
"--import-realm",
"--hostname-strict=false",
"--proxy=edge"
] |
| Code Block |
|---|
echo "--- keycloak_deploy ---" helm upgrade --install --atomic --debug \ --wait --timeout 15m \ --namespace "$K8S_NAMESPACE" \ --set serviceExt.enabled=$K8S_EXPOSE_SERVICES \ --set-string config.realmConfigFile="config/pubsub-realm-sandbox.json" \ keycloak-chart ./pubsub/keycloak |
| Code Block |
|---|
echo "--- x_road_metrics_uninstall ---"
helm uninstall --debug --wait --timeout 60m \
--namespace "$K8S_NAMESPACE" \
xroad-metrics
echo "--- x_road_metrics_deploy ---"
helm upgrade --install --atomic --debug \
--wait --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set-string database.initDb.image="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/xroad-metrics/init-db" \
--set-string database.initDb.image_tag=${X_ROAD_METRICS_IMAGE_TAG} \
--set-string collector.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/xroad-metrics/collector" \
--set-string collector.image.tag=${X_ROAD_METRICS_IMAGE_TAG} \
--set-string corrector.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/xroad-metrics/corrector" \
--set-string corrector.image.tag=${X_ROAD_METRICS_IMAGE_TAG} \
xroad-metrics ./x-road/x-road-metrics |
PubSub
Set up limited access to X-Road Admin UIs (see https://github.com/GovStackWorkingGroup/sandbox-infra/blob/SND-651/live/playground/kube/im-xroad.tf for an example).
Create MANAGEMENT_API_XROAD_ADMIN_API_KEY on security server SS3
| Code Block |
|---|
MANAGEMENT_API_XROAD_ADMIN_API_KEY=<API KEY>
echo "--- x_road_artemis_deploy ---"
helm upgrade --install --atomic --debug \
--wait --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set serviceExt.enabled=${K8S_EXPOSE_SERVICES} \
--set-string artemis.topLevelDomainName=${K8S_TLD_NAME} \
--set-string artemis.subnetAllowList="${K8S_SUBNET_ALLOW_LIST}" \
artemis ./pubsub/artemis/
echo "--- x_road_im-msg-db_deploy ---"
helm upgrade --install --atomic --debug \
--wait --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set-string topLevelDomainName=${K8S_TLD_NAME} \
--set-string subnetAllowList="${K8S_SUBNET_ALLOW_LIST}" \
im-msg-db oci://registry-1.docker.io/bitnamicharts/postgresql \
-f ./pubsub/im-msg-db/values.yaml
echo "--- im-msg-db-schema_uninstall ---"
helm uninstall --debug --wait --timeout 60m \
--namespace "$K8S_NAMESPACE" \
im-msg-db-schema
echo "--- x_road_im-msg-db-schema_deploy ---"
helm upgrade --install --atomic --debug \
--wait --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set-string imMsgBbSchema.image.tag=${PUBSUB_DB_SCHEMA_IMAGE_TAG} \
--set-string imMsgBbSchema.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/pubsub/schema" \
im-msg-db-schema ./pubsub/im-msg-db-schema/
echo "--- x_road_messaging-api_deploy ---"
helm upgrade --install --atomic --debug \
--wait --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set serviceExt.enabled=${K8S_EXPOSE_SERVICES} \
--set-string ingress.topLevelDomainName=${K8S_TLD_NAME} \
--set-string ingress.subnetAllowList="${K8S_SUBNET_ALLOW_LIST}" \
--set-string messagingApi.image.tag=${PUBSUB_MESSAGING_API_IMAGE_TAG} \
--set-string messagingApi.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/pubsub/messaging-api" \
messaging-api ./pubsub/messaging-api/
echo "--- x_road_management-api_deploy ---"
helm upgrade --install --atomic --debug \
--wait --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set serviceExt.enabled=${K8S_EXPOSE_SERVICES} \
--set-string serviceExt.topLevelDomainName=${K8S_TLD_NAME} \
--set-string serviceExt.subnetAllowList="${K8S_SUBNET_ALLOW_LIST}" \
--set-string managementApi.image.tag=${PUBSUB_MANAGEMENT_API_IMAGE_TAG} \
--set-string managementApi.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/pubsub/management-api" \
--set-string managementApi.xroadAdminClient.securityServerBaseUrl="${MANAGEMENT_API_XROAD_SECURITY_SERVER_BASE_URL}" \
--set-string managementApi.xroadAdminClient.apiKey="${MANAGEMENT_API_XROAD_ADMIN_API_KEY}" \
--set-string managementApi.xroadAdminClient.pubsubSubsystemIdentifier="${MANAGEMENT_API_XROAD_PUBSUB_SUBSYSTEM_IDENTIFIER}" \
--set-string managementApi.oauth2.issuerUri="<https://iam-${K8S_TLD_NAME}/realms/pubsub-realm>" \
--set-string managementApi.cors.allowedOrigins="<https://management-ui-${K8S_TLD_NAME}>" \
management-api ./pubsub/management-api/
echo "--- x_road_management-ui_deploy ---"
helm upgrade --install --atomic --debug \
--wait --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set serviceExt.enabled=${K8S_EXPOSE_SERVICES} \
--set-string ingress.topLevelDomainName=${K8S_TLD_NAME} \
--set-string ingress.subnetAllowList="${K8S_SUBNET_ALLOW_LIST}" \
--set-string managementUi.image.tag=${PUBSUB_MANAGEMENT_UI_IMAGE_TAG} \
--set-string managementUi.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/pubsub/management-ui" \
--set-string managementUi.iamIssuerUri="<https://iam-${K8S_TLD_NAME}/realms/pubsub-realm>" \
--set-string managementUi.managementApiUri="<https://management-ui-${K8S_TLD_NAME}>" \
management-ui ./pubsub/management-ui/
echo "--- x_road_subscriber-mock_deploy ---"
helm upgrade --install --atomic --debug \
--wait --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set-string subscriberMock.image.tag="${PUBSUB_SUBSCRIBER_MOCK_IMAGE_TAG}" \
--set-string subscriberMock.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/pubsub/subscriber-mock" \
subscriber-mock ./pubsub/subscriber-mock/ |
...