Page Comparison

...

Code Block

K8S_NAMESPACE=im-xroad
X_ROAD_IMAGE_TAG="7.2.2-IAM"
K8S_CS_SS_DB_STORAGE_CLASS_NAME=gp3
K8S_TLD_NAME=im-xroad.playground.sandbox-playground.com
K8S_EXPOSE_SERVICES=false
AWS_ACCOUNT=`<account id>`
AWS_DEFAULT_REGION=eu-central-1
X_ROAD_METRICS_IMAGE_TAG=latest
PUBSUB_TAG=0.0.1-develop-c5e275ed
PUBSUB_MESSAGING_API_IMAGE_TAG=$PUBSUB_TAG
PUBSUB_MANAGEMENT_API_IMAGE_TAG=$PUBSUB_TAG
PUBSUB_MANAGEMENT_UI_IMAGE_TAG=$PUBSUB_TAG
PUBSUB_SUBSCRIBER_MOCK_IMAGE_TAG=$PUBSUB_TAG
PUBSUB_DB_SCHEMA_IMAGE_TAG=$PUBSUB_TAG
MANAGEMENT_API_XROAD_SECURITY_SERVER_BASE_URL=<https://sandbox-xroad-ss3.${K8S_NAMESPACE}.svc.cluster.local:4000/api/v1/>
MANAGEMENT_API_XROAD_PUBSUB_SUBSYSTEM_IDENTIFIER=SANDBOX:GOV:PROVIDER:TEST
K8S_SUBNET_ALLOW_LIST=""

Core X-Road Deployment

...

Most deployments in the provided Helm charts do not specify resource requests and limits, which does not work well with Karpenter and autoscaling. One possibility is to add a default resource limit for the namespace ( 1Gi is probably too much for some services):

Code Block

#limits.yml
apiVersion: v1
kind: LimitRange
metadata:
  name: default-mem-limit
spec:
  limits:
  - default:
      memory: 1Gi
    defaultRequest:
      memory: 1Gi
    type: Container
#
kubectl apply -f limits.yml -n $K8S_NAMESPACE

Core X-Road Deployment

Code Block

echo "--- xroad_deploy ---"
helm upgrade --install --atomic --debug \
--wait --timeout 60m \
--namespace "$K8S_NAMESPACE" \
--set apiService.create=true \
--set global.serviceExt.enabled=$K8S_EXPOSE_SERVICES \
--set-string global.storageClassName=${K8S_CS_SS_DB_STORAGE_CLASS_NAME} \
--set-string sandbox-im-x-road-xroad-ss.iamAuthorizationUritokenPin="<https://iam-${K8S_TLD_NAME}/realms/pubsub-realm/protocol/openid-connect/auth>"1234" \
--set-string xroad-cs.tokenPin="1234" \
--set-string sandbox-im-xxroad-road-sscs.iamTokenUriserverTag="<https://iam-${K8SX_ROAD_TLDIMAGE_NAMETAG}/realms/pubsub-realm/protocol/openid-connect/token>-cs" \
--set-string sandbox-im-x-road-ss.iamUserInfoUriservers.ss1="<https://iam-${K8SX_ROAD_TLDIMAGE_NAMETAG}/realms/pubsub-realm/protocol/openid-connect/userinfo>-ss1" \
--set-string global.registrysandbox-im-x-road-ss.servers.ss2="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.comX_ROAD_IMAGE_TAG}-ss2" \
sandbox--imset-xroad ./x-road/string sandbox-im-x-road

...

-ss.servers.ss3="${X_ROAD_IMAGE_TAG}-ss3" \
--set-string sandbox-im-x-road-ss.iamIssuerUri="<https://iam-${K8S_TLD_NAME}/realms/pubsub-realm>" \
--set-string sandbox-im-x-road-ss.iamAuthorizationUri="<https://iam-${K8S_TLD_NAME}/realms/pubsub-realm/protocol/openid-connect/auth>" \
--set-string sandbox-im-x-road-ss.iamTokenUri="<https://iam-${K8S_TLD_NAME}/realms/pubsub-realm/protocol/openid-connect/token>" \
--set-string sandbox-im-x-road-ss.iamUserInfoUri="<https://iam-${K8S_TLD_NAME}/realms/pubsub-realm/protocol/openid-connect/userinfo>" \
--set-string global.registry="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com" \
sandbox-im-xroad ./x-road/sandbox-im-x-road

Keycloak is part of PubSub, but it is shared with X-Road to provide authentication. For keycloak deployment, create new pubsub/keycloak/config/pubsub-realm-sandbox.json file based on existing template and change the URLs to be consistent with the selected external domain (K8S_TLD_NAME). Also update `pubsub/keycloak/values.yml` with

Code Block
keycloak: ... httpPort: 8080 ... args: [ "start", "--http-port=8080", "--import-realm", "--hostname-strict=false", "--proxy=edge" ]

Code Block

echo "--- keycloak_deploy ---"
helm upgrade --install --atomic --debug \
--wait  --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set serviceExt.enabled=$K8S_EXPOSE_SERVICES \
--set-string config.realmConfigFile="config/pubsub-realm-sandbox.json" \
keycloak-chart ./pubsub/keycloak

...

Code Block

MANAGEMENT_API_XROAD_ADMIN_API_KEY=<API KEY>

echo "--- x_road_artemis_deploy ---"
helm upgrade --install --atomic --debug \
--wait  --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set serviceExt.enabled=${K8S_EXPOSE_SERVICES} \
--set-string artemis.topLevelDomainName=${K8S_TLD_NAME} \
--set-string artemis.subnetAllowList="${K8S_SUBNET_ALLOW_LIST}" \
artemis ./pubsub/artemis/

echo "--- x_road_im-msg-db_deploy ---"
helm upgrade --install --atomic --debug \
--wait  --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set-string topLevelDomainName=${K8S_TLD_NAME} \
--set-string subnetAllowList="${K8S_SUBNET_ALLOW_LIST}" \
im-msg-db oci://registry-1.docker.io/bitnamicharts/postgresql \
-f ./pubsub/im-msg-db/values.yaml

echo "--- im-msg-db-schema_uninstall ---"
helm uninstall --debug  --wait  --timeout 60m \
--namespace "$K8S_NAMESPACE" \
im-msg-db-schema

echo "--- x_road_im-msg-db-schema_deploy ---"
helm upgrade --install --atomic --debug \
--wait  --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set-string imMsgBbSchema.image.tag=${PUBSUB_DB_SCHEMA_IMAGE_TAG} \
--set-string imMsgBbSchema.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/pubsub/schema" \
im-msg-db-schema ./pubsub/im-msg-db-schema/

echo "--- x_road_messaging-api_deploy ---"
helm upgrade --install --atomic --debug \
--wait  --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set serviceExt.enabled=${K8S_EXPOSE_SERVICES} \
--set-string ingress.topLevelDomainName=${K8S_TLD_NAME} \
--set-string ingress.subnetAllowList="${K8S_SUBNET_ALLOW_LIST}" \
--set-string messagingApi.image.tag=${PUBSUB_MESSAGING_API_IMAGE_TAG} \
--set-string messagingApi.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/pubsub/messaging-api" \
messaging-api ./pubsub/messaging-api/

echo "--- x_road_management-api_deploy ---"
helm upgrade --install --atomic --debug \
--wait  --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set serviceExt.enabled=${K8S_EXPOSE_SERVICES} \
--set-string serviceExt.topLevelDomainName=${K8S_TLD_NAME} \
--set-string serviceExt.subnetAllowList="${K8S_SUBNET_ALLOW_LIST}" \
--set-string managementApi.image.tag=${PUBSUB_MANAGEMENT_API_IMAGE_TAG} \
--set-string managementApi.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/pubsub/management-api" \
--set-string managementApi.xroadAdminClient.securityServerBaseUrl="${MANAGEMENT_API_XROAD_SECURITY_SERVER_BASE_URL}" \
--set-string managementApi.xroadAdminClient.apiKey="${MANAGEMENT_API_XROAD_ADMIN_API_KEY}" \
--set-string managementApi.xroadAdminClient.pubsubSubsystemIdentifier="${MANAGEMENT_API_XROAD_PUBSUB_SUBSYSTEM_IDENTIFIER}" \
--set-string managementApi.oauth2.issuerUri="<https://iam-${K8S_TLD_NAME}/realms/pubsub-realm>" \
--set-string managementApi.cors.allowedOrigins="<https://management-ui-${K8S_TLD_NAME}>" \
management-api ./pubsub/management-api/

echo "--- x_road_management-ui_deploy ---"
helm upgrade --install --atomic --debug \
--wait  --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set serviceExt.enabled=${K8S_EXPOSE_SERVICES} \
--set-string ingress.topLevelDomainName=${K8S_TLD_NAME} \
--set-string ingress.subnetAllowList="${K8S_SUBNET_ALLOW_LIST}"$K8S_NAMESPACE" \
--set-string managementUiserviceExt.image.tagenabled=${PUBSUBK8S_MANAGEMENT_UI_IMAGE_TAGEXPOSE_SERVICES} \
--set-string managementUiingress.image.repositorytopLevelDomainName="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/pubsub/management-ui"K8S_TLD_NAME} \
--set-string managementUiingress.iamIssuerUrisubnetAllowList="<https://iam-${K8S_SUBNET_TLDALLOW_NAME}/realms/pubsub-realm>LIST}" \
--set-string managementUi.image.managementApiUri="<https://management-ui-${K8S_TLD_NAME}>" \
management-ui ./pubsub/management-ui/

echo "--- x_road_subscriber-mock_deploy ---"
helm upgrade --install --atomic --debug \
--wait  --timeout 15m \
--namespace "$K8S_NAMESPACEtag=${PUBSUB_MANAGEMENT_UI_IMAGE_TAG} \
--set-string managementUi.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/pubsub/management-ui" \
--set-string managementUi.iamIssuerUri="<https://iam-${K8S_TLD_NAME}/realms/pubsub-realm>" \
--set-string subscriberMockmanagementUi.image.tagmanagementApiUri="<https://management-ui-${PUBSUB_SUBSCRIBER_MOCK_IMAGE_TAG}" \
--set-string subscriberMock.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/pubsub/subscriber-mock" \
subscriber-mock ./pubsub/subscriber-mock/

Fixes

Most deployments do not specify resource requests and limits, which does not work well with Karpenter and autoscaling. One possibility is to add a default resource limit for the namespace (1Gi is probably too much for some services, and can be too little for others)

...

K8S_TLD_NAME}>" \
management-ui ./pubsub/management-ui/

echo "--- x_road_subscriber-mock_deploy ---"
helm upgrade --install --atomic --debug \
--wait  --timeout 15m \
--namespace "$K8S_NAMESPACE" \
--set-string subscriberMock.image.tag="${PUBSUB_SUBSCRIBER_MOCK_IMAGE_TAG}" \
--set-string subscriberMock.image.repository="${AWS_ACCOUNT}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/im/pubsub/subscriber-mock" \
subscriber-mock ./pubsub/subscriber-mock/

Versions Compared

Old Version 2

New Version Current

Key

Core X-Road Deployment

Core X-Road Deployment

Fixes