Overview

This is an architecture to deploy IDBB in the EKS K8s Cluster

Open

• IDBB(MOSIP) modules are deployed in the form of microservices in a Kubernetes cluster.

• https://docs.mosip.io/1.2.0/deployment/wireguard/wireguard-bastion is used as a trust network extension to access the admin, control, and observation pane.
  Note: As Wireguard is not needed in Govstack Playground, we have not deployed wireguard.

• It is also used for on-the-field registrations.

• MOSIP uses AWS load balancers for:

  • SSL termination

  • Reverse Proxy

  • CDN/Cache management

  • Load balancing

• Kubernetes cluster is administered using the Rancher and EKS

• In V3, we have two Kubernetes clusters:

  • Observation Cluster - This cluster is a part of the observation plane and it helps in administrative tasks. By design, this is kept independent of the actual cluster as a good security practice and to ensure clear segregation of roles and responsibilities. As a best practice, this cluster or its services should be internal and should never be exposed to the external world.
    Note: but, as we have not needed this in Govstack Playground, we have not deployed the rancher cluster.

    • Rancher is used for managing the Mosip cluster.

    • Keycloak in this cluster is used for cluster user access management.

    • It is recommended to configure log monitoring and network monitoring in this cluster.

    • In case you have a internal container registry, then it should run here.

  • IDBB(MOSIP) Cluster - This cluster runs all the MOSIP components and certain third-party components to secure the cluster, API’s and Data.

    • MOSIP External Components

    • MOSIP Services

• For more information → docs.mosip.io

Deployment Repo’s to install IDBB using EKS cluster

• k8s-infra : contains scripts to install and configure Kubernetes cluster with required Istio, monitoring, logging, and alerting tools.

• mosip-infra : contains deployment scripts to run charts in the defined sequence.

• mosip-config : contains all the configuration files required by the MOSIP modules.

• mosip-helm : contains packaged helm charts for all the MOSIP modules.

• Helm repos you need to add to your local before proceeding with deployment.

The below repositories can also be used as optional to deploy IDBB, which will deploy using tf-govstack helm charts.

• IDBB-infra

• IDBB-helm

• Helm repos you need to add to your local before proceeding with deployment.
  helm repo add tf-govstack https://tf-govstack.github.io/mosip-helm

AWS EKS Cluster Setup for IDBB

IDBB K8’s Cluster Global configmap, Ingress, and Storage Class setup

• Global configmap: Global configmap contains the list of necessary details to be used throughout the namespaces of the cluster for common details.

  • Use this branch global-configmap

  • Copy global_configmap.yaml.sample to global_configmap.yaml.

  • Update the domain names in global_configmap.yaml and run.

    • kubectl apply -f global_configmap.yaml

Ingress and load balancer (LB)

Ingress is not installed by default on EKS. We use Istio ingress gateway controller to allow traffic in the cluster. Two channels are created - public and internal.

• Install Istioctl as given here Version : 1.17.0

• Install ingress Istio

Note: In case you need all services/api’s to be accessible without wireguard then please edit istio gateways once istio deployed. You can refer below screenshots.

• public gateway (redirecting to the internal host to access all services)

Open

• Internal gateway (redirecting to external host it may use or may not)

• Make sure you want to access all services publicly at the istio-level. Then, once this above step is done you need to configure all idbb/mosip internal and external services gateway and VirtualServices accordingly. Please find the screenshot below for the same.

• At gateway level (for each services need to check and edit this)

• At VirtualService level (for each services need to check and edit this)

• If you don't want all services publicly available then go with port forwarding for the respective services you want access. But not sure this method may affect to access all the api's.

Note : In normal deployment scenario Internal-gateway is redirecting to internal host and Public-gateway is redirecting to public host. As we need all the services to be publicly available without wireguard we have inter changed the redirecting hosts.

Load Balancers

The above istio install will spin off a load balancer on AWS. You may view them on the AWS console. These may be also seen with this command and make sure CLUSTER-IP and EXTERNAL-IP should be present. Then only you are able to map domains to load balancer.

• Make sure in EKS-IDBB deployment use only one ingress-gateway-public LB for both public and private routing and keep ingress-gateway-internal as optional.

• TLS termination is supposed to be on LB. So all our traffic coming to the ingress controller shall be HTTP.

• Obtain AWS TLS certificate as given here. Ignore if you have already created one.

• Add the certificates and 443 access to the LB listener.

  • Update listener TCP->443 to TLS->443 and point to the certificate of domain name that belongs to your cluster. Same as you have done for rancher LB.

  • Forward TLS->443 listner traffic to target group that corresponds to listner on port 80 of respective Loadbalancers. This is because after TLS termination the protocol is HTTP so we must point LB to HTTP port of ingress controller.

• Update health check ports of LB target groups to node port corresponding to port 15021. You can see the node ports with

• Enable Proxy Protocol v2 on all target groups.

• Make sure all subnets are included in Availability Zones for the LB. Description --> Availability Zones --> Edit Subnets

• Make sure to delete the listenrs for port 80 and 15021 from each of the loadbalancers as we restrict unsecured port 80 access over HTTP. Find the below image for the same changes.

DNS Requirements for IDBB-Govstack cluster:

• Point all your domain names to EXTERNAL LoadBalancers DNS/IP.

• On AWS this may be done on Route 53 console.

• After the Go-live decision enables public-access.md → You can refer to this. The same steps you have done on istio level above.

• Find the below image for routing on Route53.

• The below image shows an example placeholder for hostnames, the actual name itself varies from organization to organization.

• Only proceed to DNS mapping after the ingress gateways are installed and the Nginx reverse proxy is set up. You can refer sandbox route53 for now.

Check Overall if nginx and Istio wiring is set correctly

• Install httpbin for testing the wiring.

IDBB Govstack External Dependencies setup

• External Dependencies are a set of external requirements needed for the functioning of MOSIP’s core services like DB, object store, hsm, etc. Please follow the sequence from install-all.sh and install the services one by one shown in install-all.sh. ref: sequence to install
  Note: Ignore docker-secrets not required and conf-secrets you will find while installing mosip-internal-modules.

  Connnect to DB using Postgres password and check all the DB’s got created or not. ( You can connect with any database app, ex: PGadmin, DBeaver)

• And once DB deployment is done you can follow from keycloak installation as it is and complete the external modules installation.

• Create a Google Recaptcha v2 ("I am not a Robot") from the Google Recaptcha admin while installing the Captcha service. Refer: reCAPTCHA

IDBB Govstack Modules Deployment

• Now external dependencies are installed, you can continue with IDBB(MOSIP) services deployment.

• Please follow this sequence to install them one by one mosip-internal services. sequence to install ( No need to install pre-reg as part of govstack don't need prereg module)
  NOTE : Please do not run install-all.sh it may fail while installing the services, so just refer to it for the sequence of installation.

• And while installing config-server please make sure you create a branch with respect to your env/cluster name in the mosip-config repository from this branch (govstack-v1.2.0.1-B3 ) Repository you can use to create your branch → tf-govstack/mosip-config and then edit values.yaml file in config-server to take your config branch.

• While installing partner onboarding do it last after regclient module and make sure all services are up and running or not, then install and provide the minio URL (http://minio.minio:9000) and Bucket name (onboarder). But make sure you need to onboard partners only for mosip-services and esignet, digital card services onboarding will take care of once esignet and digtalcard deployment is done because this is considered as an external module so.

• Make these false in values.yaml while running partner onboarder.

  Esignet Deployment Procedure

• Clone the repository to your local to deploy Esignet

  • git clone -b tf-develop-infra https://github.com/tf-govstack/esignet/tree/tf-develop-infra

  • git clone -b tf-develop-infra https://github.com/tf-govstack/esignet-mock-services/tree/tf-develop-infra

CREATING MOSIP_ESIGNET DATABASE:

COPY POSTGRES PASSWORD. EKS CLUSTER -> POSTGRES NS -> STORAGE -> SECRETS -> COPY postgres-postgresql PASSWORD

Run below command to deploy DB
./deploy.sh deploy.properties OR bash deploy.sh deploy.properties

enter DB password multiple times it will be asked

NOTE: VERIFY IN PGADMIN WEATHER MOSIP_ESIGNET DB IS GENERATED OR NOT

GO TO RANCHER -> POSTGRES -> STORAGE -> SECRETS -> COPY db-common-secrets PASSWORD and Copy

OPEN PG ADMIN
click on Login/Group Role --> right click on esignetuser --> properties --> defination (paste db-common-secrets as a password)
click on save

DEPLOY REDIS, ESIGNET AND OIDC-UI :

cd esignet/helm/esignet
helm dependency build
helm repo update
cd esignet/helm
./install-all.sh

CREATING MOSIP_MOCKIDENTITYSYSTEM DATABASE:

cd esignet-mock-services/db_scripts/mosip_mockidentitysystem
chmod +x deploy.sh
nano deploy.properties

COPY POSTGRES PASSWORD. GO TO EKS CLUSTER -> POSTGRES NS -> STORAGE -> SECRETS -> COPY postgres-postgresql PASSWORD

RUN BELOW COMMAND TO GENERATE IDP DATABASE
./deploy.sh deploy.properties OR bash deploy.sh deploy.properties

enter DB password multiple times it will be asked

GO TO EKS CLUSTER -> POSTGRES NS -> STORAGE -> SECRETS -> COPY db-common-secrets PASSWORD and Copy

OPEN PG ADMIN
click on Login/Group Role --> right click on mockidentitysystemuser --> properties --> defination (paste db-common-secrets as a password)
click on save

DEPLOY ESIGNET-MOCK-SERVICES :

cd esignet-mock-services/helm/mock-relying-party-service
helm dependency build
helm repo update

cd esignet-mock-services/helm/mock-identity-system
helm dependency build
helm repo update

cd esignet-mock-services/helm/mock-relying-party-ui
helm dependency build
helm repo update

cd esignet-mock-services/helm
./install-all.sh

Privatekey-oidc :-

Open

Please provide client private key file : /home/shiv/B3-E2E/privatekey-oidc (path of above file in your local)
Please provide jwe userinfo private key file : /home/shiv/B3-E2E/privatekey-oidc (path of above file in your local)

Please provide Esignet service url : ( default: http://esignet.esignet/v1/esignet ) (do enter)

Please provide mock relying party ui domain (eg: healthservices.sandbox.xyz.net ) : healthservices.tf1.idencode.link (provide you host)

DEPLOYMENT OF PARTNER-ONBOARDER FOR ESIGNET :

To onboard esignet partners (use this branch partner-onboarder )
Make sure edit values.yaml as per below instructions

nano values.yml

– name: esignet; enabled: true – name: resident-oidc; enabled: true (Make sure others should be false apart from these two)

Make sure edit install.sh with this name while uploading certs or else it will give error.

RUN:
./install.sh

ONBOARDING THE DEFAULT ESIGNET PARTNER

a). After successfull partner onboarder run for esignet , download html reports from onboarder bucket of object store minio.

b). Get licensekey from response body of request create-the-MISP-license-key-for-partner from the report e-signet.html

c). Update & commit value of mosip.esignet.misp.license.key parameter with licensekey value from last step in esignet-default.properties

d). Restart esignet pod.

ONBOARDING THE DEFAULT RESIDENT-OIDC PARTNER

a). After successfull partner onboarder run for resident-oidc , download html reports from onboarder bucket of object store minio .

b). Get clientId from response body of request create-oidc-client from the report resident-oidc.html .

c). Update & commit value of mosip.iam.module.clientID parameter with clientId value from last step in resident-default.properties

d). edit the mimoto-default.properties config for the below mentioned keys :

e). Restart resident pods.

DEPLOYMENT OF PARTNER-ONBOARDER FOR ESIGNET MOCK SERVICES :

• name: demo-oidc enabled: true (make sure apart from this others should be false)

Make sure edit install.sh with this name while uploading certs or else it will give error.

RUN:
./install.sh

FOLLOW THE BELOW SETPS CAREFULLY :-

1. ONBOARDING THE DEFAULT DEMO-OIDC PARTNER

   a). After successfull partner onboarder run for demo-oidc partner , download html reports from onboarder bucket of object store minio.

   b). Get CLIENT_ID from response body of request create-oidc-client from the report demo-oidc.html

   c). Update deployment of mock-relying-party-ui in esignet namespace with CLIENT_ID value from last step .

   d). As per screenshot get the private and public key pair (shown as selected in the screenshot ) from the response of the get-jwks request from the report demo-oidc.html

f). Restart mock-relying-party-service pod

Note: If facing intermittent connectivity issues while login esignet then please disable istio layer from softhsm namespace.

kubectl label ns softhsm istio-injection=disabled --overwrite

Deploy Digital Card Service

CLONE THE REPOSITORY MENTIONED BELOW

digital-card-service

CREATING MOSIP_DIGITALCARD DATABASE :

cd digital-card-service/db_scripts/mosip_digitalcard
nano deploy.properties

DB_SERVERIP=api-internal.tf1.idencode.link ---> edit internal api hostname EX. api-internal.tf1.idencode.link

COPY POSTGRES PASSWORD. GO TO RANCHER -> POSTGRES -> STORAGE -> SECRETS -> COPY postgres-postgresql PASSWORD

RUN BELOW COMMAND TO GENERATE DIGITAL CARD DATABASE
./deploy.sh deploy.properties OR bash deploy.sh deploy.properties

enter DB password multiple times it will be asked

RUN THE BELOW COMMAND FOR RESOLUTION OF THE ABOVE ERROR:

GO TO RANCHER -> POSTGRES -> STORAGE -> SECRETS -> COPY db-common-secrets PASSWORD and Copy

OPEN PG ADMIN
click on Login/Group Role --> right click on digitalcarduser --> properties --> defination (paste db-common-secrets as a password)
click on save

DEPLOY DIGITALCARD

cd digital-card-service/helm/digitalcard
./install.sh

Troubleshooting

• If you are facing any issues while accessing the domain names that could be because proxy-protocol is not enabled in the target groups. or routing is not done properly and LB listners configurations are not done properly so check everything once again.

• When accessing istio-system from terminal it should show DNS name of load balancer in EXTERNAL-IP section or else not able to access endpoints. It causes because of multiple security-groups attached to your nodes. Make sure only one security-group attached to each node.

• Sometimes while creating clusters the cluster will create successfully but it will not generate a cluster-config file in .kube folder then run this command to generate the config file after cluster creation.
  eksctl utils write-kubeconfig -n <clustername>

• Sometimes you will face PVC mount volumes issue for all the stateful sets, and completely env will go down because of restarting nodes or scaling down the nodes. So please avoid this in case of EKS or else you can use EFS as a default storage class to avoid such issues..

• If you want to decrease/increase the no of nodes, you can do that on EKS section NodeGroup group in AWS.

• If you want to delete the whole EKS setup follow this link here