Govstack Building Block Cookbook
Version | Author / Editor | Date | Changes |
---|---|---|---|
0.8 | Author - Max Carlson | May 6 2021 |
|
0.9 (Draft) | Editor - Dr. P.S. Ramkumar @PSRAMKUMAR | Nov 4, 2022 |
|
|
|
|
|
Summary: This document contains the main organization of activities, milestones and deliverables of Govstack BB workgroups mappable to a tracking process and platform such as sprints, epics and tasks on JIRA or such other project management platforms. The document also explains with examples the flow of activities from use case definitions, breakdown of steps within, mapping to specific BB services, definition of internal functional requirements, data models, interfaces and other technical aspects of building blocks leading the publication of building block specifications in specific templated formats. The document is also useful to identify the dependency, overlap and handover between various groups within Govstack (country engagement/product/tech committees/comms,etc).
Welcome to Govstack Building Block workgroup!
We’re so happy you joined the GovStack project! This document outlines the roles, goals and processes we’re using to work together for development of Govstack building blocks. It’s intended to help bring new team members up to speed more quickly.
BB group goals
The scope of involvement of building block groups in the overall Govstack initiative is important and critical to the overall goal of this project. At present the BB groups are involved in supporting at the least following parts of the GovStack:
1. Define Building Block Specification: Breakdown various use-cases into cross BB flows and map to BB microservices and assess feature readiness/gaps to fill in order to serve different country level programs. Develop BB service definitions with functional and technical specifications of respective building blocks, including interfaces between the BB and external applications, considering best practices, features and standards.
Develop BB prototypes: Define test cases for request-response exchange protocols and formats between BB APIs to qualify the BBs for functional compliance, system integration and interoperability with other Govstack compliant BBs and Apps along to manifest example use-case implementations.
Test and Integrate BBs: Work with external vendors and partners to explore/support for requirements to develop/adapt/integrate their building blocks and applications in compliance with GovStack BB specifications
Support Community engagement: Represent Govstack in open source communities to promote larger participation in promotion, development, adoption and training of GovStack approach to ICT transformation in e-Governance
Terminology Reference:
Important terms described and used in this document includes
· Government program: Package of services offered by a specific initiative of the government (e.g. mother and child care, rural agricultural advisory, insurance, education programs, etc.) defined with policies and processes to enable implementation involving different entities, facilities and staff to carryout various activities
· Use-Case: A portion of a government program that intersects with Govstack implementation that represents the process within a single thread of service delivery under the program.
· Use-Case Journey: The workflow within a use-case that may be carried out discretely to deliver a service
· Actors: People or machines that interact within a use-case journey in specific roles to deliver a service
· Logical process blueprint: breakdown of use-case journey into discrete logical steps (that may happen typically at different epochs) involving different actors
· Episode: One step of a logical process flow which groups all activities that can happen in same epoch
· “BB microservice”: the specific functionality of a BB that is consumed in order to carry out an activity in the logical process flow of a use case,
· Use-case Scenario: Specific context in which a BB micro service is utilized, identified with specific expected pre/post conditions, requests and response patterns
BB group activities, milestones, and deliverables
Building block groups carry out several activities towards the stated goals The development of Specifications of each building block involves several deliverables along specific milestones indicated below. These steps are elaborated further in this document
Milestones | Typical Activities | Deliverables |
Scoping | Identify use cases that may use services in domain addressed by this BB in its workflow. Define journey within these use-cases with episodic serial logical steps that groups activities that happen in parallel. | Logical process blueprint describing use-case journey with episodic steps |
Map activities within a logical process step (episode) and identify activities which are handled by specific BBs. Create list of BB microservices needed in each episode. | Logical process blueprint updated with list of BB microservices needed with priority(current/future scope/ needed but from other BBs) | |
Describe Scenarios of service calls | Define unique scenarios in which each microservice may be consumed with different inputs and outputs | BB Use-Case Scenarios defined for all prioritized BB microservices |
For each unique scenario, describe the preconditions, assumptions, actors, data inputs, normal and alternate internal workflows, outputs and post conditions, exceptions, dependency on other BBs (egress calls from this BB) expected to deliver the requested service. | Use-case scenario definition and link updated in relevant BB specification docs | |
Defining Key digital Functionalities (KDF) | Identify key functionalities needed within each BBs in order to provide a specific microservice. | KDF updated in relevant BB specification docs |
Define assumption and prerequisites to use each functionality | Functional requirement section of respective BB specification documents updated in BB definition document with relevant references to standard practices / features Enumerate candidate s/w products that might satisfy some of the functional requirements | |
Identify common internal functional blocks that may be required to orchestrate all KDS of the BB | ||
Layout internal functional blocks and workflow along with block diagrams, flowchart, etc. for each KDF | ||
Defining Data Structures | Identify data elements, groups and schema needed by functional blocks to orchestrate services of this BB in a MVP footprint | Resource model defined in BB specifications with references to relevant standards |
Cross cutting requirements | Describe security and architectural deviations if any needed in this BB that is in addition to or exclusion of those defined in Govstack Security and Architecture specification documents | Define deviations with justification and populate links to other compliant sections of BB security and Architecture guidelines |
Key decisions | Comprehensively Log any key decisions taken either internally or due to external feedback from reviews along with references and justifications | Updated key decisions log in bb specification |
Terminology | Identify and describe new terms if any, used in the BB specifications for this BB | Terminology document with links inserted in Glossary of BB specification |
TAC Review | Finalized BB definition template submitted and presented for review by Tech-advisory group review | BB Spec presentation PPTS, draft documents signed off by BB leads |
TAC feedbacks collection and assessment, discuss recommendations leading to corrections/pun into future or out of scope bins | Current specs updated with changes or posting for future scope / not in scope signed off by BB leads | |
Functional specification Publication | Clean up presentation for typos, consistency, clarity and reader friendly presentation. | BB specifications verified for accuracy and signed off by BB leads |
Specification with appropriate versioning details published on Govstack and other sites | BB documents hosted publicly and Govstack Website links updated accordingly | |
Open API Interface Definition | Define interaction with external applications/BBs specified in workflow during ingress and egress service calls of this building block | Interaction protocol flowcharts with associated data fields defined in English into BB definition document with references to relevant standards |
Open API specification of service calls and return formats for success/error/exception conditions | Swagger OPENAPI V3.0+ service APIs | |
BB API compliance Tests | Define Rest-Api according to service request-response protocols and data structure from BB specs | APIs in Open-API V3.0 Yaml/JSON format |
For each API write test requirements in English including all associated API calls (both ingress and egress requests) | Set of test cases with expected requests and responses for service calls landing on the BB and calls emanating from this BB to other BBs | |
Develop stubs with mock requestor-responder code or API adapter (if BB code exists) to cover set of test cases | Dockerized Stubs with mock requestor-responders / BB Adapters | |
Launch Mocks with Stubs and run test cases between requestor and responder stub endpoints | APIs qualified for compliance with BB specs | |
BB API Integration Tests | Add IM interface into Mock requestor/responder code | BB APIs mock/adapter codes upgraded with IM interface and protocols |
Integrate with information mediator | BB Apis adopted into IM (service discovery and access) | |
Run same test cases with IM in between the requestor and responder stubs | BB service integration qualified to work with same requests coming through IM | |
BB API Interoperability Tests | Replace requestor stubs with other IM integrated BBs (stubs/adaptors) | BB configured to access APIs of other BBs on Govstack as well as receive service requests through info med |
Run interoperability tests with other BBs that are integrated as consumers / service providers for the BB under test | BB APIs qualified to interoperate with APIs of other integrated BBs (both ingress and egress) | |
API Publication | Document qualified APIs and associated test case definitions as part of BB specifications | Published Open API interface specifications |
User journey integration | Integrate Qualified APIs into Sandbox | Sandbox updated with qualified BB APIs |
Develop cross-building block workflows at backend and application front end according to logical process flow | Application frontend (screen flow) and backend(workflow) developed according to logical process blueprint | |
Conduct tests of logical process flow | Test automation scripts and mock applications, test results | |
Create automated demo presentations | Hosting of qualified demos |
Building Block Specification:
Every use-case of Gov-Stack is defined as a journey of several steps along which services of one or more BBs are called by the main workflow. At the beginning of the specification a brief description of the main goals achieved by the building blocks, different application sectors in which it can be used, terminologies specific to the BB and clear indication of things that fall within the scope and out of scope of the building block’s functionalities are identified. A building block may provide multiple services to address the needs of several use-cases. Each use-case may consist of a journey of activities among one or more participant people and systems (actors).
Establishing Logical process blueprint:
To start deriving a BB specification, a given user journey is typically broken down into “logical process steps” where each step depends on completion of previous steps. For example, in a use-case of mother and infant care program, one may find the following logical process steps typically starting by promoting awareness to citizens, registration of eligible willing citizens, helping them to book, track and avail services such as doctor consultation, therapy and medicine such that the mother and health assistant get incentives on successful completion of stated activities in adherence with rules of the program. This use-case can be broken into a journey of several logical process steps accordingly as shown in diagram below:
The breakdown of this use-case journey is captured in the “logical process blueprint (LPG)”. Different LPGs may exist depending on the condition of the mother and child. BB groups can define different LPGs using the LPG template available “here”.
Mapping service calls to Building blocks:
Further, a single step within a LPG may involve several sub-steps. For example, the “Registration” step in mother and childcare LPG may involve the following sub-steps that map to different Building Block service calls as shown below:
Mapping BB microservice calls : Typically this breakdown is at a level where a sub step may request a specific BB to get the desired work done. Such BB micro-services return specific outcomes based on the input parameters. Micro Service Definitions: All microservices of the building blocks are defined as request-response pairs in a list as shown in an example subset below.
ID | Service call | Service returns |
1 | Create_session(sessionconfig[fieldnames, field values]) | Returns new session ID after creating new Session in worklist with given details in JSON array of [field name: field value]pairs |
2 | update_session(Session Id, sessionconfig[fieldnames, field values]) | Returns success code after changing details of given session ID in the worklist, as per given details in JSON array of [field name: field value]pairs |
3 | cancel_sessions(session_id) | Returns success code after deleting given Session in worklist |
4 | get_session_details (Session ID, sessiondetails [field names]) | Returns details as per JSON array of {fields} for a specific session ID |
Note that the building block may have both ingress calls (external sources calling services of the BB) and egress calls (BB calling services of other BBs to complete part of its internal workflow). Depending on the inputs of a use-case in which a microservice is called, the BB may respond with different outcomes from same service. A BB’s behaviour in response to different scenarios are defined in multiple scenario templates. The template will detail the context (preconditions, assumptions, inputs) and expected functional outcomes from the BB microservice using the template here – BB use-case scenario definition template, An example use-case template for scheduler building block is produced below:
Example Use-case scenario definition Post-Partum Care Program (Apointment booking)
ID | PPCP_Scheduler_UC_01 |
Logical process step | Consultation Appointment: Post-Partum Care Programhttps://github.com/GovStackWorkingGroup/BuildingBlockAPI/projects/1 |
Scenario | Health worker assisted appointment management for consultation |
Description
| Brief description of the scenario in which this service of the building block is consumed by another BB or Application, by giving specific inputs and return of specific outputs. Note the same service could have multiple modalities depending on infra maturity, each becomes a separate scenario, although service may be same. For example payments through mobile money or through banks create two alternative scenarios for the payment BB. |
Trigger | PPCP App sends request to scheduler BB to create/cancel consultation event. |
Preconditions | Entities that provide consultation services are registered under the PPCP program …. |
Data inputs | Query parameters to find free slots 2… |
Actor | mother, Doctor, Health Worker |
Normal Course | Appointment booking: Organizer(Health worker) captures subscriber’s (Patient) preferences (such as doctor, dates, time-zones, entities, etc.) in PPCP App and submits appointment request for a consultation session ….. Appointment Cancellation: Organizer(Health worker) submits appointment ID and request for cancellation of booked session through PPCP_app to Scheduler[planner]…. |
Alternative Course (Scenario 2) | In an alternative scenario, patients may themselves login to the PPCP_App application and book appointments without support of Health worker. This is a subset of the workflow mentioned above. |
Data output | List of available free slots Event related alert Messages…….. |
Post-Conditions | Appointment booking: Specific session of event plan is updated with either completed or no show status Reporting: Relevant datasets are returned as JSON formatted string….. |
Exceptions | PPCP_App prompts Organizer(Health worker) to request new appointment if confirmation is not received from Scheduler within a predefined timeout period ….. |
Related BBs | Service ordering, Registration, workflow, Information mediator, Messaging, PPCP App |
Remarks | This use case model can be used for other types of consultations such as legal/financial/etc. between subscribers (plaintiff/investor/etc.) and consultants (lawyer/auditor/etc), organized by respective assistants. |
Workflows | Link internal workflow diagrams here for various micro services used: e.g booking/cancellation/tracking and alerting/status reporting of appointments |
Deriving key digital functionalities: By inspection of micro service definitions from multiple use-case scenarios, a set of common key digital functionalities are identified. These functionalities are organized into internal function blocks with suitable control and dataflows so as to orchestrate all define micro services appropriately. Each functional block will have to be described in the building block specification and associated interactions with other internal or external blocks be illustrated in suitable workflow diagrams ( Websequence diagram is a tool commonly used currently for drawing these flows and http://draw.io is used for block diagrams).
Example functional blocks given below:
Example illustration of microservice workflow given below:
Data modelling: Data elements needed by the functional blocks are identified and grouped into structured lists and resourced models on an MVP footprint. An example is given below:
Resource Class | Element | Description |
Worklist | ID | Unique ID of a specific session |
Start_Date | Starting date of the session | |
Start_Time | ending date of the session | |
…… | ….. |
Log | Log_ID | Unique identifier of a specific transaction |
Category | Unique identifier of a specific type of transaction (Based on what change did the administrator affect) | |
…… | Details of transaction done and who did it | |
BB Config | Block_ID | Name of internal block which is to be configured |
Config_data | Latest Configuration settings of various parameters in this block given in JSON format | |
………. |
| |
Alert List | ID | Unique identifier of a specific alert message bound to a session |
Category | Type of alert message (appointment alert/ process trigger/ healthalert/crimealert/etc.) | |
…... |
| |
Resources | ID | Unique identifier of a specific Resource |
Phone | Resource phone number | |
…… |
| |
Subscribers | ID | Unique identifier of a specific subscriber |
Phone | Subscriber phone number | |
Location | Address of subscriber | |
… | ……. | ……. |
API interface definition:
All service transactions with a BB will happen through specific protocol interfaces defined as per OPENAPI V3.0+ Resfult API definition standards. These interfaces are defined to indicate hosting requirements, API calling input formats, expected output formats and dependencies if any for all ingress calls. The egress calls are expected to comply with the appropriate target BB’s APIs. The definitions may be drafted with any editor, swagger tools are most common.
An example API definition snippet is shown below:
openapi: 3.0.1
info:
title: Govstack Scheduler BB API
description: Interfaces to services rendered by Scheduler building block
termsOfService: TBD
contact:
email: psramkumar2@gmail.com
license:
name: Apache 2.0
version: 1.0.0
externalDocs:
description: Find out more about scheduler
servers: [
{
"url":
"description": "Scheduler BB Security server"
}
]
tags:
name: session
description: session API
name: find
description: find API
name: alert
description: alert API
name: resources
description: resources API
name: subscribers
description: subscribers API
name: status
description: status API
name: get_log_report
description: log report
paths:
# session
/session/create_session/{requestor_Entity}/{requestor_Role}/:
post:
tags:
session
summary: create a new session
operationId: createSession
parameters:
name: requestor_Entity
in: path
required: true
schema:
type: string
name: requestor_Role
in: path
required: true
schema:
type: string
requestBody:
content:
'json':
schema:
$ref: '#/components/schemas/sessionconfig'
responses:
200:
description: success
content:
application/json:
schema:
type: string
example: "4524458935"
description: ID of created session
'405':
description: Invalid input
'403':
description: requestor doesnot have permission to create a new session
default:
description: Unexpected error
Cross cutting requirements:
The BB group can, after studying the Govstack architecture and security guidelines documents, can identify any additions or deviations from the recommended security and architectural applicable to this BB. Such deviations shall be noted with due reference to relevant sections o f architecture and security guidelines documents and justification for such deviations / additions / deletions if any.
Other components of specification documentation: Taking notes of other important elements of the specifications, the BB groups will enter key decisions taken with justifications, include key references to other published standards and best practices and describe or link to description of terminology used in the bb specifications. The specifications will also make additional recommendations as various modalities incorporated inside the BB to achieve the same services with different level of infrastructural maturity of target demography. For example, Govstack payments building block clarifies requirements of alternate modalities for making payments, based on paper vouchers to institutional memberships, mobile money transfer to banking transactions.
Building Block API Testing
The APIs define the software interface on the surface of a building block. Associated with each API there are service request formats with a set of input parameters that trigger specific actions in a building block and a set of response parameters depending on how the building block processed the inputs to produce specific outputs. Infact the unit testing phase involves on the BB without any integration. Each test is design to check the API compliance with a specific service functionality such that given a specific inputs the response carries correct results from the BB. Initially instead of a real life BB implementation, mock requestor and responder applications are built to emit appropriate data stubs for an api call and response. When run, the mocks emulate the BB’s expected behaviours and should hence provide exactly the same expected functional behaviour at its outputs as per specification. This qualifies the BB API is functionally compliant and syntactically correct with respect to the specification. If a functionally API compatible product replaces the mock responder, the requestor should get the same responses from that product as well, to qualify the target product is Govstack API compliant.
Task | Output |
Identify focused resource to work on the API specs and API testing | BB lead to appoint a focused person/ nodal point for the API spec and testing effort. |
Complete the OpenAPI specifications for each BB (openapi.json) | Swagger version of the API specs |
Take a subset of the APIs for testing | BB lead to help in identifying the most important APIs that need to be tested first. |
Develop Gherkin features/ scenarios specification file for the identified subset of APIs | Gherkin file with features, scenarios for testing APIs |
Testing API endpoints using Cucumber scripts | Implementation of runner class for Gherkin spec and Cucumber scripts to test candidate APIs |
Develop mock example for testing (this is the step where we test the API stub) | Build an application that takes an openAPI spec as an INPUT and provides a REST API as an OUTPUT. This should be based on the mock application template developed by Taylor Downs and Benjamin Bach. (Paste the |
Add candidate mocks as examples for testing the APIs. | Examples of candidate products meeting the Govstack BB specs |
Deploy the example application(s) | Docker or Docker compose scripts to containerize and deploy the example/ mock example application. |
Deployment, and API testing. Add .circleci/config.yml to BB repo. Generate test reports. | .circleCI config file to automate process of deploying environment and testing the APIs for compatibility with GovStack specs. Generate test reports. |
Building Block Integration testing:
Once an API is qualified as functional compatible to specifications and technical compatible t the interface definitions, the associated mock requestor and responder BB emulator applications can be registered as services into an information mediator.
Doing this will expose the API as a microservice discoverable and securely accessible from across the internet. In a typical situation, the requestor would be an application or a building block connecting to the internet through their own Information mediator as illustrated below:
The information mediator has specific protocol to register a new building block service provider BB into the system, publish such services to external consumers for discovery and access request, and subscribing of requestor BBs/apps to access the services through IM on demand. The requester and responder mocks/BBs will be following these steps to become available for service exchange through IM at specific endpoints through the internet. Once these are ensured, the requestor and responder are ready to run the same tests that were run in direct connection earlier, except that it will now run across request-response sequences through information mediator and internet and produce expected results. This test ensures compliance with Secure access of the BB services through internet from other apps/bbs. This step will help build up a stack of BB APIs that are guaranteed to be functionally compliant and accessible through internet.
Interoperability tests:
The next step would be to check if building block apis can be called from other building blocks (mocks or actual BBs). This assumes that both provider and requestor BBs are registered with Information mediator and are replacing the mocks with Govstack API spec compliant APIs or Adapters that transform native interfaces to Govstack compliant API requests and responses. For example, a third-party registration building block can work with another third party payment building block, as long as they are both Govstack compliant either directly or through respective adapters.
The BBs at this stage are guaranteed to send and received correct data expected in a service request and response stream between actual building blocks either directly or through respective API adapters Interoperability.
The first step (API testing) involves only the BB under test, the second step (Integration testing) involves the tested BB and IM BB, the third step (interoperability testing) involves testing the BB under test and is client BB connected through the IM and the fourth step (handshake testing) involves complete logical handshakes connected with service calls from request to completion of service.
Handshake Testing :
Beyond the API interfaces, there are handshake protocols necessary between a caller and called party. For example upon requesting a service the requester may expect confirmation if the provider has accepted the request. The requestor may further query to know if the service is completed and obtain the results. The requestor may wait till service is completed or may proceed to other work expecting the servicer provider BB to proactively inform the requestor when service is completed. There are some common underlying assumption that APIs are designed to carry out work asynchronously by default and may only by exception work in synchronous calls with due justification.
A synchronous call waits for a response while an asynchronous call does not. In Govstack architecture Async calls to any BB service are also expected to get back a synchronous token number in return to indicate that the request has been received or rejected, whereas the results of the gets pushed later along with token number to the requesting BB through a call back function exposed by the requestor BB. Alternately the requestor BB may query status giving the token number and get results if ready from the responding BB.
A third mechanism adopts the pub-sub model wherein, the service provider BB publishes results in its pubsub room and if requestor BB is a subscriber to that room, the pubsub will automatically push the message to the requester BB’s exposed status update endpoint.
All these mechanisms are useful in difference scenarios, and are to be tested for inter-BB communication. These mechanisms can be tested to make sure that beyond syntax and formats, the logical handshakes are performed properly.
The BB APIs that have passed all above tests are stacked as qualified for consumption in manifesting and demonstration of different use-case. The sandbox team can at this stage pick well defined usecases and develop demos using qualified BB APIs