Digital Registries (FGA)

Differences between Registries Platform and specification:

Main actors' responsibilities and possibilities

Registry platform supports only DB, Digital Registries supports multiple

One root entity in Digital registries for one database. Registry platform allows multiple root entities

Registry Platform supports only relational database to communicate with

Registry platform have almost the same functionality on a RestAPI level but a bit different approach how to map logical methods onto http rest api methods and organization of rest API in general (POST methods for data read)

Name

Fit Gap Comments

Administrator

1.Create a new register/database (API or Web user interface);

Implemented

2.Create and configure schema of the register(API or Web user interface);

Implemented

3.Change schema configuration and publish new version of the database and API services (API or Web user interface);

Not the same approach

Changes that lead to the table or column removal are not allowed.
Schema versioning approach implemented, previous schemas stored in separate PostgreSQL schema (archive schema)

4.Enter data to the register (API or Web user interface);

Implemented

5.View data records in the register (API or Web user interface);

Could be implemented

6.Update data in the register (API or Web user interface);

Could be implemented

7.Import/export data from/to external files;

Implemented

8.Import/export registry database schema;

Implemented

9.Create API services;

Implemented

10.View statistics(API or Web user interface);

Implemented

11.Inspect transaction log of registry data operations (API or Web user interface).

Implemented

12.Manage access to registry data. Authorize users to see and edit registry records or data field (ABAC based access management).

Could be implemented

13.Share data with other users via e-mail, or via unique and secure URL. sharing can be field level or record level.

Could be implemented

Applicant

1.Search data from the register (API service);

Implemented

2.Read/pull data from the register (API service);

Implemented

3.Create data in the register(API service);

Implemented

4.Update data in the register(API service);

Implemented

5.Delete data in the register (API service);

Implemented

6.Validate if given content exists in specified register (API service);

Implemented

7.Read statistics (API service).

Could be implemented

Citizen-Centric (2.1 in Architecture Blueprint)

Could be implemented

Open (2.2 in Architecture Blueprint )

Implemented

Robust ( 2.7 in Architecture Blueprint)

Implemented

Databases MUST not include business logic (3.21 in Architecture Blueprint)

Implemented

Privacy and protection of user data

Could be implemented

Name

Fit Gap Comments

6.1 User Story 1 - Registry Schema User Interface

DRS-1

Analysts must have the option to create a new registry database by filling the following information:
Name of the database;
A short name;
Schema of the database (see DRS-3 ).

Must have
UC1

Implemented

Implemented (not part of RR and not over API)

DRS-2

Analysts can create multiple databases into one system instance.
Databases must be linkable with foreign keys. See foreign key API description example in Appendix 2. Analysts can configure which databases and which fields are linked in the user interface.
In this document and foreign key function we consider databases as database tables that can be linked with one another. See example illustration here.
User story: As a user I can browse database content (Data) in the UI and when databases are linked, then I can click and move from one database to another where the corresponding linked data will open in the UI.
In the Digital Registries Data user interface, it must be possible to open another database by clicking on the record ID in one database and all corresponding records from the other Database will open.
It is required to have at least two levels of ID’s (database ID and field ID) to link the databases. See example API below in Appendix 2.
Example: In one registry database we store information about Mother and Child record. In the second registry database we store information about payments made for the mother. System must enable a foreign key link between the payment database to the Mother and child record database. Users can click in the payment database record UI to the Mother ID field and the system UI must open the corresponding record in the Mother and Child database.

Must have

Implemented

Implemented

DRS-3

Analysts must have the option to add fields to the database schema. Fields of the database must contain at least the following elements:
1.Field name;
2.Field type, at least with following types:
1.Text;
2.Number;
3.Boolean;
4.Date/time;
5.Date;
6.Time;
7.File (pdf, doc etc). File extensions/types must be configurable;
8.List/Array/Edit grid (sub-table/array of values inside a field);
9.Block container (optional, to group fields visually);
10.List of Values/Catalog (holding value and key).

1. Field properties (see more DRS-33)

Must have
UC1

Could be implemented

1 and 2 implemented.
3 could be implemented.

DRS-4

Analysts must have the option to publish the database. Publishing will reveal the database to users.
Publish uses versioning. Every publish creates a new version of the database schema and API services;
Old database schemas must be available to the users;
Data stored in the old database versions must be usable in old versions and in new versions;
Analysts can delete database schema versions. Same version API services must be deleted at the same time.

Must have

Implemented

Administrators can publish a new version of the database schema. Migration of data should be managed by a migrations script or changes should be compatible by design.
Database can be treated as Registry instance

DRS-5

Analysts must be able to configure the API services per registry database.
The system automatically creates API services to:
create data;
read data;
update data;
delete data;
validate data (if exists);
update or create data.
Analysts can hide API services;
Analysts can delete API services;
Analysts can copy API services;
Analysts can create custom API services;
The system generates the API data structure from the dynamic database structure automatically each time a publish is done.

Must have

Implemented

API can be hidden or deleted
(custom operation for read only)

DRS-6

Authorization to:
create and manage databases;
API usage per service, per record, per data field;
access to DATA.
Analyst must have the option to manage user rights of a database and data via API and via UI.
Attribute Based Access Control (ABAC) logic could be used (API, Schema, data fields, record filter, users);
Anonymous user role must be available;
Any logged in user role must be available;
Per user, per group of users option must be available.
Group is a set of users in a role.
Role is a set of rights.
Authorization/user rights are stored in addition in the central GovStack IAM system- read more here.

Must have

Implemented

Partially implemented

DRS-7

The system must log all data processing in the database.
Schema changes must be logged;
Data processing (CRUD) must be logged;
Logs must be visible and searchable to the Analyst via UI;
Every data owner (e.g. physical person) must have the option to see who has processed his/her data (PersonalData). The function is standard function for all registries. See more (DRS-14 API example)
Change logs are protected with highest level of integrity (chaining of logs)
Database logs could be logged with external blockchain(Nice to have).
See more Audit Logging requirements here.

Must have
UC3

Could be implemented

Schema changes stored in git.
Technical log and audit - implemented.
Personal Data - not yet implemented.

DRS-8

Personal Data usage.
System must automatically store all data read requests and store these in log table.
Covers data read events via UI and via APIs.
Personal Data logs are stored with PersonalData data tag, storing at least the following information.
Log ID;
Data record ID;
Field ID;
PersonalDataID(unique and unchangeable identifier of a person);
Reader ID- who read the data;
Reader name- name or initial of a person;
When- the moment when the Personal Data was read.
Personal Data report is visible only for Analyst to see all data read logs and Data Owners (physical person) to see their own personal data usage log. Input is PersonalDataID field.
PersonalData report is usable as API service (read)
System must have API for PersonalData reports. API is per registry(database)
System must log Personal Data log read events to the log table.

Must have
UC3

Could be implemented

Planned to be implemented

DRS-9

Analysts must be able to create views of a database.
View is a selection of data from a database;
View can be opened as OPEN DATA (anonymous user);
View can be created and it can be as a base for an API service (Custom API);
View is not for changing or deleting data, only for reading;
View rights are managed by the user rights management system.

Optional

Implemented

Implemented without an anonymous user.

DRS-10

Must have the option export/import database schema to JSON file, (optional: xls file).

Must have

Implemented

Implemented in XML

DRS-11

Import DB structure
Must have the option to import database structure from JSON file
Must have the option to import database structure from XLS file

Optional

Implemented

Schema first approach is used, so schema defined in XML format from the beginning (liquibase xml)

DRS-12

Service usage statistics
System must record all API service usage information.
System must record all searches made in the Registry UI and via APIs.

Optional

Implemented

Grafana statistic and audit

DRS-13

Analyst must be able to mark a field as PersonalData log object- This field contains personal data.

Optional
UC3

Could be implemented

Planed

DRS-14

Analyst must be able to mark a field as PersonalDataID. This is the data owner’s ID.

Optional
UC3

Could be implemented

Planed

DRS-15

Analyst must be able to mark a field as secret- This field contains secret data (credit card number). E.g. secret data (card data) must be encrypted while at rest.
Information in transit between the BB-s is secured with encryption. Information in Transit is described and governed by Information Mediator BB.

Must have

Could be implemented

Planed

DRS-16

Analyst must have the option to read database schema in the web UI.

Must have

Implemented

Partially implemented in admin portal, the active phase of development at the moment

DRS-33

Analyst has capabilities to configure database field properties:

1. API related field properties:

Validation options- required, unique, max, min.
blinded/ encrypted (DRS-15, DRS-18);

1. UI related field properties:

field mask, format,
read-only,
personal data,
enum list selection;
blinded/ encrypted (DRS-18);
multiple value/array. User can add more values (e.g. multi select from catalog list) to the same field. Multiple values is array type field;
validation options- Required, Unique, max, min.
Foreign keys (to link other databases in the same ecosystem). See example schema in Appendix 2;
Triggers to automate field content related actions:
create ID-s,
merge fields,
add prefix,
suffix,
conditional logic,
trigger will be activated if certain condition(s) are true,
transform-upper/lower case/ java script);
Triggers are automated when record is created/changed. Trigger is record level automation.

Implemented

API validation options: could be configured using liquibase configuration

API related fields
encryption will be implemented

UI related field properties: partially Implemented there is no Field advanced properties and Triggers

DRS-34

Analyst must have capabilities to add encryption key per database.
Encryption key is used to encrypt and decrypt data (DRS-17).
Encryption key can be used by applications to read encrypted data. Each database has a unique encryption key defined by the analyst.
Encryption key is blinded in the UI.
If applications want to read encrypted data via API they must know the encryption key. Data is decrypted in the user interface.

Must have

Implemented

Database data encrypted at transit using SSL

Database data is encrypted at rest using storage capabilities utilized by RDMS (so data encrypted at rest, but this is not the responsibility of RDMS itself)

Read encrypted data via API is not restricted by encryption key

DRS-35

Analyst must have capabilities to automate data exchange between databases internally and externally via API.
Automation is triggered automatically after a pre configured time interval as a loop (finishes when all corresponding records have been processed).
Automation processes one record at a time.
Automation has configurable conditions (business rules in Rules Engine). E.g. IF field A = 123 then true. Conditions can be grouped with AND and OR operators.
Automation is configured by mapping (input, output) registry data fields to:
another database in the same instance.
API in an external database.
Mapping involves:
query part (input)
answer part (output)
Mapping can be done from many to one and one to many. Mapping may have transformation option to convert data to another format. E.g. est->EST; Expected outcome: the Automation can be activated automatically when certain conditions are true and the system sends data to another database or to an external API.

Must have

Implemented

We need to run several instances of registry instead of having several databases for single instance of registry

DRS-36

Analyst may have capabilities to use database schema templates so that the registry creation is faster.
Schema templates can be shared in the same instance (internal marketplace).
Schema templates can be shared in a marketplace.
Schema templates can be imported and exported.

May have

Could be implemented

We have a single template defined for the “schema first” database definition. So the generic schema template mechanism is in place, it could be extended to have several templates defined and used to build a database schema definition.

6.2 User Story 2 - CRUD Data in User Interface

DRS-17

Analysts must have a view to see all data in the registry.
Two main views:
Main registry records grid view.
Record detail view.
See data;
See documents(open if image, download if other type);
Data log view (changes (create, update, delete). Data before and after).
Data read view (information about who has looked the data, /exported data). Data and data readers information is stored in the log registry.

Must have
UC2
UC3

Could be implemented

The data accessible over API
No explicit separate UI view exists for a moment especially for this case

DRS-18

Analysts must have a view to edit data in the registry. Two main views:
Main grid (inline editing);
Detail record edit view.
Edit data
Remove/add documents (upload)
All data changes are logged.
Analyst must have option to delete data in the registry.
All data changes are logged.

Must have

Implemented

Can be implemented by changing configuration
Accessible over API

DRS-19

Analysts can use additional functions to simplify data searching:
Filtering by search criteria by field content
Full text data search
Order by each data field

Must have
UC2

Implemented

Accessible over API

DRS-20

Import data to the registry
Analyst must have an option to import information to the database. Import formats are: json, csv, xls.

Must have

Implemented

An initial data load mechanism already in place. CSV data format is used currently. Support for additional file formats could be added.

DRS-21

Export data from the registry
Analyst must have an option to export selected/filtered data from a registry to CSV, XLS, Json.

Must have
UC2

Implemented

Data accessible via RestAPI. Searching/filtering of data supported. Exact exporting of data to a specific format could be implemented.

Redash reports could be used as an alternative to get the data.

DRS-22

Statistical queries.
The system should have the ability to:
1.Produce standard statistical reports
1.System must show statistics of all registered items in the registry, with various criteria for filtering. For example:
1.Details of registered people
2.Details of registered services
3.Time series: Change in registration of people/services over time
4.Details of change to data elements (audit logs)
2.Generate customizable reports based on the fields registered in the registry.
2.Allow the analyst/user to analyze data collected in the system in various ways:
1.(Option) Develop functionality to allow custom dashboards for analysts to analyze data within databases
2.Provide APIs for extracting data from databases to analyze in external data analytics systems (eg. Tableau)

Must have

Could be implemented

Grafana dashboard and time series data could be used to build proper statistic reports
Log files could be used for building a report

DRS-33

Users can share data with other users.
Share data with other users via e-mail, or via unique and secure URL. Sharing must be record level and field level.
Data sharing can be turned off in authorization module.
Data can be shared to anonymous users. The data shared to anonymous users is Open Data.

May have

Not the same approach

There are no sharing capabilities of the data in the database itself.

The exported data to a specific file or report could be shared potentially and controlled by those shared files in the future (needs to be implemented)

6.3 User Story 3 - CRUD Data APIs

DRS-23

BB must enable client systems to process (CRUD) the database records via Open API services.
Applicant can search data;
Applicant can create data;
Applicant can read data;
Applicant can update data;
Applicant can delete data
Applicant can create or update data
BB authorizes client systems and users to process data.

Must have
UC1;UC2

Implemented

Implemented

DRS-24

BB must have an Open API service list (Swagger) to visualize all API services and API service versions.
Client systems must be able to see all API service descriptions including:
Description of each field
Example data of each field.
If possible then the example must be real so that whoever is looking at the API specifications can test the example data in the service (try it).

Must have

Implemented

Implemented but not used by Applicant

DRS-25

System must have an API for PersonalData usage report.
API input must be configurable by the analyst. Input must be a unique identifier of the data owner(e.g. personal identification number).
If registry database schema is designed to store personal data then the analyst must be able to link the personal data to the owner of personal data (e.g. citizen).

Must have
UC3

Could be implemented

Planned

DRS-26

Statistical queries via API.
System should make data accessible through the API
Registration Data
Programme Data
API should allow querying data with multiple parameters
Date, time ranges
Registered Program
Only authorized data should be available through the API.

May have

Implemented

Implemented

DRS-27

Using viewing event logs- every data owner has the right to see who has looked at their personal data.
Data owner is a physical person whose personal data is stored in the registry.
Data owner has the right to access data reading/processing event logs of the personal data they own. Personal data in a registry is marked accordingly (PersonalData) by the analyst.
PersonalData logs are visible via API or via user interface (PersonalData report).

Must have
UC3

Could be implemented

Planned

6.4 User Story 4- Registry Schema API

DRS-28

Developer must have the option to create a new registry database by sending data via API:
Name of the database;
A short name;
Schema of the database (see DRS-3 ).
See full list of example API descriptions /database/modify here.

Must have

Implemented

Developers creating the database schema definition using admin portal UI
As there is no possibility to create several databases within a single registry, there is no need to define the database name.

DRS-29

Developer can create multiple registry databases into one system instance.

Must have

Implemented

No need to create several databases in a single registry instance. Multiple registry instances could be up an running instead.

DRS-30

Developer must have the option to publish the database. Publishing will reveal the database to users.

Must have

Implemented

Implemented. The registry publication pipeline and admin portal are responsible for integrating and publishing new database definitions.

DRS-31

Developer must be able to modify API services per registry database.
The system generates the API data structure from the dynamic database structure automatically each time a publish is done.
The system automatically creates API services to:
create data;
read data;
update data;
delete data;
validate data (if exists);
update or create data.
Developer can hide API services;
Developer can delete API services;
Developer can copy API services;
Developer can create custom API services.

Must have

Implemented

Implemented for the single database

DRS-32

Developer must have the option to read database schema via API.
Developer must have the option to read the list API services available per Database.

Implemented

Developers can read database schema details via admin portal RestAPI.
Developers could use OpenAPI document to get the list of API services available for managing database data.

7.1 Standards/Protocols

1.All dates should follow ISO 8601.

Implemented

2.RFC 7159 - The JavaScript Object Notation (JSON)

Implemented

3.Open -API Version 3.1.0

Implemented

7.3 Data Elements

Could be implemented

Name

FitGap Comments

8.1 DATA CREATE

Implemented

8.2 DATA UPDATE

Implemented

Implemented and partial update

8.3 DATA UPDATE-OR-CREATE

Implemented

8.4 DATA UPDATE-ENTRIES

Could be implemented

Batch update not implemented, could be implemented if required

8.5 DATA LIST (Search)

Could be implemented

Not by regex, search possible by sql operators (like, eq, gt, lt, in, etc)

8.6 DATA READ

Implemented

8.7 DATA READ-VALUE

Implemented

Can be implemented by SC (returned false )

8.8 DATA EXISTS

Implemented

Can be implemented by SC

8.9 DATA DELETE

Implemented

8.10 DATA My personal data usage

Could be implemented

8.11 DATABASE SCHEMA READ

Implemented

8.12 DATABASE SCHEMA MODIFY

Could be implemented

Will be available via admin-portal (currently in an active phase of development),
so separate RestAPI endpoints, which expose operations for modifying database definition (currently only modification of liquibase changelog definitions).

8.13 DATABASE SCHEMA DELETE

Implemented

No need to create several databases for single registry instance. Multiple registry instances could be running instead.

8.14 DATABASES LIST

Implemented

No need to create several databases for single registry instance. Multiple registry instances could be running instead.

8.15 DATA CREATE-ENTRIES

Could be implemented

Batch create not implemented but could be implemented if required