A privacy-preserving system for storing and matching de-identified Personal Identifiable Information (PII) records.
Piipan is a reference model for privacy-preserving systems that store and match de-identified Personal Identifiable Information (PII) records across multiple tenants. This allows finding matches between different data sources without revealing the individuals' identities.
Under this model:
- Tenant systems share de-identified participant data to Piipan daily
- Duplicate participation is prevented by using Piipan to search for matches
Paramount quality attributes of this system include:
- Preserving the privacy of program participants
- Accuracy of matches
- Adaptability to multiple programs
To achieve this, Piipan incorporates a Privacy-Preserving Record Linkage (PPRL) technique to de-identify the PII of program participants. Please see our high-level treatment and our technical specification for more details.
High-level architecture, process, and (sub)system documentation, as well as Architectural Decision Records (ADRs), are organized in this index.
Piipan is implemented with .NET and Microsoft Azure, using a Platform as a Service (PaaS) and Function as a Service (FaaS) approach.
Piipan is organized into subsystems. A subsystem can contain one or more .NET projects. Each subsystem has a top-level build.bash script that can run builds, unit tests, and deployments for all projects in its subsystem.
Piipan also has a top-level build.bash script that can perform these operations for all subsystems.
Install Piipan's list of prerequisites before starting development.
Once the prerequisites are installed in a development environment, you can locally build all projects in a particular subsystem by navigating to it's top-level directory and executing build.bash.
Example:
$ cd match
$ ./build.bashTo build all subsystems at once, navigate to Piipan's top-level directory and execute ./build.bash
For a more detailed analysis on testing best practices in Piipan, review the following guide:
Run all unit tests for a particular subsystem by navigating to it's top-level directory and executing build.bash test.
Example:
$ cd match
$ ./build.bash test
When testing, an optional flag [-c] can be passed to run in Continuous Integration mode:
Example:
$ ./build.bash test -c
To run all subsystem unit tests at once, navigate to Piipan's top-level directory and run ./build.bash test.
A subsystem may have one or more integration test suites. Because they require containerized environments, each integration test suite must be run individually. Consult each subsystem documentation for integration testing instructions.
Deploy all projects for a particular subsystem by navigating to it's top-level directory and executing build.bash deploy -e [env].
Example:
$ cd match
$ ./build.bash deploy -e tts/dev
To deploy all Piipan subsystems, navigate to Piipan's top-level directory and execute build.bash deploy -e [env].
These scripts rely on a top-level solutions (sln) file for each subsystem. See tools/build-common.bash for more details.
As an alternative for deploying with the bash scripts, you may choose to use Visual Studio's publish feature. This allows you to publish with just a few clicks without ever having to leave Visual Studio. Before you are able to complete these steps, you must:
- Set up VS to utilize Azure Gov Cloud (https://learn.microsoft.com/en-us/azure/azure-government/documentation-government-connect-vs)
- In Visual Studio, go to File-> Account Settings and add your account
- In Visual Studio, go to Tools -> Options -> Azure Service Authentication and chose your account you just added
To do so, right click the project you want to publish and select Publish. On the publish window, add a New publish profile by clicking the "+ New" button. On this screen, connect to your Azure resource that you would like to publish to. You may need to re-enter your Azure credentials on this screen. If you do, you will need to enter them, close out of the publish dialog, and start the process over.
A few additional things to note when publishing web applications:
- Make sure to publish the main application, not the Client one. For example, publish Piipan.QueryTool, not Piipan.QueryTool.Client.
- You will need the correct appsettings.*.json file for your environment. Please contact a team member and they will show you where the files can be found. Do NOT check this file into GitHub!
- If you get errors in the Javascript console because of "Integrity" issues, it's possible your deployment files got some bad files in it. Usually this occurs when upgrading to a newer version of a NPM package and still having a copy of the old one in your bin and/or obj folders. To fix this, delete your bin and obj folders in both the main project and the Client project and try again.
For more information for developers, see Piipan's architecture and implementation notes, our team practices, and our other technical documentation.
This project is in the worldwide public domain. As stated in CONTRIBUTING:
This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.
All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.