PrivAPI is a Python package that allows the classification of sensitive data flows within REST API communication using Deep Neural Networks (DNN). It relies on Google's Keras and TensorFlow.

As explained in PrivAPI - Detecting Personal Data within API Communication Using Deep Learning

---

____       _        _    ____ ___

 _ _ __(_)_ __/ | _ _ _|
'__| / / _ | |
 __/| | | |

_/_/ __|

---

Quickstart

Requirements

Make sure you're running Python 3.5 or newer.

Setup environment

It's strongly recommended to build your own environment so that your local python packages don't get in the way. Create a Virtualenv environment within the venv project folder.

python3 -m venv venv

Then set the PYTHONPATH and activate the environment so the Python scripts can be found:

export PYTHONPATH=.
source venv/bin/activate

Install dependencies

pip install -r requirements.txt

Create a Neural Network Model

In order to detect sensitive data flows the system has to learn from confidential and non-confidential REST API request payloads. The project ships with a pre-generated dataset that can be used to train an LSTM neural network. If you want to generate your own, please refer to the next section.

python privapi/train.py

This will generate both the model and token directionary files living within the out folder. It's strongly recommended to use a GPU box in order to speed up this process, as the model requires 100 epochs to converge.

Detect sensitive dataflows

Once a model has been trained, it's time to run predictions based on it. There are two examples: one that is sensitive (positive class) and one that is not (negative class). Both live within the predict folder.

python privapi/predict.py

This command will output predictions to the predictions.csv file within the project root.

	payload_file	is_sensitive	probability
0	magento-payload.json	1	0.9999695
1	slack-payload.json	0	0.40711942

As you can see, Magento's sensitive request payload has been classified as confidential with 99% confidence. The non-confidential Slack request payload was classified correctly, even if it contained a first and last name.

Drop the request payloads you wish to classify onto the predict folder and re-run the predict.py script. Any file having the .json extension will be picked up.

Enjoy!

Generate

For generating your own training dataset use the following command :

python privapi/generate.py

By default, the dataset will be saved as training.csv within the data folder.

In order to obtain relevant metrics of the generated dataset use :

python privapi/analyze.py

Configuration

What will heavily determine the accuracy of the predictions is the quality of the training dataset. In addition, to generate sound request payload examples we need to make sure that the associated label - whether sensitive or not - is correct.

In order to label an example, the generator will look in the config.py descriptor to determine whether there's a matching entry for a given OpenAPI operation parameter name that matches the name_type_to_gen dictionary. If there is, it will use the associated generator and label the example as positive (i.e. having PII).

Here's an example configuration file. Feel free to add your own custom entries in order to consider additional PII fields.

from privapi.fakers import (
    _full_name_, _date_, _id_, _key_, _company_business_id_, _company_, _bank_account_, _first_name_, _last_name_,
    _address_, _bban_, _city_, _country_, _country_code_, _ssn_, _email_, _phone_number_, _gender_,
    _building_number_, _iban_, _postal_code_, _state_, _street_, _province_, _amount_, _credit_score_,
    _credit_card_number_, _alphanumeric_, _location_, _latitude_, _longitude_, _timestamp_, _latitude_str_,
    _longitude_str_, _timestamp_str_, _amount_str_, _credit_score_str_)

name_type_to_gen = {'string':
                        {'[uU]ser': _full_name_,
                         '[fF]ullName': _full_name_,
                         'firstname': _first_name_,
                         'lastname': _last_name_,
                         '[aA]ddress': _address_,
                         '[nN]ationality': _country_,
                         '[dD]ate': _date_,
                         '[tT]axId': _company_business_id_,
                         '[sS]erial': _id_,
                         '[oO]rganization': _company_,
                         '[cC]ompany': _company_,
                         '[dD]ba': _company_,
                         '[dD]oingBusinessAs': _company_,
                         '[bB]usinessName': _company_,
                         '[aA]ccount': _bank_account_,
                         '[uU]UID': _id_,
                         '[sS]hareholder': _full_name_,
                         '[pP]ostalCode': _postal_code_,
                         '[zZ]ip': _postal_code_,
                         '[bB]ic': _bban_,
                         '[bB]ankCity': _city_,
                         '[bB]usinessContact': _full_name_,
                         '[cC]ity': _city_,
                         '[cC]ountryCode': _country_code_,
                         '[cC]country': _country_,
                         '[dD]ateOfBirth': _date_,
                         '[dD]ob': _date_,
                         '[dD]ocumentNumber': _ssn_,
                         '[pP]assport': _ssn_,
                         '[iI]dentityDocument': _ssn_,
                         '[iI]dNumber': _ssn_,
                         '[iI]dCard': _ssn_,
                         '[dD]rivingLicense': _ssn_,
                         '[cC]reditCard': _credit_card_number_,
                         '[eE]mail': _email_,
                         '[pP]hone': _phone_number_,
                         '[pP]honeCountryCode': _country_code_,
                         '[gG]ender': _gender_,
                         '[hH]ouse': _building_number_,
                         '[bB]uilding': _building_number_,
                         '[aA]partment': _building_number_,
                         '[aA]pt': _building_number_,
                         '[iI]ban': _iban_,
                         '[sS]tate': _state_,
                         '[pP]rovince': _province_,
                         '[sS]treet': _street_,
                         '[rR]ecordLocator': _alphanumeric_,
                         '[rR]eservationCode': _alphanumeric_,
                         '[lL]ocation': _location_,
                         '[lL]atitude': _latitude_str_,
                         '[lL]ongitude': _longitude_str_,
                         '[lL]at': _latitude_str_,
                         '[lL]on': _longitude_str_,
                         "[tT]imestamp": _timestamp_str_,
                         "[sS]ignature_sha1": _id_,

                         },
                    'number':
                        {
                            "[tT]imestamp": _timestamp_str_,
                            "[dD]ate": _timestamp_str_,
                            "[bB]alance": _amount_str_,
                            "[aA]mount": _amount_str_,
                            "[cC]redit": _amount_str_,
                            "[cC]reditScore": _credit_score_str_,
                            "[sS]core": _credit_score_str_,
                            "[lL]atitude": _latitude_str_,
                            "[lL]ongitude": _longitude_str_
                        }
                    }

exclusions = [".*amazonaws.com"]

Tests

Run tests:

python -m unittest

Contribute

Please see CONTRIBUTING.

License

PrivAPI is released under the Apache License. See the bundled LICENSE file for details.