Pull request description

This PR contains changes in the greenmask core. This is a base of the next major version v0.2. The changes in this PR are a result of the 5 months of continuous development and close connection with greenmask users.

Changes overview

• Introduced dynamic parameters in the transformers
  • All transformers (where required) have been moved to the dynamic parameters support
  • The dynamic parameters are strict, but If you need to cast the value to another type greenmask provides two template and pre-defined cast functions that can be provided via cast_to. The cast_to functions represented the most on the frequent cast operations such as UnixTimestampToDate, IntToBool, etc
• The transformation logic is significantly refactored. Transformers are more customizable and flexible than before
• Introduced transformation engines
  • random - generates the transformer value based on pseudo-random
  • hash - generates the transformer value based on hash functions. For now, it is sha3 hash functions that have low performance but are secure. In the stable release, it is expected to have the option to choose between sha3 and SipHash

Notable changes

Core

• Introduced Parametrizer interface and it has been implemented for dynamic and static parameters
• Renamed most of the toolkit types now they are clear and covered with docs
• Refactired Driver initialization logic
• Added validation warnings for Driver for overridden types
• Migrated already implemented built-in transformers to the new rails with Parametrizer interface
• Implemented new abstraction TransformationContext that is a first step towards new feature transformation conditions Feature: conditional transform #34
• Most of the transformers have been optimized for working in dynamic and static mode. While the dynamic mode is flexible the static mode remains performant. By using only the required transformation features the transformation time is predictable.

Transformers

• Email - a new transformer that supports both random and deterministic engines. Generate email value in a flexible way. You can use the column values in the template for the generation of the email. At the same time, you have the option to keep the original domain or use any domain from the domains parameter.
• NoiseDate, NoiseFloat, NoiseInt - Supports random and deterministic engines. Provides parameters with dynamic mode support that can control threshold values in the min and max range. The old implementations have only one ratio parameter and the noise can be generated between the values [0, ratio]. In a new release we reviewed that logic and introduced two parameters min_ratio and max_ratio where the noise value will be in the range [min_ratio, max_ratio]. By applying the hash engine in Noise* transformers we can complicate statistical analysis for the attacker when the same salt is used for the backups over a long period of time.
• NoiseNumeric - implemented a new transformer that supports the same features as in the NoiseInt and NoiseFloat but is implemented to work only with numeric (huge int or float values). Since a Numeric value can be with a fraction the parameter precision is introduced.
• RandomChoice - added hash engine support
• RandomDate, RandomFloat, RandomInt - added hash engine support. The threshold parameters min and max now supports dynamic mode.
• RandomInt - implemented a new transformer that supports the same features as in the RandomInt and RandomFloat but is implemented to work only with numeric (huge int or float values).
• RandomString - added hash engine mode support
• RandomUnixTimestamp - implemented a new transformer that generates Unix Timestamp with units sec, milli, micro, nano. The logic is similar to RandomDate. The transformer supports the hash engine and dynamic parameters for the min and max thresholds. The units for min and max parameters can be overridden by min_unit and max_unit parameters.
• RandomUuid - added hash engine support
• RandomPerson - implemented a new transformer instead of RandomName, RandomLastName, * RandomFirstName, RandomFirstNameMale, RandomFirstNameFemale, RandomTitileMale and RandomTitileFemale. This new transformer represents the same but highly customizable features that the previous versions. The transformer generates Personal data such as FirstName, LastName, and Title based on the provided gender parameter. The gender parameter supports dynamic mode. In the next minor versions, we will provide the possibility to override the default names database.
• Added tsModify template function
• Most of the static parameters support the template as a value. You can generate the parameter value using a template, for instance min: '{{ now | tsModify "-30 years" | .EncodeValue }}' # 1994