v0.2.0b1

Greenmask 0.2.0b1

This major beta release introduces new features and refactored transformers, significantly enhancing Greenmask's flexibility to better meet business needs.

Playground usage for beta version

If you want to run a Greenmask playground for the beta version execute:

git checkout tags/v0.2.0b1 -b v0.2.0b1
docker-compose run greenmask-from-source

Changes overview

• Introduced dynamic parameters in the transformers

  • Most transformers now support dynamic parameters where applicable.
  • Dynamic parameters are strictly enforced. If you need to cast values to another type, Greenmask provides templates and predefined cast functions accessible via cast_to. These functions cover frequent operations such as UnixTimestampToDate and IntToBool.

• The transformation logic has been significantly refactored, making transformers more customizable and flexible than before.

• Introduced transformation engines

  • random - generates transformer values based on pseudo-random algorithms.
  • hash - generates transformer values using hash functions. Currently, it utilizes sha3 hash functions, which are secure but perform slowly. In the stable release, there will be an option to choose between sha3 and SipHash.

• Introduced static parameters value template

Notable changes

Core

• Introduced the Parametrizer interface, now implemented for both dynamic and static parameters.
• Renamed most of the toolkit types for enhanced clarity and comprehensive documentation coverage.
• Refactored the Driver initialization logic.
• Added validation warnings for overridden types in the Driver.
• Migrated existing built-in transformers to utilize the new Parametrizer interface.
• Implemented a new abstraction, TransformationContext, as the first step towards enabling new feature transformation conditions (#34).
• Optimized most transformers for performance in both dynamic and static modes. While dynamic mode offers flexibility, static mode ensures performance remains high. Using only the necessary transformation features helps keep transformation time predictable.

Documentation

Documentation has been significantly refactored. New information about features and updates to transformer descriptions have been added.

Transformers

• RandomEmail - Introduces a new transformer that supports both random and deterministic engines. It allows for flexible email value generation; you can use column values in the template and choose to keep the original domain or select any from the domains parameter.

• NoiseDate, NoiseFloat, NoiseInt - These transformers support both random and deterministic engines, offering dynamic mode parameters that control the noise thresholds within the min and max range. Unlike previous implementations which used a single ratio parameter, the new release features min_ratio and max_ratio parameters to define noise values more precisely. Utilizing the hash engine in these transformers enhances security by complicating statistical analysis for attackers, especially when the same salt is used consistently over long periods.

• NoiseNumeric - A newly implemented transformer, sharing features with NoiseInt and NoiseFloat, but specifically designed for numeric values (large integers or floats). It provides a decimal parameter to handle values with fractions.

• RandomChoice - Now supports the hash engine

• RandomDate, RandomFloat, RandomInt - Now enhanced with hash engine support. Threshold parameters min and max have been updated to support dynamic mode, allowing for more flexible configurations.

• RandomNumeric - A new transformer specifically designed for numeric types (large integers or floats), sharing similar features with RandomInt and RandomFloat, but tailored for handling huge numeric values.

• RandomString - Now supports hash engine mode

• RandomUnixTimestamp - This new transformer generates Unix timestamps with selectable units (second, millisecond, microsecond, nanosecond). Similar in function to RandomDate, it supports the hash engine and dynamic parameters for min and max thresholds, with the ability to override these units using min_unit and max_unit parameters.

• RandomUuid - Added hash engine support

• RandomPerson - Implemented a new transformer that replaces RandomName, RandomLastName, RandomFirstName, RandomFirstNameMale, RandomFirstNameFemale, RandomTitleMale, and RandomTitleFemale. This new transformer offers enhanced customizability while providing similar functionalities as the previous versions. It generates personal data such as FirstName, LastName, and Title, based on the provided gender parameter, which now supports dynamic mode. Future minor versions will allow for overriding the default names database.

• Added tsModify - a new template function for time.Time objects modification

• Introduced a new RandomIp transformer capable of generating a random IP address based on the specified netmask.

• Added a new RandomMac transformer for generating random Mac addresses.

• Deleted transformers include RandomMacAddress, RandomIPv4, RandomIPv6, RandomUnixTime, RandomTitleMale, RandomTitleFemale, RandomFirstName, RandomFirstNameMale, RandomFirstNameFemale, RandomLastName, and RandomName due to the introduction of more flexible and unified options.

Contributors

@dev-comrade
@wwoytenko

v0.1.14

Greenmask 0.1.14

This release introduces bug fixes.

Changes

• Fixed panic caused by Large Object dumper

Contributors

@wwoytenko

Special thanks

@TCY16

v0.1.13

Greenmask 0.1.13

This release introduces only improvements in documentation deployment. The core greenmask utility does not contain any changes.

Changes

• Added documentation deployment with versioning

Contributors

@tarbaev-vl

v0.1.12

Greenmask 0.1.12

This release introduces improvements and bug fixes

Changes

• Fixed config decoding issue
• Fixed TOC entries merge behavior when the data section is empty
• Fixed integration tests for S3 storage

Contributors

@wwoytenko

Special thanks

@viniciuschiele

v0.1.11

Greenmask 0.1.11

This release introduces improvements and bug fixes

Changes

• Added support for generated columns in the table #77
• Fixed transformer parameters encoding issue caused by spf13/viper #76
• Fixed table scoring for transformed table
• Refactored connection management logic in restore command - fixes connection idle timeout #75

Contributors

@wwoytenko
@tarbaev-vl

Special thanks

@janmeier
@dani
@MamesPalmero

v0.1.10

Greenmask 0.1.10

This release introduces improvements and bug fixes

Changes

• Fixed panic caused in RandomString transformer
• Fixed wrong table size calculation. Now the table size includes TOAST table size
• Added custom transformer interaction API defaults if not set
• Changed docker workdir to greenmask home
• Removed bucket name from object path prefix

Contributors

@joao-zanutto
@wwoytenko

Special thanks

@janmeier
@dani

v0.1.9

Greenmask 0.1.9

This release introduces improvements and bug fixes

Improvements

• Implemented tables scoring according to the table size and transformation costs. This correctly spreads the tables dumping
  between the requested worker's pool and reduces the execution time. Now greenmask introspects the table size, adds
  the transformation scoring using the formula
  score = tableSizeInBytes + (tableSizeInBytes * 0.03 * tableTransformationsCount), and uses the strategy "Largest
  First". The problem is described here
• Introduced no_verify_ssl parameter for S3 storage
• Adjusted Dockerfile
  • Changed entrypoint to greenmask binary
  • The greenmask container now runs under greenmask user and groups
• Refactored storage config structure. Now it contains the type that is used for the storage type determination
• Most of the attributes may be overridden with environment variables where the letters are capitalized and the dots
  are replaced with underscores. For instance, the setting storage.type might be represented with the environment
  variable STORAGE_TYPE
• Parameter --config is not required anymore. This simplifies the greenmask utility user experience
• Directory storage set as the default
• Set the default temporary directory as /tmp
• Added environment variable section to the configuration docs

Fixes

• Fixed S3_REGION environment variable usage. Tested cases where the S3 storage is set up using S3 variables that
  uses by github.com/aws/aws-sdk-go
• Updated project dependencies to the latest version

Contributors

@joao-zanutto
@wwoytenko

Special thanks

@janmeier

v0.1.8

Greenmask 0.1.8

This release introduces improvements and bug fixes

Improvements

• Implemented --exit-on-error parameter for pg_restore run. But it does not play for "data" section restoration now. If any error is caused in data section greenmask exits with the error whether --exit-on-error was provided or not. This might be fixed later

Fixes

• Fixed dependent objects dropping when running with the restore command with the --clean parameter. Useful when restoring and overriding only required tables
• Fixed show-dump command output in text mode
• Disabled CGO. Fixes problem when downloaded binary from repo cannot run
• Fixed delete dump operation

Contributors

@wwoytenko

v0.1.7

Greenmask 0.1.7

This release introduces improvements, bug fixes, and documentation updates.

New features

• Added restoration filtering by --table, --schema and --exclude-schema parameters
• Validate command without parameters validates only the configuration file
• Added the --schema parameter, which allows to make a schema diff between the previous dump and the current. This
  is useful when you want to check if the schema has changed after the migration. By controlling it we can exclude
  data leakage after migration
• Validate command divided by many stages that can be controlled using parameters
  • Configuration validation
  • Transformer validation
  • Constraint violation check
  • Data difference check
  • Schema difference check

Improvements

• Improved Hash transformer
  • Added salt parameter that can be set via config or via GREENMASK_GLOBAL_SALT
  • Added sha3 functions support in different modes (sha3-224, sha3-256, sha3-384, sha3-512)
• Refactored Cmd transformer logic
  • Json API: Now it allows to use of column names instead of column indexes in JSON format
  • Csv API: Now it can use the column order from config via column remapping
• The validate command was rewritten almost from scratch.
  • New option --transformed-only - displays only columns that are transformed with a primary key (if exists). This
    allows to reduce the output data and make it more readable
  • Implemented json format for output
  • Added the --table-format parameter which is responsible for the vertical and horizontal table orientation.
    This works only when --format=text
  • Added the --warnings parameter, if it is specified then not only fatal-warnings will be displayed, but also
    those with a lower severity

Fixes

• Fixed --use-list option - now it applies to entries according to the order in the list file
• Fixed --use-list option behavior together with --list-format option (json or text). Now it
  generates a temporal list file in text format for providing it to the pg_restore call
• Updated documentation according to the latest changes

Contributors

@wwoytenko

v0.1.6

Greenmask 0.1.6

This is a minor release that introduces a bug hotfix

Fixes

• Fixed uncontrolled buffer growth in the restore command #22

Contributors

@wwoytenko

Special thanks

@janmeier