Data Type Adapters

[fretz12]

What

A way for a user to define data type mappings from synth schema -> data source, and from data source -> synth schema.

Why

Synth schemas have generic, "superset" data types, but many data sources have very specific data types that synth currently has no way of mapping into. For example, Postgres has an inet data type, but synth knows nothing about the inet and the best it could do today is treat it as a string.

On a related problem, certain data sources need an extra layer of instructions before de/serializing to/from the synth schema. For example, #15 requires the user to provide some extra info on how to read from or write to binary files. The same can apply to #32 and #33.

In either cases, explicit user intervention is a must. We need some sort of unifying schema so the user can define these "adapters". On the other hand, there is data source specific dev work to serve the other end of the adapter.

How

Here are some preliminary proposals on how we could solve this.

1. Define an adapters map in-band the current synth schema

{
  "type": "array",
  "length": {
    "type": "number",
    "constant": 1
  },
  "content": {
    "type": "object",
    "my_ip_address": {
      "type": "string",
      "faker": {
        "generator": "ipv4"
      },
      "adapters": {
        "postgres": {
          "type": "inet"
        }
      }
    },
    "big_blob": {
      "type": "string",
      "pattern": "[a-zA-Z0-9]{0, 255}",
      "adapters": {
        "mysql": {
          "type": "blob"
        },
        "postgres": {
          "type": "bytea"
        }
      }
    },
    "id": {
      "type": "number",
      "subtype": "u64",
      "adapters": {
        "binary_file": {
          "type": "plain_bytes",
          "endianness": "little",
          "payload_start_bytes": "4096",
          "entry_offset_bytes": "8"
        }
      }
    }
  }
}

Pros:

• Adapter context is maintained; it's clear what the adapter applies to
• No need to manage multiple files

Cons:

• It can quickly clutter the current schema
• This works when exporting from a schema, but not importing a schema. At that point a synth schema doesn't exist yet, so there must be another means to define the adapters

2. Define an adapters map in out of band schema.

{
  "postgres": [
    {
      "type": "inet",
      "direction": "any",
      "target_fields": [
        "my_ip_address"
      ]
    },
    {
      "type": "bytea",
      "direction": "import",
      "target_fields": [
        "big_blob"
      ]
    }
  ], 
  "mysql": [
    {
      "type": "blob",
      "direction": "export",
      "target_fields": [
        "big_blob"
      ]
    }
  ],
  "binary_file": {
    "type": "plain_bytes",
    "direction": "any",
    "file": "/path/to/exported/file",
    "endianness": "little",
    "payload_start_bytes": "4096",
    "entry_offset_bytes": "8",
    "target_fields": [
      "id"
    ]
  }
}

Perhaps this file can live inside an adapters dir inside a namespace

Pros:

• Clean separation from existing synth schemas
• Accommodates both import and export

Cons:

• More files to maintain
• Need to link back to synth schema fields, making it harder to follow

3. Your input is appreciated!

[llogiq]

• I am totally with you in wishing to enrich our data model. For some use cases (e.g. binary data), this is a necessary precondition
• Both @christoshadjiaslanis and I have thought about a separation of semantics (the generators) and representation (in this case the data types). Especially with the addition of new import/export strategies, we will need a richer value type with more variants for some things
• Note that the representation is strategy-dependent, where some (e.g. JSON) lack certain data types (e.g. byte arrays). Therefore, there can be multiple representation specs, one for each supported strategy, or one spec with extensions depending on strategy
• We want a sensible default representation for each data type. This way, for most cases, we won't need any representation spec at all. The hope is that the representation specs can remain sparse, with type or attribute matchers mapped to representation rules
• I am currently preparing a departure from the JSON based configuration language (look in the feat/lang_bindings branch if you're interested), but this will likely take a while until ready.

[fretz12]

Excellent points @llogiq !

• I am totally with you in wishing to enrich our data model. For some use cases (e.g. binary data), this is a necessary precondition
• Both @christoshadjiaslanis and I have thought about a separation of semantics (the generators) and representation (in this case the data types). Especially with the addition of new import/export strategies, we will need a richer value type with more variants for some things

Agreed. Maybe this is a dumb suggestion- but what if for just simple type forcing, we simply express with an extra field, like this:

    "my_ip_address": {
      "type": "string",
      "faker": {
        "generator": "ipv4"
      },
      "explicit_type": "inet"
    },

and synth will behind the scenes try best effort to force it into that type if it finds a match, else do nothing. Sounds a bit hackish, but from the user standpoint it's dead simple.

• Note that the representation is strategy-dependent, where some (e.g. JSON) lack certain data types (e.g. byte arrays). Therefore, there can be multiple representation specs, one for each supported strategy, or one spec with extensions depending on strategy

Oh interesting... it never occurred to me a non-JSON config would be considered. That's a good idea... certain strategies sit well with certain formats, I'll keep that in mind.

• We want a sensible default representation for each data type. This way, for most cases, we won't need any representation spec at all. The hope is that the representation specs can remain sparse, with type or attribute matchers mapped to representation rules

Agreed... most cases should really just work right off the bat. It's really these corner/custom cases that need to go down this other route.

• I am currently preparing a departure from the JSON based configuration language (look in the feat/lang_bindings branch if you're interested), but this will likely take a while until ready.

Yeah I'm totally digging using a scripting language to solve the customization problem. See my comments in #69