When syncing a source to a destination not all columns get copied

[antonioorct]

Version: Jitsu Next @ latest

Source postgres table:

Destination postgres table:

The source postgres table contains only one row whose scope and workspaceId fields are null (these fields are missing from the destination table). So it seems to be only happening when a column in all rows are null.

I assume this is an issue with Airbyte but I still want to check in.

[antonioorct]

Going into the json_schema fetched after a call to /discover I get the following for this source table:

{...
    "json_schema": {
        "type": "object",
        "properties": {
            "id": {
                "type": "string"
            },
            "scope": {
                "type": "string"
            },
            "userId": {
                "type": "string"
            },
            "createdAt": {
                "type": "string",
                "format": "date-time",
                "airbyte_type": "timestamp_without_timezone"
            },
            "updatedAt": {
                "type": "string",
                "format": "date-time",
                "airbyte_type": "timestamp_without_timezone"
            },
            "preferences": {
                "type": "string"
            },
            "workspaceId": {
                "type": "string"
            }
        }
    }
}

I'm not sure if it's related that none of the fields have got any property to indicate nullable.

[absorbb]

Look like something that we can fix on our side. We will take a look

[vklimontovich]

@absorbb is this still an issue?

[vklimontovich]

It's not a bug, it's how Jitsu is designed at the moment. The connector sends data to Bulker as an array (stream) of JSON object, and Bulker omits all null values. Also, it create columns on demand, so as a result the columns are not being created. But once the column will have a non-null value at least in one record coming in, it will be created.

That being said, I agree that the current design is not ideal. We will make a change that will remove null value deletion if the data is coming from Jitsu Connectors

[absorbb]

Fixed in jitsucom/bulker:v1.0.1