You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Multiple core docs specify that properties can be renamed with mappers:
The meltano mappers guide documentation indicates, in the Property (Column) Level, section, that Aliasing: properties can be renamed in the resulting stream e.g. altering a column name from id -> customer_id
The Meltano SDK stream_maps documentation, in the Property-Level Mapping Applications section, indicate that stream_maps support Property-level aliasing: properties can be renamed in the resulting stream.
However, in v3.0.0, either this no longer works or all documentation that provides examples have been lost, as I searched through the docs as well as the source code for both the SDK and the meltano-map-transform plugin.
The stream/table name is renamed but the property/column is not. There is a partial work around to make a copy of the column and then delete the old column.
Sorry if this isn't quite a "bug" or if issues with the mapper belong somewhere else - it was not clear where communication / contribution streams are or go (and Slack is unfortunately blocked at my workplace).
There weren't code examples of the nested column format (aside from some "this would be nice" in feature requests), but the two main mapper doc pages both list changing property names as an existing feature of mappers (meltano mappers guide documentation and Meltano SDK stream_maps documentation). So, if property renaming isn't actually an existing (or soon to be coming) feature, it should be removed from the docs.
The copy/delete is doable (especially since it looks like __else__: __NULL__ will clear out the original column names) - but the real need is to be able to handle renames/reformats of the property schema in bulk - Is there some other way to do inline property/schema changes? The meltano/singer Oracle tap pulls in all uppercase, and so what I'm really looking to do as a starting point is something like (and it does seem like it would require a special key to refer to the stream metadata definition):
config:
stream_maps:
{STREAM_NAME}:__meta__[col['name'].lower()]: col
Meltano Version
3.0.0
Python Version
3.10
Bug scope
CLI (options, error messages, logging, etc.)
Operating System
Linux - 20.04
Description
Multiple core docs specify that properties can be renamed with mappers:
meltano mappers guide
documentation indicates, in the Property (Column) Level, section, thatAliasing: properties can be renamed in the resulting stream e.g. altering a column name from id -> customer_id
Property-level aliasing: properties can be renamed in the resulting stream.
However, in v3.0.0, either this no longer works or all documentation that provides examples have been lost, as I searched through the docs as well as the source code for both the SDK and the
meltano-map-transform
plugin.For example, for a config file that looks like :
The stream/table name is renamed but the property/column is not. There is a partial work around to make a copy of the column and then delete the old column.
Sorry if this isn't quite a "bug" or if issues with the mapper belong somewhere else - it was not clear where communication / contribution streams are or go (and Slack is unfortunately blocked at my workplace).
Code
The text was updated successfully, but these errors were encountered: