[Feature Request]: <Productize DataFlow engine so that automation could be obtained using natural key meta information for each topic/row/eventdata>

[allamsudha]

Related Template(s)

Kafka to BigQuery

What feature(s) are you requesting?

1. Provide UI for mapping data types from source to BigQuery. In the case of JDBC, these are mapped in the solution itself, whereas, sources such as pub/sub, kafka, etc., may need customer provided mapping.

2. DataStream reads the natural key from source systems using CDC. Some of the customers may prefer relational data publishing to Kafka using mirror copies. In which case, it is ideal for customers to set the natural key via UI or read from a BigQuery table populated by them. Provide this functionality. Either data type mapping or natural key should be possible to be entered from UI or read from customer populated tables in BigQuery.

3. Add effective_begin_date, effective_end_date and current_flag fields for each table created as these fields dictate how the history is preserved and as well how various different analytical queries work. Almost all the customers would be happy to have these fields auto populated when merging is happening.

4. Provide another UI element for customers to edit Dead letter messages for them to put back into the queue for processing. Provide an easy way setting up notifications from an UI that interfaces with cloud logging so as to send groups of users notifications to their cell phones.