You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm evaluating DuckDB to store structured logs. The queries will typically be aggregations on values of one key, so naturally I would prefer a column-major DB.
And a query could be an aggregation of the number of ERROR logs in a given timeframe.
However, I'm tripping up on the topic of an unknown schema: If some application that reports its logs to the DB adds a new field, the DB needs to be able to handle it without me manually changing the schema.
Now, surely I could just dump all the data in a a JSON blob, but I would assume that loses the benefit of having an analytics DB. DuckDB will probably not manage to have all the matching keys in the JSON blob in a neat column for fast querying.
I thought this is a common problem for an OLAP DB, but I can't find any good info. How do people manage this? Does the Map type help here?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
I'm evaluating DuckDB to store structured logs. The queries will typically be aggregations on values of one key, so naturally I would prefer a column-major DB.
Think:
And a query could be an aggregation of the number of ERROR logs in a given timeframe.
However, I'm tripping up on the topic of an unknown schema: If some application that reports its logs to the DB adds a new field, the DB needs to be able to handle it without me manually changing the schema.
Now, surely I could just dump all the data in a a JSON blob, but I would assume that loses the benefit of having an analytics DB. DuckDB will probably not manage to have all the matching keys in the JSON blob in a neat column for fast querying.
I thought this is a common problem for an OLAP DB, but I can't find any good info. How do people manage this? Does the Map type help here?
Beta Was this translation helpful? Give feedback.
All reactions