[Appender] Add AppendDefault

[Tishj]

This feature has been requested many times, so I figured it was time to add it.

We add AppendDefault to the Appender class (not BaseAppender).
This method can be used in place of Append(Value val)

The behavior of AppendDefault is the same as calling INSERT INTO <name> VALUES (DEFAULT) through SQL

[Mytherin]

Thanks for the PR! Great idea. Some comments:

[taniabogatsch]

Hey, thanks for the PR! These changes come in handy!

The main functionality I am missing in this PR is writing a default value directly to a vector in a data chunk. Any performance-focused appender implementation most likely uses duckdb_append_data_chunk instead of appending value-by-value.

From my understanding, in order to write default values to a vector in a data chunk, we need the context of the appender, as we need to know the context of the table. I.e., we cannot add this functionality to the data chunk API. So, maybe it is sensible to extend this PR with a function Appender::AppendDefaultToVector, which takes a vector and a row index and writes the default value to the expected position... 🤔

[Tishj]

So, maybe it is sensible to extend this PR with a function Appender::AppendDefaultToVector, which takes a vector and a row index and writes the default value to the expected position... 🤔

That makes sense, I think to be performant it makes more sense to take a list of integers instead of a single row index.
Since internally we use an ExpressionExecutor, running this expression on multiple rows at once is likely more performant.

[Tishj]

So, maybe it is sensible to extend this PR with a function Appender::AppendDefaultToVector, which takes a vector and a row index and writes the default value to the expected position... 🤔

That makes sense, I think to be performant it makes more sense to take a list of integers instead of a single row index. Since internally we use an ExpressionExecutor, running this expression on multiple rows at once is likely more performant.

@taniabogatsch I've made a commit that starts this, taking a SelectionVector and a count for the amount of defaults that need to be appended (taking the indices from the SelectionVector)

I've had to make some adjustments to the ExpressionExecutor that make me think this is not it's intended purpose (even though it does already take a const SelectionVector *sel)
Please check out this commit first PoC for AppendDefaultToVector

[Giorgi]

Any performance-focused appender implementation most likely uses duckdb_append_data_chunk instead of appending value-by-value.

From my understanding, in order to write default values to a vector in a data chunk, we need the context of the appender, as we need to know the context of the table. I.e., we cannot add this functionality to the data chunk API. So, maybe it is sensible to extend this PR with a function Appender::AppendDefaultToVector, which takes a vector and a row index and writes the default value to the expected position... 🤔

Not only performance-focused, duckdb_append_data_chunk is needed to append more "complex" data types (uuid, decimal, enum, list, etc) so we definitely need a way to append a default value to a vector.

[Tishj]

I didn't feel like the appender was the right place to add methods that query table information.
As such I've created a duckdb_table_description object, which wraps a TableDescription object, which is also present inside the Appender.

This can then be queried for information; duckdb_column_has_default(table, 0)
I imagine this can be extended in the future to include duckdb_column_is_generated, duckdb_column_get_type, duckdb_column_get_name, etc..

[taniabogatsch]

Hi @Tishj, the changes and improvements look nice! 😄 I've added mostly nits and also some comments.

[Giorgi]

@Tishj Will you expose AppendDefaultsToVector in the C API as part of this PR?

[taniabogatsch]

Nit: we can remove temp_idx and use i, no?

[taniabogatsch]

I like this. We can eventually use it to add another appender creation function allowing non-default schema appenders. Should we add it to this PR and change the parameters of duckdb_table_description_create?

[Tishj]

//! The qualifier contains a path into the database.
//! Must be destroyed with `duckdb_qualifier_destroy`.
typedef struct _duckdb_qualifier {
	void *__qual;
} * duckdb_qualifier;

I really hope it doesn't need a destroy method, but since it will contain std::string variables, it probably needs it right?

[Tishj]

@Mytherin how do you feel about this?

Like Tania said, we are already suffering from this issue in the appender, possibly other places.
This way we wouldn't need to make a breaking change to add a char *catalog to duckdb_appender_create and can instead deprecate it, in favor of duckdb_appender_create_with_qualifier ?

[Tishj]

Also, somewhat related, I think I'll separate out the duckdb_table_description addition into a separate PR, as they can now be made standalone
Then we can worry about whether we want the duckdb_qualifier on that PR

[Mytherin]

Let's just add a duckdb_appender_create_ext method that also takes the catalog, and leave the current method alone? I think adding the duckdb_table_description stuff seems overkill.

[taniabogatsch]

The duckdb_table_description also serves other use cases such as getting the column count, has_default, column_name, etc. If a separate struct is overkill, we need to add this functionality to the appender. I.e., duckdb_appender_column_has_default and duckdb_appender_column_name.
cc @Mytherin

[Giorgi]

By the way, there is already suggestion for duckdb_appender_column_name #11737

[taniabogatsch]

@Giorgi, we plan to expose a way for the appender to write default values when using append_data_chunk in the C API. We'll expose it in a follow-up PR if it is not included in this PR.

[Giorgi]

@Giorgi, we plan to expose a way for the appender to write default values when using append_data_chunk in the C API. We'll expose it in a follow-up PR if it is not included in this PR.

Great! If we don't specify that we want to use default values for some of the columns, will the DuckDB appender automatically use default values for that column? For example, if I have a table with ID auto-increment, will we still need to tell DuckDB that we want to use default for that column?

[taniabogatsch]

Great! If we don't specify that we want to use default values for some of the columns, will the DuckDB appender automatically use default values for that column? For example, if I have a table with ID auto-increment, will we still need to tell DuckDB that we want to use default for that column?

Currently, yes. In this first iteration, it is necessary to specify that a value is a default value (AppendDefault or by using AppendDefaultsToVector). For example, not writing any value (specific or default) to an auto-increment ID will result in an error.