Support multiple and/or multi-character and/or regex comment_char in read_csv()

[Wainberg]

Problem description

Multiple comment_char meaning e.g. ('#', '%') both start comments. Multi-character meaning e.g. // starts a comment, like in C++.

In particular, it would be very nice to support comment_char='##' for VCF files, one of the most common file formats in computational biology. In VCF files, the first few lines are metadata starting with a ## (and should be excluded), but the header line starts with a single #, so comment_char='#' would erroneously exclude the header.

Multi-character comments were requested in pandas, but the feature request (which was originally about multiple rather than multi-character comments) was closed for being "difficult" to implement. I'm sure it would be no problem for the polars team though :)

[stinodego]

For your specific use case, I would recommend setting skip_rows to skip the metadata lines. If you don't know in advance how many lines there are, you could write some util to determine this.

Supporting multiple chars / non-ASCII chars would be nice (for separator / quote_char / comment_char / eol_char), but definitely not simple. If someone manages to implement this elegantly, I wouldn't mind a PR.

[ritchie46]

@stinodego I have refused to add this on many occassions and I really don't think we should add this. This would have very large performance impacts which I don't think are worth it. I want the csv-parser to be performant and close to a formal csv format as possible. Multiple character and worse regex delimiters will have very negative performance impacts.

[ritchie46]

I would only accept mutli-char comments as this can be implemented cheaply.

[Wainberg]

Honestly, I'd be inclined to agree - regex doesn't seem that much more useful and could have a large performance impact unless it's implemented as an entirely separate code path. @ritchie46 would you be inclined to accept #12519 for the multi-char comments?

[Wainberg]

Closing as completed via #12519, thanks everyone!