ENH: Support for multiple comment characters with readers

[ebran]

I would be very pleased if Pandas supported multiple comment characters when reading data from files. According to:

import pandas as pd
df = pd.read_table("data.dat", comment=("#","@"), delim_whitespace=True)

I don't know if this is requires a minor or major implementation effort?

Best,
Erik

[jreback]

this would be a bit of an effort. the reader is basically byte by byte (with some backref capability). So it would have to check agains a buffer of the comment chars (it just checks against the single char now, but only if its not NULL), in a performant manner. Could be done.

[dansondergaard]

Related:

Would be great if a comment character could actually also be two characters, e.g. "##". For example, in VCF files, some meta data is specified at the beginning of the file with "##" before the actual table starts:

http://www.internationalgenome.org/wiki/Analysis/vcf4.0/

Often one just wants to ignore these, but:

df = pd.read_csv("data.vcf", comment="##")

doesn't work. Note that for VCF it won't work to just use comment="#" since the header line actually starts with a single "#".

[wesm]

This would be difficult. I'm closing this for now

[loodvn]

I ran into this feature request when reading a VCF file as the others did.

## Metadata
## ...
# Header
col1, col2, etc

Is the cparser the difficult part of this change?

[loodvn]

Starting a draft PR at #48615, I imagine getting the look-ahead into the state machine of C might be a bit tricky but I'll try give it a go if necessary. Also will check benchmarking for any regressions