CR and LF inside a quote could be treated as part of content and not as new lines

[attilah]

Thanks for this neat library, helped me solve parsing of a large file without any extra effort with little modification to the sources.

The issue I ran into is this: in my CSV file I've a a line like this, which has an exported JSON payload field, which can be multiline:

"","{
    ""id"": ""00000"",
    ""recipient"": """",
    ""components"": {
        ""street_name"": ""foobar"",
        ""zip_code"": ""1111"",
    }
}","","2022-01-01T00:00:00.000Z","false","true","false","false","","","200.0"

IMHO this could be a configuration option on how to treat CR, LF, TAB, other non-visible characters when the tokenizer is inside a string already, because the expected outcome can vary based on different use cases:

• This can be just a simple invalid line so whenever there is a new line, treat it as record ending
• This can be valid content, so parse until the next proper string closing. (Notice how the quotes are escaped within the content)

My specific use case was processing the output of Salesforce Data Loader's CSV export, which is treating strings like this during export: https://github.com/forcedotcom/dataloader/blob/7bf0d84eba97d7c898064222a4e30c10ea9c8dd8/src/main/java/com/salesforce/dataloader/dyna/StringConverter.java#L148