Very large files fail to parse

[apkrieg]

Describe the bug

I'm currently trying to parse a very large YAML file, approximately 4.8 GB, and it's failing.

UPDATE: I was able to parse the 4.8 GB file by changing the type of Index from int to long on Mark, Cursor, and SimpleKey.

To Reproduce

Try to parse a YAML file that is more than ~2.2 GB in size.

[apkrieg]

If this is something that would be accepted in a PR, I'd love to contribute, but I believe it would break things like Mark.

[apkrieg]

I plan to make a PR for this sometime next week. If this change would not be accepted because it could be breaking, please close this so I don't waste the time. Cheers!

[EdwardCooke]

Initially I think converting ints to longs where needed would be fine. I can’t really see much of a downside other than the potential for slightly increased memory footprint (4 bytes vs 8). Would be interesting to compare the numbers in your use case of your massive file.

[EdwardCooke]

If you don't have time to work on this, then I may be able to find time. Do you have a download link for that yaml file you could share, I'd like to compare apples to apples when doing this work.

[lahma]

Sounds quite natural that int won't fit a index large enough when parsing such a huge file. Unfortunately changing ints to longs will have negative performance impact for smaller files.. I'm not entirely sure if such big files ought to be supported without them being split to logical parts. Many parsers do not expect input files to be that large.