JSON file will become too big

[mrworf]

The way data is stored and processed today is using JSON. This works pretty well until you eventually have too many changes and JSON file gets too big. Since it's loaded into memory as one big block, this will become a scaling issue down the road.

Need to investigate other methods of storing the metadata or simply splitting it into multiple files to keep it simple (would prefer not to use sqlite or similar).

This issue is created so I can track thoughts and ideas as I experiment with different ways. Needless to say it must be backward compatible and not break existing backups.

[securityfrog]

I've had to complete some prelimnary investigation on this before @mrworf you may be interested in checking out BSON or YAML due to the size issues

[mrworf]

It's not so much the file size as holding it in memory. I like JSON because it's "human readable" meaning that it can be rescued/repaired/converted easier without having the reader. But hold it all in memory is less than ideal once the file gets big enough.