Data storage and retrieval

[Varunram]

Data Storage and Retrieval

When a platform primarily relies on data(such as openclimate), multiple problems related to data storage and retrieval come into picture. The platform in this case must have access to a certain subset of data but oracles must have access to all data in order to ensure that the data subset was indeed accurate. There are three approaches to giving another person access to data:

1. Direct access to data
2. Access to an Oracle which allows some queries to be made against data
3. Not allowing access

There is a fourth category which is third party audits (parties don't permit direct access to data but allow certain third parties to audit said data) but that would not fall under a technical solution.

The first approach is relatively simple - the party copies a portion of data into another directory and gives people access to that directory. The party could also store data in ipfs, publish the encryption key and allow people to retrieve data from there.

The second approach resonates with #30 and the construction will be quite similar as well

The third approach of denying access to data might sound weird but there are some categories within this class:

1. Publishing Zero knowledge proofs of data that allow independent third party verifiers to verify that the data is accurate
2. Publishing proofs that somehow prove that they are inline with their promised commitments
3. No data

The first two categories above are interesting to explore since most companies will refuse to share data publicly and will not agree to have an oracle which would allow some queries to be made.

The data itself can be stored in multiple places - on ipfs, on a traditional db system, on a blockchain, etc. It is important that access to this falls inline with the methods of data retrieval discussed above.

Another parameter to tweak would be the maximum amount of data that a particular entity can store. Since data will be encrypted and the platform itself would not have an idea about what data is being stored, a malicious party can upload arbitrary data to the platform, increasing storage costs. The parameter must not be too small such that committed reporters find it difficult to report their emissions while at the same time, it must not be too big such that bad actors take advantage of the platform. The best way would be to have an adjustable file size limit (people who report more have more storage access) with a default of 10MB.