High RAM consumption

[samuel-rufi]

Hi @shrimalmadhur, I would like to give you a status update from my side:
The very high RAM consumption due to the "memory_limit_disabled" = true hotfix for #245 now is indeed a problem as rosetta-cli breaks down after some hours already with 16 GB RAM.
Unfortunately I also can't run the rosetta-cli without "memory_limit_disabled" = true else it will break in the account bootstrapping phase as it can't handle large bootsrapping.json files #245.

I guess it all comes back to the Badger Tx size / or how the Badger commits are done. Maybe this could be adjusted or are there other options?

Many thanks for your time and efforts.

[shrimalmadhur]

Hey @rufsam, thanks for raising this. Yea it does come back to tx size. We might need to optimize how we do badger transactions. I have not gotten a chance to work on it yet but it is in my radar.

Also feel free to contribute if you have some cycles and can work on resolving this. We always welcome contributions.

[shiatcb]

Hi @rufsam , would you mind providing the bootstrap_balances.json with 35k accounts so we can repro this issue? Feel free to drop the file in your comments or email shi.cheng@coinbase.com, whatever you prefer.

[samuel-rufi]

Thanks alot @shrimalmadhur @shiatcb for your fast answers and help. This is the bootstrap_balances.zip I'm using. So after loading around 35k accounts the badger error should appear. If there is anything I can help you with, just tell me! 👍

[shiatcb]

@rufsam thanks for the quick reply! Also are you able to provide the config for peering.json (devnet/mainnet) with multiaddress? I am now following the deployment procedure and trying to bring up node svc & rosetta svr.

[samuel-rufi]

Hi @shiatcb, many thanks for working through it 👍 Maybe we should switch to e-mail so that I can provide you with better assistance. I will contact you with all the required info!

[shiatcb]

My pleasure, you can email shi.cheng@coinbase.com whenever you have the info.

[samuel-rufi]

Sent! 😄

[muXxer]

Hey! Just wanted to mention that we had a lot of problems with BadgerDB in our own project for years.
After switching to RocksDB, memory usage dropped significantly, and speed also increased dramatically.

We implemented an Interface for our software called KVStore so we can switch the used database engine easily (gRocksDB, Badger, bbolt, pebbledb).

[shrimalmadhur]

@muXxer that's a very interesting point. Are there any data points which I can refer on how much improvement was there? If this makes us more memory efficient, we could definitely consider moving to RocksDB