What is the best practice to do persistence?

[Catstyle]

I like this lib, but i am now having a storage issue with just a little amount of objects.
I got around 40K embeddings in a collection, which size should be around 100MB, but it turns out used almost 20GB storage which seems too much!

It will fetch more embeddings and do a save_collection for a fixed period in my use case.
It seems creating a file include the whole collection data ???

My question is how to do save_collection/persist the right way?

[edwinkys]

Hey, first of all, thank you for using OasysDB 😁

It'd be great if you can share a minimum reproducible example of your code you use to save the collection so I can help you troubleshoot it better.

But in general, what you want to do is:

1. Open the database to a file.
2. Create collection or do what ever your want with the collection.
3. Save the collection. If you are saving an existing collection, use the same name you use to retrieve the collection.

# This is the directory the database will be stored in.
# This directory shouldn't change unless you want to create a new database.
db = Database("data/example")

# Create a collection or get collection from the database.
# ...

# If you only want to save an existing collection, make sure to use the same namespace.
# Otherwise, it will create a new collection.
db.save_collection("my_collection", collection)

Let me know if that helps 😁

[Catstyle]

Thanks for your reply.

I'm using oasysdb==0.4.2

from oasysdb.prelude import Vector, Config, Database, Collection, Record

db = Database("test")
config = Config.create_default()
collection = Collection(config)

for idx in range(300):
    # do some actual work and save collection every 100 objects
    # just use Random instead
    records = Record.many_random(dimension=512, len=100)
    collection.insert_many(records)
    db.save_collection("storage", collection)

# here we have 30K objects and the size should be around 60MB + some extra data space
assert db.len() == 1
assert collection.len() == 30_000

But it used almost 10GB storage!

[edwinkys]

You are actually right!

I tested the code you provided againts the optimal way to do persistence and they produce totally different result:
https://colab.research.google.com/drive/1bCOa0e6F8eZvRVo89h5CXp66V9438K4y?usp=sharing

This is the most storage optimal way to save a collection.

from oasysdb.prelude import Vector, Config, Database, Collection, Record

db = Database("test")
config = Config.create_default()
collection = Collection(config)

for idx in range(300):
    # do some actual work and save collection every 100 objects
    # just use Random instead
    records = Record.many_random(dimension=512, len=100)
    collection.insert_many(records)
    # db.save_collection("storage", collection)

# Moving it outside the loop.
db.save_collection("storage", collection)

# here we have 30K objects and the size should be around 60MB + some extra data space
assert db.len() == 1
assert collection.len() == 30_000

After digging further, it seems to be caused by dirty IO buffers not flushed.

But I also understand why you prefer to do it that way. Saving the collection periodically can prevent restarting the process all over again if anything happens especially if dealing with large and compute-heavy datasets.

I really appreciate you finding this out! I will see what I can do to fix this and get back to you with the solution 😁

[edwinkys]

Hey @Catstyle 👋

Sorry for the ping. I come up with a solution here in PR: #84. This adds 2 methods flush and async_flush that you can use to reduce the size of the dirty IO buffers in the database which will subsequently reduce the size.

I'll make a release of v0.4.5 with this that you should be able to use soon. After you try out this solution, please let me know if this new feature solves your issue.

[Catstyle]

Cool, I'll give it a try on Monday.

[Catstyle]

Unfortunately, it doesn't solve the case.
I had try both putting the db.flush() before or after the save_collection call, and even after the for loop, it still used 10GB storage.

I also have something to share.
I noticed that the blobs folder always got 300 files, corresponding to the loop times, and i removed every files but the one with largest number, it can still reopen the db and work!

I also have a look at Sled, it seems like it is using LSMT, than i think it should be the key.

[edwinkys]

I see. Sorry about that.
I am currently communicating with people who create and maintain Sled and try to figure out what's the best way to solve this problem.

I'll keep you updated with this issue and release the fixes to this issue when I found the solution.
Thank you for finding this issue 😁

Update

So I talked to Sled team and it seems the way OasysDB did persistence needs to be optimized. This optimization requires OasysDB to write and read from the disk directly and thus likely would not be backward compatible. Though, I will try my best to make it backward compatible.

This fix will likely be included in release v0.5.0.

[edwinkys]

Hi @Catstyle
Sorry for the ping.

I just want to let you know that we just release the v0.5.0 which will solve the problem we had when performing the save operation repeatedly. I have tested this functionality in this notebook and the database size of one-time save and periodical saves are both around 70MB:
https://colab.research.google.com/drive/1bCOa0e6F8eZvRVo89h5CXp66V9438K4y?usp=sharing

Please let me know if this works for you 😁

[Catstyle]

Hi
Sorry for the late.

I had tested the v0.5.0, problem solved!
Thanks for your work!