Index keeps growing in size

[RvZEU]

We have created an static index. Which loops through each product and create an flat index of all the product data based on translation data, so we can easy query through translation data based on translation ids.

But this type of index keeps growing in size. When we compact an database the size of the index is 1.5 GB but in one day it suddenly grows to > 2 GB. When nothing changes in the documents you should think that the size will be the same.

Can somebody explain why this is/happens and if we can prevent this.

[ayende]

It relates to how RavenDB does allocation of disk space. See: https://ayende.com/blog/193697-A/disk-space-management-inside-voron

We are doing that on 1GB boundary once you hit the 1GB mark. When you compact, we drop the size to the minimum required.
But the next time we need any space, we'll raise that to 2GB. And if you are using more than 2GB, it will jump to 3 immediately.

That has major advantages in terms of performance and disk optimizations. Note that RavenDB manage this internally and this isn't unbounded growth. It's merely pre-allocation and then RavenDB uses that directly.

[RvZEU]

Thanx for the clarification. What is also really interesting is that the increase of the index is when we run this query:

From Products update{this.isUpdated=0}

because this does not change the size of the document itself.

To add more context: the size of the index after compact is 512 MB and after this query it trippled in size.

I would like to understand why it does this, perhaps something is not optimized at our application.

[ayende]

Basically, this sort of change force us to reindex the documents. They changed.
Depending on what indexing engine you use, that means that we need to allocate more space for the new versions, and we'll delete the old versions later.

If you'll use Corax, the disk space is likely going to be more stable.

[RvZEU]

is there a way to set that if this field is changed, do not reindex.?

[ayende]

No, any change to the document will trigger an index