Additional 'createdAt' TTL index in MongoDB collections #762
Replies: 4 comments 1 reply
-
Hi,
What do you mean ? |
Beta Was this translation helpful? Give feedback.
-
Hello, Any news on this? |
Beta Was this translation helpful? Give feedback.
-
We just ran into this issue as well, recording started to fail because we hit our Mongo limits. We don't want to pay for a huge Mongo instance, so some way to expire or batch delete test runs that are old would be nice. Or some way to set size limits and have old runs deleted as new ones are created, similar to a FIFO queue. Looking at our data, the bulk of the size is in the instance collection, which stores the data for every single spec. By deleting some of those, the rest of the UI still works, it just shows no data when you click on an individual spec file in a run. I can see 2 possible approaches, the proposed one here that is just a TTL for runs (or just instances if people are okay with older runs losing the details). The other is perhaps to use Capped Collections to set a size limit, and it works like a circular buffer overwriting older data with newer data. |
Beta Was this translation helpful? Give feedback.
-
There was a PR long time ago that was trying the TTL approach, but I required the field to be a proper |
Beta Was this translation helpful? Give feedback.
-
Problem
Director and API services work with MongoDB and store data about runs and other details. After running Sorry-Cypress for some time, I noticed that MongoDB storage usage keeps growing until it reaches storage limits. This happens because there is no TTL for the data and it is kept forever. Normally, test runs details only need to persist in a database for a finite amount of time.
I did some research and I found that MongoDB can have TTL indexes (https://www.mongodb.com/docs/manual/core/index-ttl/), but there is a requirement - field value should be a date or an array that contains date values. Currently, Sorry-Cypress uses collections which doesn't have any index with date inside, so MongoDB TTL indexes can't be created.
Proposed solution
Add additional TTL index 'createdAt' for records in collections when creating DB entries and allow to specify expireAfterSeconds value. This would allow people to define how long they want to see test runs and avoid problems by reaching MongoDB storage limits.
Beta Was this translation helpful? Give feedback.
All reactions