You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The more videos there are, the larger the media_sync table becomes.
The reason for the bloat is that each metadata json is hundreds of KB.
It need a way to solve this.
Detail
I am syncing a source with about 1500 videos.
Each sync_media record has a data size of approximately 500KB.
The reason is because of the bloat of metadata, and a way to reduce this is needed.
The sqlite file reached 700MB, and CPU usage and memory usage were observed to be high due to the overhead caused by - reading/dumping the huge metadata json.
AS-IS
TO-BE
There are many unverified and unworked Mock parts to create P/R.
I am attaching a sample of my work locally.
Sample Source
models.py
classSource(models.Model):
...
LIGHTWEIGHT_METADATA_TYPE_RAW='RAW'LIGHTWEIGHT_METADATA_TYPE_FEATHER='FEATHER'LIGHTWEIGHT_METADATA_TYPES= (LIGHTWEIGHT_METADATA_TYPE_RAW, LIGHTWEIGHT_METADATA_TYPE_FEATHER)
LIGHTWEIGHT_METADATA_TYPE_CHOICES= (
(LIGHTWEIGHT_METADATA_TYPE_RAW, _("(LARGE) Save raw metadata")),
(LIGHTWEIGHT_METADATA_TYPE_FEATHER, _("(TINY) if the capacity is large, Treeshake it event if it is in use")),
)
lightweight_metadata=models.CharField(
_('lightweight metadata'),
max_length=20,
default=LIGHTWEIGHT_METADATA_TYPE_RAW,
choices=LIGHTWEIGHT_METADATA_TYPE_CHOICES,
help_text=_('Lightweight metadata')
)
If you delete formats and thumbnails from the metadata then thumbnails can't be downloaded and downloading media won't work as the media format can't be evaluated. This occurs on model save at the moment to determine if an item can be downloaded when there's a match for the requested format. While you may want to ignore the thumbnails, currently the formats (which get refreshed as and when the metadata is updated) are required.
Over the years I've had a good look at the large metadata myself, probably the most sensible may be to move it out of the database and store them as msgpack'd blobs on disk in the config dir or similar. There isn't much you can truncate from the metadata without losing functionality. You can save 5-10% but that never really seemed that worth it.
Version
Abstract
Detail
AS-IS
TO-BE
Sample Source
Sample View
Add/Edit source
Media item view (one of the media details)
The text was updated successfully, but these errors were encountered: