preview directory in appdata_xxxx has grown over 120GB of Total size

[Githopp192]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

preview directory in appdata_xxxx has grown over 120GB of Total size

Steps to reproduce

1. check the size on RHEL 8.9 with du -hs /appdata_xxxx/preview
2. size is over 120GB

Expected behavior

There should be a mechanism where the old, obsolete and/or long unused previews are automatically deleted.

An admin should also be able to perform such an operation manually.

QUESTION:

1 - Is there such an automatic mechanism ?
2 - Is there a manual mechanism ?

I know, i could remove the oldest previews in the preview folder, run a fresh "occ files:scan-app-data" afterwards.

But i think there must be a more proper, stable way to do that !?

Installation method

Community Web installer on a VPS or web space

Nextcloud Server version

27

Operating system

RHEL/CentOS

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Updated from a MINOR version (ex. 22.1 to 22.2)

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

not relevant here

NC version":"27.1.9.1"

List of activated Apps

No response

Nextcloud Signing status

no errors

Nextcloud Logs

No response

Additional info

No response

[joshtrichards]

1 - Is there such an automatic mechanism ?

Yes. It runs hourly: https://github.com/nextcloud/server/blob/master/lib/private/Preview/BackgroundCleanupJob.php

Are you getting any errors in your logs like perhaps was reported in #42540?

2 - Is there a manual mechanism ?

There is no single command that does this currently. Though this would be a reasonable enhancement.

[st3iny]

The job mentioned above will only clean previews of deleted files. There is no mechanism for deleting previews of old files.

[Githopp192]

@joshtrichards - no - as far i see i do not have those or some of those errors mentioned in #42540

i analysed how many of directories i've got in the preview folder for every year:

[root@server ~]# grep ' 2010' /tmp/file_list_of_preview_dir|wc -l
0
[root@server ~]# grep ' 2017' /tmp/file_list_of_preview_dir|wc -l
357
[root@server ~]# grep ' 2018' /tmp/file_list_of_preview_dir|wc -l
925
[root@server ~]# grep ' 2019' /tmp/file_list_of_preview_dir|wc -l
8382
[root@server ~]# grep ' 2020' /tmp/file_list_of_preview_dir|wc -l
4292
[root@server ~]# grep ' 2021' /tmp/file_list_of_preview_dir|wc -l
345
[root@server ~]# grep ' 2022' /tmp/file_list_of_preview_dir|wc -l
224
[root@server ~]# grep ' 2023' /tmp/file_list_of_preview_dir|wc -l
248

so - i suppose the issue began on 2019/2020 within an earlier version of nextcloud

I did count all image files [jpg/jpeg/png/gif/bmp/tiff/svg] for all users :

Totally 313 GB of used space for all images - Nextcloud Preview will use 38% of space of them !
So, it is assumed, that the cleanup operation will not work as expected.

That means a manual (controlled by the cloud admin) mechanism for cleanup preview space is urgently needed.

[kesselb]

i suppose the issue began on 2019/2020 within an earlier version of nextcloud

Please elaborate on the issue, what did you observe, what do you think is not right, what is an obsolete preview?

The expected behavior is: Previews are generated for supported file types. If the source file is deleted, the previews are removed as well.

We don't track the last access for files, and therefore cannot know when a preview was accessed the last time.

[Githopp192]

thanks Daniel for those specific questions.

obsolete - preview files, where no co-existing image is existing
observe - Ratio between the size of the preview folder (121GB) and the actual size of all images (313GB)

Yes, that's my view of things too: "If the source file is deleted, the previews are removed as well."

It would be nice to have a way to find out if files exist for existing previews, something like this:

• create a list of image files
• check if the corresponding image file (still) exists for the preview
• if no, delete preview

It would also be good, if the cloud admin could set the maximum size for the preview folder (so that the system automatically deletes the previews, that have not been used for the longest time - similar to what happens with the recycle bin).

Do you need a feature request for this or do you think this is “useless” :-)

[solracsf]

Please also note that the ratio is not 1 preview per image.
There are quite a few preview sizes generated per file (64x64, 356x356, 1024x768, and so on).
Also, not only image are taken into account (MP3, Vidos, PDF, and other too can count).

So, 38% is not a very big number IMO.

See if needed:

server/config/config.sample.php

Lines 1195 to 1347 in 54afea4

	/**
	* Previews
	*
	* Nextcloud supports previews of image files, the covers of MP3 files, and text
	* files. These options control enabling and disabling previews, and thumbnail
	* size.
	*/
	/**
	* By default, Nextcloud can generate previews for the following filetypes:
	*
	* - Image files
	* - Covers of MP3 files
	* - Text documents
	*
	* Valid values are ``true``, to enable previews, or
	* ``false``, to disable previews
	*
	* Defaults to ``true``
	*/
	'enable_previews' => true,
	/**
	* Number of all preview requests being processed concurrently,
	* including previews that need to be newly generated, and those that have
	* been generated.
	*
	* This should be greater than 'preview_concurrency_new'.
	* If unspecified, defaults to twice the value of 'preview_concurrency_new'.
	*/
	'preview_concurrency_all' => 8,
	/**
	* Number of new previews that are being concurrently generated.
	*
	* Depending on the max preview size set by 'preview_max_x' and 'preview_max_y',
	* the generation process can consume considerable CPU and memory resources.
	* It's recommended to limit this to be no greater than the number of CPU cores.
	* If unspecified, defaults to the number of CPU cores, or 4 if that cannot
	* be determined.
	*/
	'preview_concurrency_new' => 4,
	/**
	* The maximum width, in pixels, of a preview. A value of ``null`` means there
	* is no limit.
	*
	* Defaults to ``4096``
	*/
	'preview_max_x' => 4096,
	/**
	* The maximum height, in pixels, of a preview. A value of ``null`` means there
	* is no limit.
	*
	* Defaults to ``4096``
	*/
	'preview_max_y' => 4096,
	/**
	* Max file size for generating image previews with imagegd (default behavior).
	* If the image is bigger, it'll try other preview generators, but will most
	* likely either show the default mimetype icon or not display the image at all.
	* Set to ``-1`` for no limit and try to generate image previews on all file sizes.
	*
	* Defaults to ``50`` megabytes
	*/
	'preview_max_filesize_image' => 50,
	/**
	* max memory for generating image previews with imagegd (default behavior)
	* Reads the image dimensions from the header and assumes 32 bits per pixel.
	* If creating the image would allocate more memory, preview generation will
	* be disabled and the default mimetype icon is shown. Set to -1 for no limit.
	*
	* Defaults to ``256`` megabytes
	*/
	'preview_max_memory' => 256,
	/**
	* custom path for LibreOffice/OpenOffice binary
	*
	*
	* Defaults to ``''`` (empty string)
	*/
	'preview_libreoffice_path' => '/usr/bin/libreoffice',
	/**
	* custom path for ffmpeg binary
	*
	* Defaults to ``null`` and falls back to searching ``avconv`` and ``ffmpeg`` in the configured ``PATH`` environment
	*/
	'preview_ffmpeg_path' => '/usr/bin/ffmpeg',
	/**
	* Set the URL of the Imaginary service to send image previews to.
	* Also requires the ``OC\Preview\Imaginary`` provider to be enabled.
	*
	* See https://github.com/h2non/imaginary
	*/
	'preview_imaginary_url' => 'http://previews_hpb:8088/',
	/**
	* If you want set a api key for imaginary.
	*/
	'preview_imaginary_key' => 'secret',
	/**
	* Only register providers that have been explicitly enabled
	*
	* The following providers are disabled by default due to performance or privacy
	* concerns:
	*
	* - ``OC\Preview\Font``
	* - ``OC\Preview\HEIC``
	* - ``OC\Preview\Illustrator``
	* - ``OC\Preview\Movie``
	* - ``OC\Preview\MSOffice2003``
	* - ``OC\Preview\MSOffice2007``
	* - ``OC\Preview\MSOfficeDoc``
	* - ``OC\Preview\PDF``
	* - ``OC\Preview\Photoshop``
	* - ``OC\Preview\Postscript``
	* - ``OC\Preview\StarOffice``
	* - ``OC\Preview\SVG``
	* - ``OC\Preview\TIFF``
	* - ``OC\Preview\EMF``
	*
	*
	* Defaults to the following providers:
	*
	* - ``OC\Preview\BMP``
	* - ``OC\Preview\GIF``
	* - ``OC\Preview\JPEG``
	* - ``OC\Preview\Krita``
	* - ``OC\Preview\MarkDown``
	* - ``OC\Preview\MP3``
	* - ``OC\Preview\OpenDocument``
	* - ``OC\Preview\PNG``
	* - ``OC\Preview\TXT``
	* - ``OC\Preview\XBitmap``
	*/
	'enabledPreviewProviders' => [
	'OC\Preview\BMP',
	'OC\Preview\GIF',
	'OC\Preview\JPEG',
	'OC\Preview\Krita',
	'OC\Preview\MarkDown',
	'OC\Preview\MP3',
	'OC\Preview\OpenDocument',
	'OC\Preview\PNG',
	'OC\Preview\TXT',
	'OC\Preview\XBitmap',
	],

And https://github.com/nextcloud/previewgenerator (specially the "I don't want to generate all the preview sizes" part).

[Githopp192]

thx @solracsf - yes, i'm aware about the ratio (not 1 preview per image).

May this could come in place to export the cloud preview to external (interim) storage ==>

/** 
  * Set the URL of the Imaginary service to send image previews to. 
  * Also requires the ``OC\Preview\Imaginary`` provider to be enabled. 
  * 
  * See https://github.com/h2non/imaginary 
  */ 
 'preview_imaginary_url' => 'http://previews_hpb:8088/', 
  
 /** 

[st3iny]

No, this will only replace the preview provider not the storage. Imaginary is an external service that offers great performance and a wide variety of supported preview formats.

It will not store your previews. Instead, it will process them on the fly and send them back to Nextcloud to be stored.

[Githopp192]

Thx @st3iny / Richard for this explanation :-)