Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error: Source file could not be loaded #440

Open
rustyscottweber opened this issue Jan 18, 2024 · 7 comments
Open

Error: Source file could not be loaded #440

rustyscottweber opened this issue Jan 18, 2024 · 7 comments
Labels
0. to triage bug needs info

Comments

@rustyscottweber
Copy link

rustyscottweber commented Jan 18, 2024 •
edited

While running nohup /root/occ preview:generate-all &

2024-01-18T10:52:30+00:00 Scanning folder /Benjamin.Weber/files
2024-01-18T10:52:30+00:00 Scanning folder /Benjamin.Weber/files/Documents
2024-01-18T10:52:30+00:00 Scanning folder /Benjamin.Weber/files/Dungeons&Dragons
2024-01-18T10:52:30+00:00 Scanning folder /Benjamin.Weber/files/Photos
2024-01-18T10:52:30+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5
2024-01-18T10:52:30+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Camera
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/DCIM
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Documents
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Download
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Facebook
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Lumii
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Messenger
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/PhotoScan
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/PhotosEditor
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Pictures
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Raw
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Screenshots
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/ToonMe
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Voice
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Photos/Pixel5/Voila
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Templates
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Weberbia
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Weberbia/Important Recordings
2024-01-18T10:52:34+00:00 Scanning folder /Benjamin.Weber/files/Weberbia/Lake Powell 2023
2024-01-18T10:52:34+00:00 Scanning folder /Bryce.Bollinger/files
2024-01-18T10:52:34+00:00 Scanning folder /Bryce.Bollinger/files/Documents
2024-01-18T10:52:34+00:00 Scanning folder /Bryce.Bollinger/files/Photos
2024-01-18T10:52:34+00:00 Scanning folder /Bryce.Bollinger/files/Templates
2024-01-18T10:52:34+00:00 Scanning folder /Bryce.Bollinger/files/Weberbia
2024-01-18T10:52:34+00:00 Scanning folder /Bryce.Bollinger/files/Weberbia/Important Recordings
2024-01-18T10:52:34+00:00 Scanning folder /Bryce.Bollinger/files/Weberbia/Lake Powell 2023
2024-01-18T10:52:35+00:00 Scanning folder /Finau.Furniss/files
2024-01-18T10:52:35+00:00 Scanning folder /Finau.Furniss/files/Documents
2024-01-18T10:52:35+00:00 Scanning folder /Finau.Furniss/files/Photos
2024-01-18T10:52:35+00:00 Scanning folder /Finau.Furniss/files/Templates
2024-01-18T10:52:35+00:00 Scanning folder /Garth.Weber/files
2024-01-18T10:52:35+00:00 Scanning folder /Garth.Weber/files/Documents
2024-01-18T10:52:35+00:00 Scanning folder /Garth.Weber/files/Documents/GCW Family Documents
2024-01-18T10:52:35+00:00 Scanning folder /Garth.Weber/files/Documents/GCW Family Documents/MISSIONARY YEARS
Error: source file could not be loaded
Error: source file could not be loaded
Error: source file could not be loaded
@rustyscottweber
Copy link
Author

Looks like this also occurs on

nohup ./occ preview:pre-generate &
[1] 7301
nohup: ignoring input and appending output to 'nohup.out'
root@cloud:~# tail -f ./nohup.out 
/usr/share/nginx/nextcloud ~
Error: source file could not be loaded

@rustyscottweber
Copy link
Author

Looking at the verbose output of either command shows a more interesting tail:

2024-01-19T01:10:39+00:00 Generating previews for /Garth.Weber/files/Documents/GCW Family Documents/MISSIONARY YEARS/021003.msg
Error: source file could not be loaded
2024-01-19T01:14:30+00:00 Generating previews for /Garth.Weber/files/Documents/GCW Family Documents/MISSIONARY YEARS/030206.msg
Error: source file could not be loaded

So, while attempting to process .msg files, which are plain text emails, the preview generator takes about 2-5 minutes before returning an error that the source file could not be loaded. At 2-5 minutes a file, when there thousands of them, the scan will essentially never finish running.

@rustyscottweber
Copy link
Author

rustyscottweber commented Jan 23, 2024 •
edited

It would appear that, if left to it's own designs, this bug will cause enough preview generation processes to pile up that it will eventually crash the server.

root@cloud:~# ps aux | grep -i preview:generate-all
www-data    3518  0.0  0.0   2892   912 ?        Ss   Jan19   0:00 /bin/sh -c /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
www-data    3520  0.0  0.3 483780 39552 ?        S    Jan19   3:45 /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
root       15503  0.0  0.0   7372  2356 ?        S    Jan19   0:00 bash /root/occ preview:generate-all -vvv
root       15504  0.0  0.0  11668  4576 ?        S    Jan19   0:00 sudo -u www-data php occ preview:generate-all -vvv
root       15505  0.0  0.0  11668   104 pts/1    Ss+  Jan19   0:00 sudo -u www-data php occ preview:generate-all -vvv
www-data   15506  0.0  1.1 479772 144292 pts/1   S    Jan19   3:54 php occ preview:generate-all -vvv
www-data   25687  0.0  0.0   2892  1000 ?        Ss   Jan20   0:00 /bin/sh -c /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
www-data   25688  0.0  1.3 478860 160828 ?       S    Jan20   4:13 /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
www-data   38955  0.0  0.0   2892   980 ?        Ss   Jan21   0:00 /bin/sh -c /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
www-data   38957  0.1  1.3 478992 165524 ?       S    Jan21   4:02 /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
www-data   53123  0.0  0.0   2892  1000 ?        Ss   Jan22   0:00 /bin/sh -c /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
www-data   53126  0.0  1.1 455924 139964 ?       S    Jan22   0:03 /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
www-data   66849  0.0  0.0   2892  1004 ?        Ss   03:30   0:00 /bin/sh -c /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
www-data   66852  0.0  1.1 455924 140048 ?       S    03:30   0:10 /usr/bin/php /usr/share/nginx/nextcloud/occ preview:generate-all
root       73334  0.0  0.0   6624  2404 pts/2    S+   14:06   0:00 grep --color=auto -i preview:generate-all

All of these separate running cron jobs cause PHP to time out meaning that the server cannot field requests to users.

@rustyscottweber
Copy link
Author

It turns out that part of the errors here are that I had an outdated cron job running preview:generate-all in stead of preview:pre-generate. I changed my cron job, which should stop the server from eventually crashing. However, the failure to generate certain previews still prevails making the pre-generate cron job run far longer than it need to.

@st3iny
Copy link
Member

st3iny commented Feb 6, 2024

The problem might be that the failed previews are tried to be generated again and again.

I have no experience with the preview provider for msg files and can't promise anything. You could disable it for now if you are not depending on it. I imagine having previews for email files is not critical.

@rustyscottweber
Copy link
Author

Indeed, part of the issue is that previews are attempted again and again and turning off previews for . Msg files would solve that. How does one disable specific previews?

@joshtrichards
Copy link
Member

Previews are managed through a combination of enabledPreviewProviders and, in some cases, some apps register preview providers themselves.

We don't do any .msg handling by default (at least not intentionally).

What apps do you have installed (occ app:list)?

Anything like the following?

https://github.com/nextcloud/files_emailviewer

(Though I don't see a preview provider in it).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
0. to triage bug needs info
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@st3iny @joshtrichards @rustyscottweber