vips dzsave has "Too many files open" #3817
Replies: 34 comments 34 replies
-
Hi @marcosocamargo, It should work. I see:
So it needed fewer than 20 open files for a 30,000 x 26,319 pixel JPEG. This is libvips 8.15.1 on ubuntu 23.10. What libvips version, what platform, how large is the file you are processing, etc. |
Beta Was this translation helpful? Give feedback.
-
I think it is related to renaming or moving the file to _files: It is a smalll PNG -rw-r--r-- 1 root root 55672 Jan 15 17:37 kitten.png fileroot@linux-web-server:/home/info/mount-folder# lsb_release -a root@linux-web-server:/home/info/mount-folder# vips -v This is Debian: |
Beta Was this translation helpful? Give feedback.
-
root@linux-web-server:/home/info/mount-folder# vips dzsave kitten.png x --vips-progress |
Beta Was this translation helpful? Give feedback.
-
The folder where the file is located was a monted folder with gcsfuser (google storage). When I move the file to just a regular directory, it works. root@linux-web-server:/home/info# vips dzsave kitten.png x --vips-progress |
Beta Was this translation helpful? Give feedback.
-
It is a small file --384 x 251 pixels. This is just a test. How do know it is thousands of files? |
Beta Was this translation helpful? Give feedback.
-
I have this error when opening image with seadragon ‘Access-Control-Allow-Origin’ is missing. Can Seadragon provide this in the header? |
Beta Was this translation helpful? Give feedback.
-
Is there any way I can find out what the metadata is before I tile it and then I can do the rotation using vips dzsave? |
Beta Was this translation helpful? Give feedback.
-
How can I add quality = 50% and tile-size 510? I tried this command but it doesn't work |
Beta Was this translation helpful? Give feedback.
-
Basically I would like to autorotate using the EXIF tag rotate, zip file output, quality of jpeg 50% and tile size 510. How do I tell the tool to do that? Can I do that all in one command? |
Beta Was this translation helpful? Give feedback.
-
It worked. I really appreciate your help! |
Beta Was this translation helpful? Give feedback.
-
How do I prevent the tool from generating very small tiles of the document? These views are so small that they aren't usable. For example: https://storage.googleapis.com/recpres/y/y_files/4/0_0.jpg |
Beta Was this translation helpful? Give feedback.
-
Thanks again! Do you unzip the files or use it as is? Also, is there a reason why you omitted the tile size of 510? |
Beta Was this translation helpful? Give feedback.
-
I issued this command: info@linux-web-server [~]# ls -l y* y_files: |
Beta Was this translation helpful? Give feedback.
-
How do I install the latest version?
Thanks,
-Marcos
…On Sat, Jan 20, 2024 at 9:51 AM John Cupitt ***@***.***> wrote:
Ah you're on 8.10, I remember. It's changed a bit since then.
—
Reply to this email directly, view it on GitHub
<#3817 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BFLR7JMRZMJ4G6UVE64WZUTYPP73DAVCNFSM6AAAAABB3UPBQGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DCOJSGA4DM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
Something like:
|
Beta Was this translation helpful? Give feedback.
-
Thanks again!
The .dzi gets created but not all the file inside the directory
IMG_0120_files.
-Marcos
…On Sun, Mar 24, 2024 at 8:44 PM John Cupitt ***@***.***> wrote:
Something like:
.\vips.exe dzsave "C:\Users\James Camargo\Downloads\IMG_0120.jpg[autorot]" "C:\Users\James Camargo\Downloads\IMG_0120" --suffix .jpg[Q=50] --tile-size 510
—
Reply to this email directly, view it on GitHub
<#3817 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BFLR7JNV7LTYYVM63MELUW3YZ6MRRAVCNFSM6AAAAABB3UPBQGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DQOJXGEYTQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
I am sorry. They were created.
-Marcos
On Mon, Mar 25, 2024 at 7:57 AM Marcos Camargo ***@***.***>
wrote:
… Thanks again!
The .dzi gets created but not all the file inside the directory
IMG_0120_files.
-Marcos
On Sun, Mar 24, 2024 at 8:44 PM John Cupitt ***@***.***>
wrote:
> Something like:
>
> .\vips.exe dzsave "C:\Users\James Camargo\Downloads\IMG_0120.jpg[autorot]" "C:\Users\James Camargo\Downloads\IMG_0120" --suffix .jpg[Q=50] --tile-size 510
>
> —
> Reply to this email directly, view it on GitHub
> <#3817 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BFLR7JNV7LTYYVM63MELUW3YZ6MRRAVCNFSM6AAAAABB3UPBQGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DQOJXGEYTQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
Beta Was this translation helpful? Give feedback.
-
Thanks again for your help. How do I make sure that every exif data is
removed from the resulting files?
-Marcos Camargo
…On Sun, Mar 24, 2024 at 8:44 PM John Cupitt ***@***.***> wrote:
Something like:
.\vips.exe dzsave "C:\Users\James Camargo\Downloads\IMG_0120.jpg[autorot]" "C:\Users\James Camargo\Downloads\IMG_0120" --suffix .jpg[Q=50] --tile-size 510
—
Reply to this email directly, view it on GitHub
<#3817 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BFLR7JNV7LTYYVM63MELUW3YZ6MRRAVCNFSM6AAAAABB3UPBQGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DQOJXGEYTQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
I tried the command it seems that it doesn't autorotate.
This is the command:
vips.exe dzsave "G:\Records Preservation\Brasil\São Paulo\São Paulo\Arquivo
Metropolitano Dom Duarte Leopoldo e Silva\Processos Gerais Antigos\Caixa
004\IMG_004_0709.JPG[autorot]" "E:\Brasil\São Paulo\São Paulo\Arquivo
Metropolitano Dom Duarte Leopoldo e Silva\Processos Gerais Antigos\Caixa
004\IMG_004_0709" --suffix .jpg[Q=50] --tile-size 510 --overlap 1
Thanks,
-Marcos
…On Sun, Mar 24, 2024 at 8:44 PM John Cupitt ***@***.***> wrote:
Something like:
.\vips.exe dzsave "C:\Users\James Camargo\Downloads\IMG_0120.jpg[autorot]" "C:\Users\James Camargo\Downloads\IMG_0120" --suffix .jpg[Q=50] --tile-size 510
—
Reply to this email directly, view it on GitHub
<#3817 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BFLR7JNV7LTYYVM63MELUW3YZ6MRRAVCNFSM6AAAAABB3UPBQGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DQOJXGEYTQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
It works if you use autorotate instead of autorot
Thanks,
-Marcos
On Tue, Mar 26, 2024 at 11:30 AM Marcos Camargo ***@***.***>
wrote:
… I tried the command it seems that it doesn't autorotate.
This is the command:
vips.exe dzsave "G:\Records Preservation\Brasil\São Paulo\São
Paulo\Arquivo Metropolitano Dom Duarte Leopoldo e Silva\Processos Gerais
Antigos\Caixa 004\IMG_004_0709.JPG[autorot]" "E:\Brasil\São Paulo\São
Paulo\Arquivo Metropolitano Dom Duarte Leopoldo e Silva\Processos Gerais
Antigos\Caixa 004\IMG_004_0709" --suffix .jpg[Q=50] --tile-size 510
--overlap 1
Thanks,
-Marcos
On Sun, Mar 24, 2024 at 8:44 PM John Cupitt ***@***.***>
wrote:
> Something like:
>
> .\vips.exe dzsave "C:\Users\James Camargo\Downloads\IMG_0120.jpg[autorot]" "C:\Users\James Camargo\Downloads\IMG_0120" --suffix .jpg[Q=50] --tile-size 510
>
> —
> Reply to this email directly, view it on GitHub
> <#3817 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BFLR7JNV7LTYYVM63MELUW3YZ6MRRAVCNFSM6AAAAABB3UPBQGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DQOJXGEYTQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
Beta Was this translation helpful? Give feedback.
-
I am using openseadragon which doesn't load anything smaller than a tile. I used the onetile version of libvips which generates the necessary tiles for OSD but the numbering is incorrect. For example, if an image requires 13 levels down to 1 pixel, then with onetile option there are 5 levels from numbers 0-4 but OSD expects from 9-13. Is it possible to name the levels starting with highest levels 13 down to 9 instead of 0 to 4? Thanks! |
Beta Was this translation helpful? Give feedback.
-
I tested by generating the pyramid with the option onetile which generates
everything down to one tile. Then I renamed the directories in the _files
directory to be the name as if they were generated with onepixel option.
This works. Is there a way to add an option to onepixel to stop at one tile
level? That would save a lot of image generation especially for those that
use OSD? How do I determine which one to delete since every image is
different? How do you determine how many levels you are going to have?
Thanks for your help.
-Marcos
…On Sat, May 4, 2024 at 10:39 AM John Cupitt ***@***.***> wrote:
Hi @marcosocamargo <https://github.com/marcosocamargo>, onetile is not a
valid deepzoom image, that's there for other layouts, like google maps.
It's just an implementation detail that openseadragon won't bother loading
smaller tiles.
I would write a regular deepzoom image (ie. no onetile option), then
delete the smaller levels if you're sure you won't need them.
—
Reply to this email directly, view it on GitHub
<#3817 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BFLR7JNRJIRJPG4JXEJIRPLZAUMOTAVCNFSM6AAAAABB3UPBQGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGMJVGQYTO>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
Thanks for the input. Since I will be processing 1 million images it is
important not to generate anything that I don't need.
So I decided to write a script that generates down to 1 tile and then
renames the files based on going down to 1 pixel.
Just want to confirm that's how you calculate the maximum number of levels:
ceiling( ln (Max(height, width))/log 2
Thanks,
-Marcos
…On Sun, May 5, 2024 at 3:19 AM John Cupitt ***@***.***> wrote:
I don't think dzsave should support non-standard DZI images, and in any
case the savings are tiny. For speed, dzsave generates several 1,000 tiles
a second, so saving 7 tiles will not be measurably faster. The space saving
will be only a few kb.
You can work out which directories OSD won't read by searching up from
level 0 for the first level containing more than one tile, then backing off
one.
For example:
$ vips dzsave nina.jpg x
$ ls x_files/*
vips-properties.xml
0:
0_0.jpeg
1:
0_0.jpeg
2:
0_0.jpeg
3:
0_0.jpeg
4:
0_0.jpeg
5:
0_0.jpeg
6:
0_0.jpeg
7:
0_0.jpeg
8:
0_0.jpeg
9:
0_0.jpeg 1_0.jpeg
So you can delete 0-7. It'll almost always be 0-7 for 256x256 tiles (ie. 2
^ (8 - 1)), though maybe there are some strange edge cases where it'll be a
different value.
—
Reply to this email directly, view it on GitHub
<#3817 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BFLR7JOZJ2QNSHGMGSRCISLZAYBT3AVCNFSM6AAAAABB3UPBQGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGMJZGEYTE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
Thanks for the help again. I issued this command with has UTF8 characters:
vips.exe dzsave "E:\Brasil\São Paulo\Jundiaí\Nossa Senhora do
Desterro\Óbitos 1744-1787\IMG_007_0157_IRF.JPG" "E:\Brasil\São
Paulo\Jundiaí\Nossa Senhora do Desterro\Óbitos 1744-1787\IMG_007_0157"
--suffix .JPG[Q=80] --tile-size 512 --overlap 1 --depth onetile
And I get this error because the diacritics get messed (PowerShell)
VipsForeignLoad: file "E:\Brasil\SA£o Paulo\JundiaA-\Nossa Senhora do
Desterro\A"bitos 1744-1787\IMG_007_0157_IRF.JPG" does not exist
What kind of encoding does libvips accept? That might be the problem.
…-Marcos
On Mon, May 6, 2024 at 4:00 AM John Cupitt ***@***.***> wrote:
It shrinks the base level by *2, rounding up each time, then stops once
the image fits within one tile. So yes I think that's the expression you
quoted.
—
Reply to this email directly, view it on GitHub
<#3817 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BFLR7JKCJ5H3UCZA367RLJDZA5PGVAVCNFSM6AAAAABB3UPBQGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGMRXHA2TI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
I installed 8.15.2 and it is working now. Thanks for the suggestion.
Is there a thumbnail generator for VIPS?
-Marcos
…On Wed, May 8, 2024 at 1:37 AM John Cupitt ***@***.***> wrote:
I tried on win10 and I see:
***@***.*** MINGW64 /f/Pictures
$ ../vips-dev-8.15/bin/vips.exe text x.png 😁 --dpi 300 --rgba
To make:
x.png (view on web)
<https://github.com/libvips/libvips/assets/580843/525e4b49-260e-4a04-8ec9-1f00f79e0602>
Filenames are different -- windows usually uses latin1 for filename
encodings, though not always. libvips will try to convert windows filenames
to utf8 automatically, but I think it may sometimes fail.
You could try the latest libvips, it might be better.
—
Reply to this email directly, view it on GitHub
<#3817 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BFLR7JM4Q73JPKKPFYCPKM3ZBHP3PAVCNFSM6AAAAABB3UPBQGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGNJRG43DM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
I am reading about thumbnails here
https://www.libvips.org/API/current/Using-vipsthumbnail.html"
If the image has an EXIF tag to rotate it, will the thumbnail rotate it
too? This is my desired outcome, that is, that it rotates the thumbnail
according to the EXIF tag.
Thanks,
-Marcos
On Wed, May 8, 2024 at 9:52 AM Marcos Camargo ***@***.***>
wrote:
… I installed 8.15.2 and it is working now. Thanks for the suggestion.
Is there a thumbnail generator for VIPS?
-Marcos
On Wed, May 8, 2024 at 1:37 AM John Cupitt ***@***.***>
wrote:
> I tried on win10 and I see:
>
> ***@***.*** MINGW64 /f/Pictures
> $ ../vips-dev-8.15/bin/vips.exe text x.png 😁 --dpi 300 --rgba
>
> To make:
>
> x.png (view on web)
> <https://github.com/libvips/libvips/assets/580843/525e4b49-260e-4a04-8ec9-1f00f79e0602>
>
> Filenames are different -- windows usually uses latin1 for filename
> encodings, though not always. libvips will try to convert windows filenames
> to utf8 automatically, but I think it may sometimes fail.
>
> You could try the latest libvips, it might be better.
>
> —
> Reply to this email directly, view it on GitHub
> <#3817 (reply in thread)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BFLR7JM4Q73JPKKPFYCPKM3ZBHP3PAVCNFSM6AAAAABB3UPBQGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGNJRG43DM>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
Beta Was this translation helpful? Give feedback.
-
vipsthumbnails doesn't seem to support UTF8. I have the latest version.
Thanks,
-Marcos
…On Wed, May 8, 2024 at 12:25 PM John Cupitt ***@***.***> wrote:
Yes, it will spin thumbnails upright.
—
Reply to this email directly, view it on GitHub
<#3817 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BFLR7JPSJD5KQECZ7PMMDTDZBJ32BAVCNFSM6AAAAABB3UPBQGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGNJZGQYTI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
It seems that the input file name for both dzsave and thumbnails doesn't
take just a UTF8 defined in a variable but accepts it if I put it through
Get-Item in PowerShell.
Should it take both? I am using Irfanview in the same script and it takes
both ways.
Thanks,
-Marcos
On Wed, May 8, 2024 at 8:53 PM Marcos Camargo ***@***.***>
wrote:
… vipsthumbnails doesn't seem to support UTF8. I have the latest version.
Thanks,
-Marcos
On Wed, May 8, 2024 at 12:25 PM John Cupitt ***@***.***>
wrote:
> Yes, it will spin thumbnails upright.
>
> —
> Reply to this email directly, view it on GitHub
> <#3817 (reply in thread)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BFLR7JPSJD5KQECZ7PMMDTDZBJ32BAVCNFSM6AAAAABB3UPBQGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGNJZGQYTI>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
Beta Was this translation helpful? Give feedback.
-
vipsthumbnails doesn't work at with whether passing a variable with
diacritics or putting the variable through Get-Item.
On Wed, May 8, 2024 at 9:30 PM Marcos Camargo ***@***.***>
wrote:
… It seems that the input file name for both dzsave and thumbnails doesn't
take just a UTF8 defined in a variable but accepts it if I put it through
Get-Item in PowerShell.
Should it take both? I am using Irfanview in the same script and it takes
both ways.
Thanks,
-Marcos
On Wed, May 8, 2024 at 8:53 PM Marcos Camargo ***@***.***>
wrote:
> vipsthumbnails doesn't seem to support UTF8. I have the latest version.
> Thanks,
> -Marcos
>
> On Wed, May 8, 2024 at 12:25 PM John Cupitt ***@***.***>
> wrote:
>
>> Yes, it will spin thumbnails upright.
>>
>> —
>> Reply to this email directly, view it on GitHub
>> <#3817 (reply in thread)>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/BFLR7JPSJD5KQECZ7PMMDTDZBJ32BAVCNFSM6AAAAABB3UPBQGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGNJZGQYTI>
>> .
>> You are receiving this because you were mentioned.Message ID:
>> ***@***.***>
>>
>
|
Beta Was this translation helpful? Give feedback.
-
I am using vips dzsave and I am getting this error:
rename: unable to rename file "./kitten.jpg-WK3HH2/kitten.jpg_files" as "./kitten.jpg_files", Too many open files
I increased the ulimit to 32768 and I still get this error.
How much do I have to up the ulimit to make it work?
Is this normal or is it a bug.
Thanks,
-Marcos
Beta Was this translation helpful? Give feedback.
All reactions