"Queued" forever

[iangilman]

had several queued ones, here is a link to one of them , any help would be really amazing , thanks guys:

You can now view it at: https://zoomhub.net/gqoP5/verify/3d7aa1bb-43e1-41ee-8d27-59fc1b31326d

@Akossimon Is this something you recently uploaded? Can you share a link to the original?

[gasi]

@Akossimon: I checked that upload and the error is related to the backend not having enough disk space to convert the image:

wbuffer_write: write failed
unix error: No space left on device

This means the file must be too big to process for some reason.

In general, ZoomHub shows ‘queued’ for any state of the upload that is not ‘success’. This can be confusing when the state is error. I will have to figure out how to improve that but this will be hard until I can secure some revenue for the site.

[Akossimon]

Hi Gasi, thanks so much for replying. may i ask what the max pixel width and pixels hight is for your site? and if a adobe pdf "vector file " could be uploaded ? i tried such a vector pdf but it does not let me. the pdf is merely 18MB, but converting it to a tiff or jpg that would keep the details , would make it a 233'333 pixel wide jpg or tiff. that huge jpg or tif, can be looked at in photoshop on your monitor, but it cannot be saved for its 233'333 pixels only allows a saving in tif and not in jpg . that tiff will be 400-800MB , even when used zip compression pyramid compression in that tiff. this will not upload to your site. The save as jpg is evern more cumbersome, photoshop does not allow more than 65'000 pixels i width, so the original 233'000 pixel wide pic is now shrunk to 65'000 and many letters cannot not be read anymore at 65000 pixels witdt. but in 65'000 pixels maximum width of the file will be a sweet 40mb now . this shrunken jpg uploads just fine. The solution could be to turn on an import /upload of adobe pdf vector files ? but i am not sure if this is technically possible at all . i read incidentally somewhere on your site, that adobe pdf files used to be accepted, but myabe i am wrong. ===== On another note i might have a company in Munich who might be interested in this webpage of yours, i just email details to daniel. is Daniel You ?

[gasi]

may i ask what the max pixel width and pixels hight is for your site?

I can’t specify an exact maximum pixel width/height. There is however a 1024MB maximum file size limitation in place. If you are interested in something higher, we can discuss a paid account.

and if a adobe pdf "vector file " could be uploaded ?

No. The site uses libvips to convert images and it only supports raster (non-vector) images.

===== On another note i might have a company in Munich who might be interested in this webpage of yours, i just email details to daniel. is Daniel You ?

Yes, I’m Daniel. Thanks for the info via email. I haven’t had a chance to reach out to that company yet, but I will try when I can.

[gasi]

@Akossimon

Re: PDF support. Yes, in the original Microsoft days, this was supported, but not currently.

---

Re: TIFF error. I spent quite a bit of time digging into the error you experienced with your TIFF file. While the original error was related to disk space, I increased both memory and ephemeral disk space on AWS Lambda to 10240MB. This resolved the disk space issue. Converting your file did not take more than 3GB of RAM and less than 500MB of disk space.

The latest bottleneck is how quickly we can transfer the tiles from AWS Lambda to S3. This is where the Lambda function times out after 900 seconds (15min) which is the max I can let it run.

I added some logging and found that it can upload approximately 65k tiles of 250k before it runs out of time.

2023-11-29T19:53:42.845Z	5ae9d5e1-2a11-4a11-b552-e319c41057f8	INFO	{
  time: '2023-11-29T19:53:42.845Z',
  message: 'DZI: tile uploaded',
  key: 'content/gqoP5_files/18/115_204.jpg',
  contentType: 'image/jpeg',
  index: 66983,
  numTiles: 251774
}
2023-11-29T19:53:42.884Z 5ae9d5e1-2a11-4a11-b552-e319c41057f8 Task timed out after 900.12 seconds

One further test could be to increase the tile size from 254 to 510 pixels and see if that’ll speed up file transfer.

To do it properly, I would need to purchase the AWS EFS extension and integrate it with Lambda:
https://aws.amazon.com/blogs/compute/choosing-between-aws-lambda-data-storage-options-in-web-apps/

I currently don’t have the budget for that but if we can get some paying customers for ZoomHub it would be doable.

[Akossimon]

@Akossimon

Re: PDF support. Yes, in the original Microsoft days, this was supported, but not currently.

Re: TIFF error. I spent quite a bit of time digging into the error you experienced with your TIFF file. While the original error was related to disk space, I increased both memory and ephemeral disk space on AWS Lambda to 10240MB. This resolved the disk space issue. Converting your file did not take more than 3GB of RAM and less than 500MB of disk space.

The latest bottleneck is how quickly we can transfer the tiles from AWS Lambda to S3. This is where the Lambda function times out after 900 seconds (15min) which is the max I can let it run.

I added some logging and found that it can upload approximately 65k tiles of 250k before it runs out of time.

2023-11-29T19:53:42.845Z	5ae9d5e1-2a11-4a11-b552-e319c41057f8	INFO	{
  time: '2023-11-29T19:53:42.845Z',
  message: 'DZI: tile uploaded',
  key: 'content/gqoP5_files/18/115_204.jpg',
  contentType: 'image/jpeg',
  index: 66983,
  numTiles: 251774
}
2023-11-29T19:53:42.884Z 5ae9d5e1-2a11-4a11-b552-e319c41057f8 Task timed out after 900.12 seconds

One further test could be to increase the tile size from 254 to 510 pixels and see if that’ll speed up file transfer.

To do it properly, I would need to purchase the AWS EFS extension and integrate it with Lambda: https://aws.amazon.com/blogs/compute/choosing-between-aws-lambda-data-storage-options-in-web-apps/

I currently don’t have the budget for that but if we can get some paying customers for ZoomHub it would be doable.

maybe considering making PDF uploads compatible and possible can fix that huge disk space problem: the very same pdf is a mere 36 MB , but its exported jpg or tiff becomes 500mb and larger, if exported at the same resolution and detail of that very 36mb PDF file. Vektor files are 10-50 times smaller. Not sure if this is anything that makes sense here ?

[gasi]

@Akossimon Thanks for the suggestion. The way the viewer (OpenSeadragon) works is that it fetches small pieces (tiles) of a larger image at various resolutions. AFAIK, this is currently always based on raster images, e.g. JPEG or PNG. FWIW, an SVG could be zoomed as well but at 36MB, it might also have to be converted into a pyramid and then tiled. Back in the days when www.seadragon.com (ZoomHub predecessor) supported PDF, it was by rasterizing it into a large image file, so the same problem would persist here. It’s not an engineering but a funding problem.

It would slightly increase monthly costs which I currently mostly pay out of pocket minus a donation we received before. If we get paying customers to sign up, we can expand the feature set of ZoomHub over time.

@iangilman Do you know if anyone has tried pan & zoom on a vector file, e.g. SVG, with OpenSeadragon?

[gasi]

@Akossimon Tada! https://zoomhub.net/gqoP5

I increased the tile size from 254px to 510px and increased the number of parallel uploads from Lambda to S3 and it worked.

Can you check and see if it works like you wanted it to?

[Akossimon]

so cool, i looked at it and it looks so perfect, i will try to also now upload a tif file that is 233'000 pixels wide and 340Mb which is higher resolution and see if it uplaoded or not .... this is very informative here, thank You so much. Grazie mille !

You should contact that german Familytree software company, they have no means yet to show such big familytrees, and i think collaborating with them, by offering them your subscriptions to their customers, would help you and them.

[gasi]

@Akossimon i will try to also now upload a tif file that is 233'000 pixels wide and 340Mb which is higher resolution and see if it uplaoded or not

FYI, reverted the tile size change as I’m not sure I’ll want that for everyone. However, I left the higher S3 upload concurrency limit, so maybe that’s enough.

I can work on a more permanent solution to increase tile size for images that are extra large, like yours.

[Akossimon]

WOW, thats so cool. i do have a newer PDF which is 14 mb with new data corrections in it. i am trying to export it via PREVIEW app to a png in 1200 pixel per inch, since the MAC preview app can do this, and photoshop actually cannot do this, then convert that png with Photoshop to a tiff, using tiff and zip inside the UI option in Photoshop, since the png is now 533 mb. this made the smallest tiff so far, and still retained detail. but it takes a long time to do these 2 steps. i can send you that corrected 14mb PDF, if You like.

[Akossimon]

its stuck : https://zoomhub.net/ZxkYk
233'000 pixels wide tif , around 225 Mb

[iangilman]

I'm not aware of anyone using OpenSeadragon for SVG or other vector graphics. We do have an SVG overlay plug-in which works nicely, but it doesn't have any mechanism for breaking the SVG down into smaller chunks.

I haven't really looked around to see if there are other libraries that do OSD-style zooming with SVGs, but I imagine there would be. In this case, with the family trees, it does seem like an obvious use case for SVG. Since the images are relatively simple (but gigantic) it feels like overkill to rasterize them.

@gasi I actually recommend 510px tiles to everyone these days; since network connections are faster it's actually more efficient to load.