Large NDPI file fix #602
Open
Large NDPI file fix #602
+61
−55
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DCO signed off ✔️All commits have been signed off. You have certified to the terms of the Developer Certificate of Origin, version 1.1. In particular, you certify that this contribution has not been developed using information obtained under a non-disclosure agreement or other license terms that forbid you from contributing it under the GNU Lesser General Public License, version 2.1. |
Addresses issue openslide#174 (and possibly others), regarding the reading of large NDPI files. This issue stems from the fact that classical TIFF (and by extension, NDPI) images only support up to 32 bit values for tagged metadata. However, due to the nature of whole slide image data, it isn't uncommon for the size of an NDPI to exceed the 32 bit range. This means that key metadata, such as the byte positions of image layers or their size in bytes, may be too large to store in a traditional TIFF IFD entry. Currently OpenSlide relies on heuristics to determine the high bits of 64 bit addresses, which fail in some cases. However, this is unnecessary, as NDPI actually stores the high bits of the offset/value of each tag in 4 byte blocks immediately after the end of the IFD. This fix modifies openslide-decode-tifflike.c to append these extra 4 bytes to each IFD entry's value/offset and, if necessary, modifies its type to LONG8. This fix also modifies openslide-vendor-hamamatsu.c to construct correct restart marker addresses. Currently only the values in TIFF tag 65426 are used, which are only the lower 32 bits of each address. High bits are stored in TIFF tag 65432, so these are now appended before mcu_starts are calculated. Signed-off-by: Nick Trahearn <n.a.trahearn@gmail.com> Signed-off-by: Benjamin Gilbert <bgilbert@cs.cmu.edu>
|
Looks good, thank you! I've rebased, updated to the current internal API, and fixed up some formatting. I have a few more cleanup commits coming, and will fix the tests. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addresses issue #174 (and possibly others), regarding the reading of large NDPI files. This issue stems from the fact that classical TIFF (and by extension, NDPI) images only support up to 32 bit values for tagged metadata. However, due to the nature of whole slide image data, it isn't uncommon for the size of an NDPI to exceed the 32 bit range. This means that key metadata, such as the byte positions of image layers or their size in bytes, may be too large to store in a traditional TIFF IFD entry.
Currently OpenSlide relies on heuristics to determine the high bits of 64 bit addresses, which fail in some cases. However, this is unnecessary, as NDPI actually stores the high bits of the offset/value of each tag in 4 byte blocks immediately after the end of the IFD.
This fix modifies openslide-decode-tifflike.c to append these extra 4 bytes to each IFD entry's value/offset and, if necessary, modifies its type to LONG8.
This fix also modifies openslide-vendor-hamamatsu.c to construct correct restart marker addresses. Currently only the values in TIFF tag 65426 are used, which are only the lower 32 bits of each address. High bits are stored in TIFF tag 65432, so these are now appended before mcu_starts are calculated