ValueError: cannot reshape array of size

[Neuroforge]

Describe the bug

Cannot open images which can be opened by other dicom viewers. Image is 768 x 1024 x 3.

Error: ValueError: cannot reshape array of size 1179648 into shape (768,1024,3).

Note: 2359296 = 2 x 1179648

Expected behavior

The pydicom should be able to open the image

Steps To Reproduce

Download this file - https://file.io/ttsJDtW5Hsou (Anonymised dicom file)
Run it with the code snippet below.

import pydicom

ds = pydicom.read_file(filename, force=True)
im = ds.pixel_array

Issue seen on Stackoverflow, but no answer provided.
https://stackoverflow.com/questions/67172228/cannot-read-dicom-image

Your environment

macOS-13.5.1-arm64-arm-64bit
Python 3.11.5 (main, Sep 11 2023, 08:31:25) [Clang 14.0.6 ]
pydicom 2.4.4
pynetdicom 2.0.2

[scaramallion]

Link to the file doesn't work (file has been deleted).

Please post the group 0x0028 elements, the transfer syntax UID and the length of the Pixel Data.

[Neuroforge]

Try this one.. https://drive.google.com/file/d/1ctjdMtSK5V9OvPIXRkG2yKAbRTWeTk_z/view?usp=drive_link

[scaramallion]

You can attach datasets if you change the extension to .zip/.txt

[Neuroforge]

Here are the tags....

https://pastebin.com/FX7xNTub

(0028, 0002) Samples per Pixel                   US: 3
(0028, 0004) Photometric Interpretation          CS: 'RGB'
(0028, 0006) Planar Configuration                US: 0
(0028, 0010) Rows                                US: 768
(0028, 0011) Columns                             US: 1024
(0028, 0014) Ultrasound Color Data Present       US: 1
(0028, 0030) Pixel Spacing                       DS: [0.0932203, 0.0932203]
(0028, 0100) Bits Allocated                      US: 16
(0028, 0101) Bits Stored                         US: 16
(0028, 0102) High Bit                            US: 7
(0028, 0103) Pixel Representation                US: 0
(0028, 0106) Smallest Image Pixel Value          US: 0
(0028, 0107) Largest Image Pixel Value           US: 255
(0028, 1050) Window Center                       DS: '127.0'
(0028, 1051) Window Width                        DS: '254.0'
(7fe0, 0010) Pixel Data                          OB: Array of 481304 elements

This is just a zipped version of the file. The extension was not changed, the file was just compressed. Open the zip file to see it's contents.

1.2.826.0.1.3680043.8.498.84891584019088827677474160028118926790.dcm.zip

[scaramallion]

Can you paste the output from

$ python -m pydicom.env_info

as well please?

[Neuroforge]

Transfer Syntax UID = 1.2.840.10008.1.2.4.90

module       | version
------       | -------
platform     | macOS-13.5.1-arm64-arm-64bit
Python       | 3.10.13 (main, Sep 11 2023, 08:16:02) [Clang 14.0.6 ]
pydicom      | 2.3.1
gdcm         | _module not found_
jpeg_ls      | _module not found_
numpy        | 1.22.3
PIL          | 9.4.0
pylibjpeg    | _module not found_
openjpeg     | _module not found_
libjpeg      | _module not found_

[scaramallion]

The High Bit/Bits Stored/Bits Allocated values are non-conformant/incorrect...

Hmm, change the Bits Stored and Bits Allocated to 8 instead?

[Neuroforge]

        ds.BitsAllocated = 8
        ds.BitsStored = 8
        im = ds.pixel_array
        cv2.imshow("Window",im)
        cv2.waitKey(0)

Allows the im object to be created but the output image appears to be a blank image.

The image does load in Weasis dicom viewer.

[Neuroforge]

Actually, that may be an opencv issue.

[scaramallion]

Are you using pillow for this? It's returning 8-bit data in 8-bit containers so the byte count is off. GDCM and pylibjpeg return the 8-bit pixels in 16-bit containers, so everything works OK

Edit: yeah, you are. This is a pillow issue combined with the incorrect Bits Stored/Bits Allocated

Nope, see below.

[Neuroforge]

Thank you for taking the time to assist with this issue. It is greatly appreciated.

[scaramallion]

OK, the J2K codestream uses 16-bit precision, so Bits Stored and Bits Allocated are correct and High Bit is wrong. Pillow is returning 2359269 bytes, which is wrong for those dimensions and precision.

from pydicom.encaps import generate_frames
from pydicom import dcmread
import PIL

ds = dcmread("2006.dcm")
frame = next(generate_frames(ds.PixelData))
im = PIL.Image.open(frame)
len(im.tobytes())  # 2359296
im.mode  # 'RGB'

This is because Pillow doesn't support RGB data with more than 8-bit depth. So Pillow downscales the 16-bit data to 8-bit. We can't fix this at our end because the downscaling is lossy. @Neuroforge if you want it to work you need to use GDCM or pylibjpeg. Also, just a note that the dataset is non-conformant in at least two ways:

1. The High Bit value should be 15
2. The JPEG 2000 encoded pixel data has a JP2 header

Welp @mrbean-bremen, yet another subset of Pixel Data we can't use Pillow for.

[mrbean-bremen]

Welp @mrbean-bremen, yet another subset of Pixel Data we can't use Pillow for.

As I wrote before - I would plead to remove pillow from the supported plugins, now that we have all the pylibjpeg plugins (thanks to you), and pygdcm is also easy to install.