Unscrambling obfuscated pixel data in Carl Zeiss Meditec (CZM) ophthalmic datasets

[scaramallion]

I was recently asked to try and decipher some extremely non-conformant DICOM datasets nominally from Carl Zeiss Meditec (CZM) ophthalmic equipment for producing OCT angiography and related images, such as the Cirrus range. The datasets are non-conformant in multiple ways, but perhaps the most serious is that CZM appear to have deliberately obfuscated the pixel data to prevent others from using it.

I'm happy to say I've succeeded in reversing that obfuscation, although for a while there I really thought it wouldn't be possible. I'm posting the solution here for two reasons:

• To help other people who have come across these datasets and would like to be able to decode the pixel data
• The deliberate obfuscation of the pixel data by CZM really annoys me, because it goes completely against the interoperability spirit of DICOM, so the larger the audience the better

The pixel data is obfuscated in two ways:

• First the J2K data has sections transposed (for example, the first 253 bytes are moved to 3/5ths of the way into the data)
• Once the transpositions are done, every 7th byte starting at offset 0 is overwritten by XORing it with 0x5A

The sample datasets I received were horribly non-conformant, but I'm not sure if that was an artefact of how they were obtained or if that's actually how CZM equipment produces them (in which case, wow). The pixel data also has a JP2 header, which again is non-conformant to the DICOM Standard, so be aware of that and remove it if required (search for the 0xFF 0x4F start-of-codestream J2K marker, which should be at offset 85 of the unscrambled data).

So without further ado, here's how to unscramble the pixel data and write it to file as a sane JPEG 2000 file with JP2 header. Alternatively you could uncompress the unscrambled frame using Pillow or your preferred third-party J2K library.

# unscramble.py

import argparse
import math
from pathlib import Path

from pydicom import dcmread
from pydicom.encaps import generate_pixel_data_frame
from pydicom.uid import JPEG2000Lossless


def setup_argparse() -> argparse.Namespace:
    """Setup the command line arguments"""
    parser = argparse.ArgumentParser(
        description=(
            "Read a CZM DICOM dataset and extract and unscramble the pixel "
            "data which is then written to file as one or more JPEG 2000 "
            "images"
        ),
        usage="unscramble path",
    )

    req_opts = parser.add_argument_group("Parameters")
    req_opts.add_argument(
        "path",
        help="The path to the CZM DICOM dataset",
        type=str
    )

    return parser.parse_args()


def unscramble_czm(frame: bytes) -> bytearray:
    """Return an unscrambled image frame.

    Parameters
    ----------
    frame : bytes
        The scrambled CZM JPEG 2000 data frame as found in the DICOM dataset.

    Returns
    -------
    bytearray
        The unscrambled JPEG 2000 data.
    """
    # Fix the 0x5A XORing
    frame = bytearray(frame)
    for ii in range(0, len(frame), 7):
        frame[ii] = frame[ii] ^ 0x5A

    # Offset to the start of the JP2 header - empirically determined
    jp2_offset = math.floor(len(frame) / 5 * 3)

    # Double check that our empirically determined jp2_offset is correct
    offset = frame.find(b"\x00\x00\x00\x0C")
    if offset == -1:
        raise ValueError("No JP2 header found in the scrambled pixel data")

    if jp2_offset != offset:
        print(
            f"JP2 header found at offset {offset} rather than the expected "
            f"{jp2_offset}"
        )
        jp2_offset = offset

    d = bytearray()
    d.extend(frame[jp2_offset:jp2_offset + 253])
    d.extend(frame[993:1016])
    d.extend(frame[276:763])
    d.extend(frame[23:276])
    d.extend(frame[1016:jp2_offset])
    d.extend(frame[:23])
    d.extend(frame[763:993])
    d.extend(frame[jp2_offset + 253:])

    assert len(d) == len(frame)

    return d


if __name__ == "__main__":
    args = setup_argparse()

    p = Path(args.path).resolve(strict=True)

    # Read and check the dataset is CZM
    ds = dcmread(p)
    meta = ds.file_meta
    if meta.TransferSyntaxUID != JPEG2000Lossless:
        raise ValueError(
            "Only DICOM datasets with a 'Transfer Syntax UID' of JPEG 2000 "
            "(Lossless) are supported"
        )

    if not ds.Manufacturer.startswith("Carl Zeiss Meditec"):
        raise ValueError("Only CZM DICOM datasets are supported")

    if "PixelData" not in ds:
        raise ValueError("No 'Pixel Data' found in the DICOM dataset")

    # Iterate through the frames, unscramble and write to file
    if "NumberOfFrames" in ds:
        # Workaround horrible non-conformance in datasets :(
        if isinstance(ds.NumberOfFrames, str):
            nr_frames = ds.NumberOfFrames.split('\0')[0]
        else:
            nr_frames = ds.NumberOfFrames

        frames = generate_pixel_data_frame(ds.PixelData, int(nr_frames))
        for idx, frame in enumerate(frames):
            with open(f"{p.stem}_{idx:>03}.jp2", "wb") as f:
                f.write(unscramble_czm(frame))
    else:
        # CZM is non-conformant for single frames :(
        with open(f"{p.stem}.jp2", "wb") as f:
            f.write(unscramble_czm(ds.PixelData))

[elliottvg]

I've just spend 2 days doing exactly the same thing only to be sent this link by a coworker :D

[etuejcm]

Your work is really helpful!
I'm doing the same thing recently, but I cant find where the OCTA data is. Could you give some suggestions?
Thanks a lot!

[kevinpasquinelli]

@scaramallion: do you have a link to an example of the DICOM file in question?