Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drawing on oblique images is really bad #1407

Open
AlexanderGetka-cbica opened this issue Mar 24, 2021 · 4 comments
Open

Drawing on oblique images is really bad #1407

AlexanderGetka-cbica opened this issue Mar 24, 2021 · 4 comments

Comments

@AlexanderGetka-cbica
Copy link
Contributor

AlexanderGetka-cbica commented Mar 24, 2021

Describe the bug
Drawing a mask using the GUI drawing tools on a loaded, oblique image is basically impossible because the drawing doesn't appear where you draw. You might draw on one slice and the mask won't be visible until you scroll through more slices, and depending on what you drew, it might appear in pieces as you scroll through slices.

To Reproduce
Load an oblique DICOM image into the images panel
Go to drawing tab
select rectangle or freehand or whatever
Draw on any slice.
See that the mask doesn't appear where your cursor was.
Scroll around confused and notice that your mask is there, just not where you put it, and sometimes on a diagonal.

Expected behavior
The drawing should appear under my cursor.

Screenshots
If applicable, add screenshots to help explain your problem.

CaPTk Version
1.8.1

Desktop (please complete the following information):
Seen on Windows. Probably present on other platforms.

Additional context
I guess it's a result of our drawing space being mapped differently than the image in the viewer. I'm not currently able to determine what exactly is causing this or if oblique images are loaded properly into the viewer (they can look kind of odd.) A collaborator is asking for some sample NIfTI segmentations on an oblique DICOM image generated by CaPTk -- they previously tried loading up a mask like that in ITK-Snap and it was way off-alignment. It's not really possible for me to check that the mask is being saved correctly, because I can't really draw it decently in the first place.

I guess it would be sufficient to have some way to correct the oblique-ness at load time, have all drawing/rendering take place on the corrected image, then "undo" the correction when saving or running code. Not sure.

Another point: if I run geodesic training segmentation on a mask I drew on an oblique image, when it finally completes, CaPTk will immediately say that it doesn't match the dimensions of the currently loaded image and opens the registration window.

Pinging @sarthakpati @ashishsingh18 for any insight on this.

@AlexanderGetka-cbica
Copy link
Contributor Author

Leaving this here as notes: According to Sarthak this has to do with how we handle direction changes. For loading into the viewer, typically we have an image with some original direction matrix that has to get collapsed to identity. We also store the "original" direction information so we can convert back. But we aren't doing this for ROIs/masks right now.
Apparently for large amounts of obliqueness, just including the original directionality information will cause extreme problems.
So we should correct this by registering the mask/roi to the image, and then undoing this registration when saving. (Sarthak also notes that we should be doing "nearest-neighbor"/"nearest-label" instead of linear interpolation when going in the reverse direction).

Also of note: We need to put this "original direction" information in anything that is produced from running on a loaded image.That way, when the derivative image gets saved, it also gets reverted back to the original direction. This needs to happen for any operation performed on loaded images, because directionality changes and registration don't operate on images loaded into the viewer anyway (same is true for all CLI apps).

@AlexanderGetka-cbica
Copy link
Contributor Author

AlexanderGetka-cbica commented Mar 26, 2021

More notes:

First principle: If a user loads an image into our viewer, regardless of its direction, any derivative output should match up with the same image they started from. That is, the user doesn't care about what image data is being manipulated in our viewer, they care about the data they had when they came in (i.e. on disk). Any output we produce to disk should be in the same space, unless the user is deliberately changing the space.

Tools on the GUI could potentially be separated into several groups that need to be handled separately with regard to this issue.

  1. Tools that don't operate on images in the viewer at all-- they take a filepath as input and a filepath as output.
  2. Tools that operate on images in the viewer, and directly create a derivative image (or mask) in the viewer.
  3. Tools that operate on images in the viewer, and create a saved file as output (and then may or may not immediately load that output file). BiasCorrection actually saves the selected image first, then calls the CLI on that image to avoid this weirdness -- should this be our standard approach?
  4. (There is a theoretical fouth category of tools that take a filepath as input and directly load output into the viewer... but besides "Load Image"/"Load ROI" I don't think we have these.)

Category 1 doesn't need this correction to be considered -- the algorithm can directly operate on the image. If image obliqueness breaks the algorithm, that is the fault of the algorithm and should be fixed there (or otherwise noted in its documentation that it requires some preprocessing).
Category 2 needs to have the directionality correction applied (and the inverse transformation held in reserve) when the input image is loaded as currently happens. But since the derivative output will be in the space of the loaded image in the viewer, the output must also have the same "inverse transform" information held as the input. Basically, that info needs to be injected into the result after it is loaded, so that at save-time it is reverted back to match the image on disk it was created from.
Category 3 is trickier. If the loaded image is in identity direction, then the derivative output file on disk is now in identity direction. So we would then need to immediately apply the inverse transformation to that output in order to ensure it is in the same space as the image on disk it was created from.

I might (read: will definitely) need some help/pointers figuring out exactly what we need to do here.

@AlexanderGetka-cbica
Copy link
Contributor Author

AlexanderGetka-cbica commented Mar 26, 2021

@sarthakpati Do we need to change ALL image/mask loading/unloading to do the registration process you described? Or does this only need to be done for masks/ROIs?

@sarthakpati
Copy link
Contributor

I would imagine that this would need to be done for all images and masks. But before looking at the registration solution (which would probably be more stable), I would recommend looking at changing the rotation matrix and directions so that everything gets squished to unity to see if that works, because that would involve the least amount of interpolation...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants