Add --output-rescale

[tfarago]

and by default do not rescale the output. @sgasilov the rescale was actually happening by default, this PR makes it so that when you do not specify --output-rescale it won't happen, otherwise yes, i.e. the old behavior.

[sgasilov]

Hello T,

it works. I can apply tofu sinos twice to 8 or 16 bit data with and get the same projections as the original imput.

I have one suggestion: would it be possible, for the sake of simplicity, to enforce rescale automatically if minimum and maximum are specified explicitly?

I'm testing the behavior of tofu reco and if I call reco with
--output-bitdepth 16 --output-minimum " -0.001" --output-maximum " 0.001"
the output is flat zeros. If I add --output-rescale then all is well.
Basically what I'm proposing is that if minimum and maximum are not specified explicitly but user wants output data with different bitdepth then rescale can be turned on if necessary (and if only min or only max is specified, the other one can be taken as the limit of the respective data type)

What do you think? I imagine it might require a couple of extra "if" statements in the tofu reco and tofu sinos but, unless I'm omitting something, it will be more transparent that way. Let me know what do you think. If you can do it then I do not have to change anything in tofu ez. If you prefer to keep it more robust and simpler in the code, then I need to add "--output-rescale" in a couple of places in tofu ez before we can merge it with master.

Cheers,
S.

[tfarago]

That makes sense, so let's turn on --output-rescale automatically if any of minimum or maximum are given. But then I wouldn't use the output data type to figure out the other limit if only one is specified, I would rather use the input data to determine the other one. This would also drop the "TODO" here. So I'd change also ufo-filters there. Why not output data type other limit: e.g. a slice grey values are like (-10, 10) and if you specify lower limit to something and upper not and use 65535 as uint16 max value it will not make much sense. What do you think?

[sgasilov]

Yeah, that is a good idea - can be useful to clip data at one end of the intensities.

[tfarago]

Here we go, please test and don't forget to pull also the branch in ufo-filters:#224 before.

[sgasilov]

Hello Tomas,

I tested it and it works. Except for when you call "reco" and specify --output-bitdepth only. Then it returns all zeros. If you specify --output-bitdepth together with --output-rescale then it returns correct values stretched to the respective clims. Interestingly, when you call tofu sinos and specify bitdepth only, it returns correct output so if you apply tofu sinos twice (projections-->tomo-->projectsions) with bitdepth 16 flag and nothing else you would get your original input projects (which is exactly what I needed to organize vertical stitching of orthogonal sections for slices reconstructed in arbitrary bitdepth)

Returning to tofu reco, I do not know why it happens, but, just in order to avoid zeros, it appears that yet another condition statement like
if params.output_minimum is None and params.output_maximum is None and params.output_bitdepth != 32:
writer.props.rescale = True
should solve the problem. Practically, however, I cannot imagine why would someone permit independent scaling of reconstructed slices without defining the global min/max for the whole stack.

Wish you very pleasant Holidays,
Sergei.

[tfarago]

Hm, when I use output bitdepth without the rescale flag, it gives me zeros also for tofu sinos. Maybe your sinograms are already pre-processed and span more than from 0 to 1? If not we need to look into that before merging.

With tofu reco the situation is a little bit more complicated beyond the issue with zeros. When the user specifies the output with the .tif extension, then the writing is actually done on the python side to ensure proper ordering of slices, i.e. I get the slices into memory and write them by tifffile. However, with recent updates to ufo-filters I should be able to unify both approaches.

[sgasilov]

Hello,

i've just confirmed what you see - if I enable ffc than I also get zeros in the output of tofu sinos if only bitdepth is set. In my initial test I was basically reslicing raw projections twice so the input was in 16 bit.

Not sure how do you want to go about that; as I mentioned in my view trying to convert floating point numbers into integers without defining the output interval is meaningless anyways so I would just leave it as is.

[tfarago]

Would it make your life easier to merge this now? If not I'd leave it open and polish also the single-tif writing and so on.

[sgasilov]

Hi, I should definitely upgrade our servers this week, it has been long overdue, but I can use the current add-rescale branch. Not sure what you mean with single tif writing, I didn't notice anything wrong with that

[tfarago]

Alright, let's leave this alive then for a little longer. By single tif writing I mean that in case you want just one output file I get the slices from the GPUs and write them on the python side, which is not optimal and I wanted to update that.

[tfarago]

Hi @sgasilov , can we close this?

[sgasilov]

Hola! If you finished all that little things which you wanted to finish then of course! Also, could you merge ez-dev then with master after you add --output-rescale to master?