Very dark colors in CAM02

[TFiFiE]

Why are the differences between virtually indistinguishable colors near pitch black (whose channels in sRGB255 differ by only a few bits) so large in the CAM02 color spaces? I have no reason to believe there is anything wrong with the implementation and the color differences seem awfully large to just blame on the chosen default viewing condition parameters or how well the typical monitor adheres to sRGB.
This shows up in some of the designed Matplotlib colormaps, as has been noticed here.

[njsmith]

I don't know. If you have questions about the design of CAM02-UCS, you should probably ask the authors of the CAM02-UCS paper; we just copied down their equations into Python :-).

I do know that generally, color science is an evolving empirical discipline; at the time we started working on viridis, CAM02-UCS was state-of-the-art, but that doesn't mean it can't be improved, it just means that improving it requires actual research. It might be that end-user monitors/viewing conditions really are just that different from the model's assumptions, or that the model has a bug, or that the model does an excellent job at matching experimental data but not many people have run experiments in that region so the data isn't there... I have no idea, really.

It'd be quite interesting to do some kind of more controlled study focused on the needs of colormaps and scientific visualization specifically. E.g., trying to measure the exact shape of the subjective lightness curve for volunteers using their every-day hardware / viewing conditions.

[KelSolaar]

@TFiFiE : Do you have any data to test, i.e. "the indistinguishable colors near pitch black"?

[TFiFiE]

One can easily find colors with dubiously large differences between them just by adding a few random bits to #000000, but here are a few examples:

sRGB255 color 1	sRGB255 color 2	CAM02-UCS difference
#010100	#010001	8.70149539890018
#010001	#000100	10.394724818083024
#000200	#010001	12.00016193292083
#030020	#000120	2.731818011356716
#02000D	#00000D	4.223941095941429
#030005	#010105	5.510464024056734
#050012	#000112	6.735231201024038
#040102	#000302	9.410823440201222
#030000	#000100	10.262845092061296
#070000	#010300	12.85215007412199
#0B0105	#000605	14.745599512816067
#040013	#000113	5.517270987503764
#030014	#000114	4.21444064216019
#090025	#010225	5.428741979630107
#09001D	#01021D	6.652166938839056

[KelSolaar]

Quite interesting, ran some computations on my end and got similar numbers for #010100 and #010001:

Colourspace	Colour Difference
CAM02UCS	8.68281451881
CAM16UCS	9.26752527059
JzAzBz	7.23782315779e-05
Delta E 1976	0.874233493433
Delta E 1994	0.869085322973
Delta E 2000	1.04570887309
Delta E CMC	1.30331810724
DIN99	0.704261598824

[njsmith]

That does seem suspicious. Looking at CAM02-UCS, a small change in sRGB greyscale values gives the expected effect of a small change in lightness (<2) and a miniscule change in hue/colorfulness:

In [3]: colorspacious.cspace_convert([0, 0, 0], "sRGB255", "CAM02-UCS")
Out[3]: array([5.54015251e-22, 0.00000000e+00, 0.00000000e+00])

In [4]: colorspacious.cspace_convert([1, 1, 1], "sRGB255", "CAM02-UCS")
Out[4]: array([ 1.8799272 , -0.16227095, -0.09773843])

But an even smaller changes that deviates from greyscale produces a large change in CAM02-UCS's hue/colorfulness coordinates:

In [5]: colorspacious.cspace_convert([1, 0, 1], "sRGB255", "CAM02-UCS")
Out[5]: array([ 1.0100188 ,  4.35876194, -2.53742912])

In [6]: colorspacious.cspace_convert([1, 1, 0], "sRGB255", "CAM02-UCS")
Out[6]: array([ 1.7978748 , -1.18748961,  4.12097901])

It's also not clear to me whether this affects viridis-style colormaps much, because not only are they uniform in CAM02-UCS, they're also uniform if you look at just the lightness component of CAM02-UCS.

[njsmith]

I just emailed the corresponding authors on the CAM02-UCS and CAM16-UCS papers pointing them to this thread; perhaps they'll have some thoughts.

[KelSolaar]

@njsmith : They are hard to get in touch with, crossing fingers!

[njsmith]

One of the emails bounced and I can't seem to find a newer email address, but the other claimed to get through, so, we'll see :-)

[clamydo]

The other way round I also get dubios results. I get dark CAM02-JCh mapped to bright sRGB

cspace_convert([1.2, 49, 275], 'JCh', 'sRGB1')     #in
array([0., 1., 0.])                                #out

Is this connected to the observation here? This produces some really annoying artifacts in my plots and I'm trying to understand, how to prevent this.

[TFiFiE]

@njsmith, you never heard back from them? Since CAM16-UCS seems to have the same issue, would it be fair to instead crown J_za_zb_z as the uniform color space of choice?

[njsmith]

I haven't heard back, no.

I'm not really qualified to crown a "uniform color space of choice", or even judge the quality of the work :-). But from a quick read the design goals for Jzazbz seem to match the things we like about CAM02-UCS, and if it works better for you then sure, go for it.

BTW, I don't really have time to implement new colorspaces in colorspacious these days. I think colorspacious's UX is really nice (in particular the automatic generation of conversion paths), so it'd be nice to preserve that somehow – either by adding new colorspaces to colorspacious (PRs accepted?), or by merging colorspacious into another library that has more dedicated maintainers.

[njsmith]

Alternatively, do any of you want to take over colorspacious maintenance?

[KelSolaar]

by merging colorspacious into another library

I was poking at auto-conversion for Colour a few months ago with NetworkX and was maybe thinking that the lightweight approach of Colorspacious was probably better fitted than dragging a huge dependency like NetworkX.

[TFiFiE]

In terms of subject matter, Colour has everything colorspacious has, no? However, I would look into why the color difference is off by almost 0.02 in that example above, as that looks pretty significant.

[KelSolaar]

@TFiFiE : It could be coming from a slight difference on how the whitepoint and XYZ to RGB matrices are expressed. Hard to say without digging in the respective codebases.

[TFiFiE]

OK, I found out the difference is mostly due to you having used 4 instead of 64/pi/5 as the CIECAM02 L_A parameter.

On a side note, it raises my eyebrow that what colour.sRGB_to_XYZ outputs is on a different scale than what colour.XYZ_to_CIECAM02 expects as input.

[KelSolaar]

OK, I found out the difference is mostly due to you having used 4 instead of 64/pi/5 as the CIECAM02 L_A parameter.

Great find! I should have included my code to avoid having you traverse everything!

On a side note, it raises my eyebrow that what colour.sRGB_to_XYZ outputs is on a different scale than what colour.XYZ_to_CIECAM02 expects as input.

There is a very good reason for that, nobody agreed on any scale in the colour science field and we cannot decide what the default should be for everybody. Taken from our documentation, here are some points illustrating the problem:

• Colour Scientist performing computations related to Munsell Renotation System would be very surprised if the output Munsell Value was in range [0, 1] or [0, 100].
• A Visual Effect Industry artist would be astonished to find out that conversion from CIE XYZ to sRGB was yielding values in range [0, 100].

One can add many more and because we try to be as much as possible close to the Reference we implement we did not picked a default. We discussed about this at length though, and, what we ended up doing earlier this year was to implement some mechanisms allowing to use a scale where everything is soft normalised in range [0, 1], you can read more about it there: https://colour.readthedocs.io/en/develop/basics.html#domain-range-scales

Keep in mind it is not released yet!

[njsmith]

One of the nice things about colorspacious's conversion graph is that it largely solves this problem: it's easy to provide both (what we call) XYZ100 and XYZ1, they're on equal footing from the user point of view, and for conversions that pass through XYZ the machinery automatically makes sure to use the appropriate scaling on both sides.

[KelSolaar]

One of the nice things about colorspacious's conversion graph is that it largely solves this problem:

Yes absolutely!

[jaklein]

This is mostly due to the sRGB transfer function having a linear segment for very dim components, while CIECAM02 uses a power function for the whole range.

[TFiFiE]

But an even smaller [change] that deviates from greyscale produces a large change in CAM02-UCS's hue/colorfulness coordinates:

This is quite interesting in light of the fact that "hue linearity" is one of the design criteria of J_za_zb_z (and is something which CAM16-UCS scores poorly on). This performance measure is less directly tied to perceptual uniformity (being, for example, useful to indicate suitability for gamut mapping) and it makes J_za_zb_z more of a compromising color space, with a seemingly (slightly) worse performance than CAM16-UCS when it comes to color differences. However, it could be that the concern for hue linearity has actually had a regularizing effect that makes up for gaps in the experimental color-difference data and that the more complex CAM02/16 models are overfitted.

https://www.osapublishing.org/viewmedia.cfm?uri=oe-25-13-15131
http://www.color.org/events/prague/MuhammadSafdar2017.pdf