[BUG 🐛] division by zero error in estimate_foreground_ml

[eyaler]

i am getting division by zero errors in estimate_foreground_ml()

what i tried:

• pymatting 1.1.1 and 1.1.3
• making sure both the image and the mask are not uniform (i've seen the error when both have min_val=0 and max_val=1)
• default parameters and different variations

the environment is google colab. also sometime this (or something else in pymatting) causes the colab itself to crash and disconnect.

[tuelwer]

Hi @eyaler, thank you for opening this issue! Could you kindly provide code, the image and the alpha matte that yield the division-by-zero-error? That would be very helpful. Thanks!

[99991]

The only place where I could imagine that division by zero errors could occur is when dividing by the determinant here:

pymatting/pymatting_aot/estimate_foreground_ml.py

Line 137 in e5b1e86

inv_det = 1.0 / determinant

A positive regularization value should in theory ensure that the determinant is positive: https://www.wolframalpha.com/input/?i=%28%28a%5E2+%2B+d%29+*+%28%281+-+a%29%5E2+%2B+d%29+-+%28%281+-+a%29+*+a%29%5E2%29+%3E+0%2C+d+%3E+0

That might not happen when the regularization parameter of estimate_fb_ml is regularization <= 0 or gradient_weight < 0, but those values are not sensible and not the default, so I doubt that this is the reason.

Either way, I agree with @tuelwer that we need a minimal reproducible example to debug this. (See test case template: https://github.com/pymatting/pymatting/blob/master/.github/ISSUE_TEMPLATE/bug_report.md)

[eyaler]

i was running this with the defaults (reg=1e-5, weight=1).
anyways to reproduce: https://colab.research.google.com/drive/1MXfC3vlU6u2muNG0WaB1p6HeB-5q2ReL
sometimes it works, sometimes crashs, sometimes divide by zero. try a few times.
screenshot:

[99991]

The alpha mask should be of shape (h, w) instead of (h, w, 1).

Replace

mask=imageio.imread('mask.png')[:,:,:1]/255

with

mask=imageio.imread('mask.png')[:,:,0].copy()/255

and it should work fine.

Apparently, Numba simply ignores the shape specified at

pymatting/pymatting_aot/estimate_foreground_ml.py

Line 170 in e5b1e86

"Tuple((f4[:, :, :], f4[:, :, :]))(f4[:, :, :], f4[:, :], f4, i4, i4, i4, f4)",

and computes garbage instead of throwing an error:

numba/numba#2757

I think this issue should be fixed in the Numba library.

[eyaler]

i can confirm this solves my issue. until numba fixes it, this could easily be solved by doing alpha.squeeze() or add a warning and mention this in the docs, instead of the current:

alpha: numpy.ndarray
    Input alpha matte shape :math:`h \\times  w \\times 1`

which is just evil...

[99991]

Sorry about that. This was definitely a mistake in the documentation. Should be fixed now.

We could of course do all kinds of checks to see whether the passed arguments are correct and convert them to a more suitable format if necessary, but there is an infinite number of ways to get it wrong, so I do not think that this doable. Just a few examples:

• None arguments where not allowed
• incorrect domain
• incorrect object type (e.g. someone might pass a pytorch or tensorflow tensor)
• incorrect numpy dtype (uint8 should probably be divided by 255, but what if the matte just happens to be binary? If the data type is uint16, it might still be in range [0, 255] because some image loaders just happen to do it like that. Or maybe it is in range [0, 32767] because it was actually int16 and not uint16)
• fortran or C data order (this definitely has to be solved by Numba)
• incorrect shape ([:, :, :1] could be converted to [:, :], but what about [:, :, :3]? Mean or median over channels? Pick a specific channel? Convert to HSL or CIELAB color space and pick L? What if someone passes a batch of images of size 1? ([:1, :, :]) What the size of all dimensions is 1? How to tell those cases apart?)

What I am trying to say is, if we start to fix incorrect input, there will be no end to it.

The updated documentation (thanks for the tip by the way) will have to do for now.

[99991]

Type hints might also be a good additional measure to issues like this. Unfortunately, NumPy does not yet support shapes, only dtype.

I'll leave these issues here so it is easier to follow the development of type hinting for shapes:

• Typing support for shapes numpy/numpy#16544
• Type hinting / annotation (PEP 484) for ndarray, dtype, and ufunc numpy/numpy#7370

[99991]

We switched back to JIT compilation today because ahead-of-time compilation caused too many issues. Numba's JIT backend is a bit stricter when checking types.

[shamindraparui]

hi, I'm facing the same issue . [ZeroDivisionError: division by zero]
step 1 :
Reading image and binary mask then generating trimap:

img = cv2.imread("blend/ILSVRC2012_test_00000250.jpg",cv2.IMREAD_COLOR)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
mask = cv2.imread("blend/ILSVRC2012_test_00000250.png", cv2.IMREAD_GRAYSCALE)
mask = mask.astype("uint8")
t, msk = cv2.threshold(mask, 127,255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)

trimap function:

def generate_trimap(mask,eroision_iter=6,dilate_iter=4):
    mask[mask==1] = 255
    d_kernel = np.ones((3,3))
    erode  = cv2.erode(mask,d_kernel,iterations=eroision_iter)
    dilate = cv2.dilate(mask,d_kernel,iterations=dilate_iter)
    unknown1 = cv2.bitwise_xor(erode,mask)
    unknown2 = cv2.bitwise_xor(dilate,mask)
    unknowns = cv2.add(unknown1,unknown2)
    unknowns[unknowns==255]=127
    trimap = cv2.add(mask,unknowns)
    labels = trimap.copy()
    labels[trimap==127]=1 #unknown
    labels[trimap==255]=2 #foreground
    return labels

generating trimap:
result = generate_trimap(msk)

step 2:
preprocess the image & mask then fed it to pymatting library:

rr = result.copy()
rr[rr == 2] = 255
rr[rr == 1] = 127
rt = rr / 255
img1 = img.astype('float64')
alpha = pymatting.estimate_alpha_cf(img1,rt, laplacian_kwargs={"epsilon": 1e-6},cg_kwargs={"maxiter":2000})

now, estimate_alpha_cf is throwing the error of "ZeroDivisionError: division by zero"
am I missing something?

[99991]

I think this is a different issue because the error happens in estimate_alpha_cf instead of estimate_foreground_ml.

Anyway, could you create a "minimal, reproducible example" with complete code and images so we can reproduce the issue?

[shamindraparui]

hi @99991 ,

I am sharing the notebook and corresponding images.
https://github.com/shamindraparui/pymatting-issue.git

[99991]

@shamindraparui Thank you, I can reproduce the issue.

PyMatting expects that all images are in the range [0, 1], but your img is in the range [0, 255]. Simply dividing it by 255 (or using pymatting.load_image) will fix it:

img = img / 255.0

However, the results are not great because ILSVRC2012_test_00000250.jpg is JPEG-compressed a lot. The compression artifacts will confuse the algorithm.

The artifacts can be reduced by using more erosion, but that will remove part of the beak of the bird. It is better to use a few steps of skeletonization in this case, but the result is still not perfect.

You might be better off using the original mask you had for that image. What did you use to compute it?

import numpy as np
import pymatting
import matplotlib.pyplot as plt

img = pymatting.load_image("blend/ILSVRC2012_test_00000250.jpg", "RGB")
mask = pymatting.load_image("blend/ILSVRC2012_test_00000250.png", "GRAY")

is_fg = (mask > 0.9).astype(np.uint8)
is_bg = (mask < 0.1).astype(np.uint8)

def incorrect_but_good_enough_skeletonization_step(mask):
    keep = np.ones(2**9, dtype=np.bool8)
    keep[[23, 27, 31, 54, 55, 63, 89, 91, 95, 216, 217, 219, 308, 310, 311,
        432, 436, 438, 464, 472, 473, 496, 500, 504]] = 0
    padded_mask = np.pad(mask, [(1, 1), (1, 1)], mode="edge")
    windows = np.lib.stride_tricks.sliding_window_view(padded_mask, (3, 3))
    return mask * keep[windows.reshape(*mask.shape, 9) @ 2**np.arange(9)]

for _ in range(3):
    is_fg = incorrect_but_good_enough_skeletonization_step(is_fg)
for _ in range(5):
    is_bg = incorrect_but_good_enough_skeletonization_step(is_bg)

trimap = 0.5 + 0.5 * is_fg - 0.5 * is_bg

alpha_cf = pymatting.estimate_alpha_cf(img,trimap,laplacian_kwargs={"epsilon": 1e-2})
alpha_lkm = pymatting.estimate_alpha_lkm(img,trimap,laplacian_kwargs={"radius": 5})
alpha_knn = pymatting.estimate_alpha_knn(img,trimap)

def blend(alpha):
    foreground = pymatting.estimate_foreground_ml(img, alpha)
    new_background = 0.5 * np.ones_like(foreground)
    blended = pymatting.blend(foreground, new_background, alpha)
    return blended

composite_cf = blend(alpha_cf)
composite_lkm = blend(alpha_lkm)
composite_knn = blend(alpha_knn)
composite_original = blend(mask)

plt.figure(figsize=(20, 10))

for i, (title, image) in enumerate([
    ("Alpha CF", alpha_cf),
    ("Alpha LKM", alpha_lkm),
    ("Alpha KNN", alpha_knn),
    ("Original mask", mask),
    
    ("Trimap", trimap),
    
    ("Composite CF", composite_cf),
    ("Composite LKM", composite_lkm),
    ("Composite KNN", composite_knn),
    ("Composite original", composite_original),
], 1):
    plt.subplot(2, 5, i)
    plt.title(title)
    plt.imshow(image, cmap="gray")
    plt.axis("off")

plt.tight_layout()
plt.savefig("plot.png")
plt.show()

[shamindraparui]

@99991
Thank you for the solution. The original mask that I have shared with you, is part of DUTS dataset for Salient Object Detection.

I am trying to compute the salient part of a given image as mask. The algorithm that I am using is U2Net. It is predicting a mask (not so good) which needs to be binarized, trimapped and alpha matted for final overlaying/segmenting the original image.

[99991]

@shamindraparui Nice, I am looking forward to the results! For animals, there is also https://github.com/JizhiziLi/GFM and it seems to generalize to other prominent image objects to some degree. Maybe it will be helpful for your project.

I have added some sanity checks for the input image and trimap. I think this will make it less confusing to provide the input in the expected format. I'll upload it to PyPI later. Until then, it can be installed from source.