New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to work with image with more than 3 bands #3
Comments
When looking into the pysnic/pysnic/examples/superPixels.py Line 32 in f4239e8
The The Line 24 in f4239e8
For both functions you can, of course, pass custom lerp and distance functions if you need to adjust the weighting of the different channels or image distances. |
Okay, understood. My question was regarding this line. If I have image of 10 bands, this function will fail, I cannot convert my 8 band image to CIELAB color space because it works only on 3 bands.
What's my alternative here? |
The transformation into the CIELAB color space is done in order to perform the super-pixelation in an perceptually uniform space. The euclidean distance in CIELAB roughly corresponds to the perceived color difference and therefore should hold better results than passing the plain RGB image. If you have multiple bands of data, you have to find a metric that correctly represents the distance between two pixels. This can either be done by transforming the input (e.g. transforming RGB to CIELAB, normalizing the individual bands, ...) and using the euclidean distance; or by writing a custom metric that handles the different kind bands accordingly (e.g. one that measures the angle between vectors if a band contains rotational data). If you do not need to transform the data any further, you can just leave out the rgb2lab transform. Just make sure, that you pass in the data as a plain python array. The algorithm performs large amounts of single pixels reads, which are much slower when using numpy arrays. |
Thanks for this explanation. As mentioned in the paper, |
Yeah right. If you just pass an int for the |
okay, understood! but this dosen't work well on
|
The image format should not matter, as the function expects an 3d array However, I quickly looked at the channel ranges in the demos and found the value range to be [0..100, -128..128, -128..128]. From the demo image I observed [0..92, -31..80, -58..63]. With an image size of 400x600 and a compactness of 10.0, you get a distance of When normalizing channels to [0..1] the color component does not have enough weight and the distance function takes over. With normalized values you have to adjust the compactness. I think a compactness between 0.01 and 0.001 should work fine. Thanks for pointing out that problem, I will add it to the documentation. |
Hmm, Im still confused. What do I have to change at my end? The compactness? |
Yeah, you have to adjust the compactness term. Internally the default metric is composed of the distance between the centroid and a candidate pixel, and the difference in color of the centroid and the candidate pixel. If you have a look at If we would only want to use the color term, it could happen, that a single cell covers the whole image. To prevent this from happening we want to penalize the inclusion of pixels that are too far away from the centroid. You have to try out (or calculate) some compactness values, so that after some time (maybe 1-3 times the initial seed distance) the distance penalization takes over. |
So I was able to run something recently with a 14-band image, However, the SNIC doesn't seem to run till 100%
What could be a reason? Also, in my final image, I'm not getting many closed polygons. What could be a possible fix here. This doesn't happen for simple rgb png file |
The "processed 100%" line missing, is a bug with logging. I just created an issue for that - #4 . If the function returns, all pixels got processed. However I'm not quite sure about the "many closed polygons" part. Every given seed results in a super-pixel of at least 1 pixel extend. The super-pixels should always be a single compact area in the image. If you see some large super-pixels and many single-pixel super-pixels, this may be an indication that the distance penalization is not strong enough. Maybe a higher compactness is needed (0.1, 1.0?). To see if there are single pixel super-pixel it may be better to plot the resulting segmentation directly using one of the qualitative color maps of matplotlib. Edit: The 'prism'-cmap worked quite well for visualization.
|
Ignoring the color space conversion, is it possible to just implement spectral euclidean distance in your script? |
As said before, the algorithm is sensitive to the magnitudes of the input data. So the spectral euclidean distance would be the euclidean distance with the correct weighting applied to the different bands. If you could provide a sample of your data as a *.npy file, I can try to work out the needed parameters for your setting or adjust the code accordingly . |
Sure, however, while I was checking the seed inputs. In the case where I just give |
If you look at the shape of the |
In you recent description, you mentioned, performance can be optimized by using raw array before passing to What's the best way to get around it? |
Oh, that's bad phrasing from my side: With "raw" array I meant python arrays (
I guess that using a C++ implementation of the algorithm and passing in the data from python would already give a significant speedup. The standard SNIC algorithm itself uses a priority queue which makes it hard to parallelize. Dropping the global priority queue and splitting it into multiple local ones should speed up the algorithm. This may result in a degradation of the segmentation quality. Either way, one needs to find a way to efficiently resolve contested pixels between the local queues. |
Is there a way I can convert this into a gpu implementation? What all functions could possibly be converted to use cuda under the hood for quicker inference |
In my opinion the primary problems with parallelizing the standard SNIC algorithm are the centroid update after each queue pop and the global queue in general. SLIC proposes the opposite approach, where none of the associations are interlocked and the pixel labels are only assigned at the end of an iteration. 1.) We may want to make the assumption that we only need the priority queue to resolve pixels near segment borders. So it might be an interesting approach to first apply one or two iterations of SLIC and only afterwards use a priority queue to refine the segment borders. 2.) Assuming that far-apart centroids will not compete over the same pixels, we can split up the global priority queue and use a per-kernel priority queue. We would then only need to resolve conflicts between neighbouring queues. In case you are also using the polygonization: The segmentation to graph conversion can be done with a simple reduction. I also found a new paper "NICE: Superpixel Segmentation Using Non-IterativeClustering with Efficiency" by Cheng Li et al. which builds up on SNIC and seems to reduce the runtime. However I haven't looked into the used optimizations and quality of the results yet. |
I have images/arrays with multiple bands, can we modify it to work for multiple bands? Or is there an alternate way ?
The text was updated successfully, but these errors were encountered: