You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am curious if GLOSH implementation in this repository correctly follows the paper's definition of "outlierness".
According to HDBSCAN* paper (R. J. G. B. Campello et al. 2015, page 25):
In order to compute GLOSH in Equation (8), one needs only the first (last) cluster to which object xi belongs bottom-up (top-down) through the hierarchy, the lowest radius at which xi still belongs to this cluster (and below which xi is labeled as noise), ε(xi), and the lowest radius at which this cluster or any of its subclusters still exist (and below which all its objects are labeled as noise), εmax(xi).
Looking at the max_lambdas function for computing εmax(xi), I think the original paper's explanation (the bold italic text above) is not correctly interpreted. It seems the max_lambdas function only considers the death of a parent cluster (not considering the latest death of its subclusters).
To reproduce this issue, please run the following code:
import hdbscan
import numpy as np
import matplotlib.pyplot as plt
# Step 1. Generate 3 clusters of random data and some uniform noise
data = []
np.random.seed(1)
for i in range(3):
data.extend(np.random.randn(100, 2) * 0.5 + np.random.randn(1, 2) * 3)
data.extend(np.random.rand(100, 2) * 20 - 10)
# Step 2. Cluster the data
k = 15
clusterer = hdbscan.HDBSCAN(alpha=1.0, approx_min_span_tree=False,
gen_min_span_tree=True,
metric='euclidean', min_cluster_size=k, min_samples=k, match_reference_implementation=True)
clusterer.fit(data)
# Step 3. Plot the outlier scores
outlier_scores = clusterer.outlier_scores_
plt.scatter([x[0] for x in data], [x[1] for x in data], s=25, c=outlier_scores, cmap='viridis')
plt.colorbar()
plt.title("Outlier scores")
plt.show()
This should show the following plot:
As you can see from the plot, the outlier scores assigned to the data points between clusters (please find the yellow points between the clusters!) do not seem to look "natural" outliers compared to other outliers. From my own understanding of the paper, we want the far away outlier points to have higher scores (just like looking at a topographical map) and the points between clusters to have lower outlier scores.
I think GLOSH is supposed to give us this instead:
It seems a fix of GLOSH may also be of help for #116. I would like to PR but I seem to have issues with building the code for now. Please let me know if there is something I might be missing.
The text was updated successfully, but these errors were encountered:
I am curious if GLOSH implementation in this repository correctly follows the paper's definition of "outlierness".
According to HDBSCAN* paper (R. J. G. B. Campello et al. 2015, page 25):
Looking at the
max_lambdas
function for computingεmax(xi)
, I think the original paper's explanation (the bold italic text above) is not correctly interpreted. It seems themax_lambdas
function only considers the death of a parent cluster (not considering the latest death of its subclusters).To reproduce this issue, please run the following code:
This should show the following plot:
As you can see from the plot, the outlier scores assigned to the data points between clusters (please find the yellow points between the clusters!) do not seem to look "natural" outliers compared to other outliers. From my own understanding of the paper, we want the far away outlier points to have higher scores (just like looking at a topographical map) and the points between clusters to have lower outlier scores.
I think GLOSH is supposed to give us this instead:
It seems a fix of GLOSH may also be of help for #116. I would like to PR but I seem to have issues with building the code for now. Please let me know if there is something I might be missing.
The text was updated successfully, but these errors were encountered: