How to get the silhouette score for an sktime clusterer`?

[fkiraly]

Answering a question from discord, "love the lib, quick questions with timeserieskmeans how do we get the silhouette scores?"

My answer:

To get the silhouette score:

• get the labels from the estimator.labels_ attribute
• get sklearn.metrics.silhouette_score
• get the time series distance you used from sktime.dists_kernels, e.g., from sktime.dists_kernels.dtw import DtwDistTslearn. Instantiate it, e.g., dist = DtwDistTslearn()
• compute the distance matrix on the data, dist_mat = dist.transform(X)
• apply the silhouette score to the distance matrix, silhouette_score(dist_mat, labels, metric="precomputed")

In one code block:

from sklearn.metrics import silhouette_score
from sktime.dists_kernels.dtw  import DtwDistTslearn  # or other distance, to use for silhouette score

# you probably did this already
my_clusterer.fit(X)

# labels are in this attribute (if estimator supports)
labels = my_clusterer.labels_

# get the distance
# the silhouette score depends on a distance, this can be but need not be the same as used in the clusterer
dist = DtwDistTslearn()

# compute distance matrix
dist_mat = dist.transform(X)

# compute silhouette score
silhouette_score(dist_mat, labels, metric="precomputed")

[justkrismanohar]

awesome, I think it should be from sktime.dists_kernels.dtw import DtwDistTslearn i.e., underscore instead of ., will let you know how i get on, its been running 15mins now, with like 20k timeseries in PySpark, but took like 2 mins to train with k_means.fit(X) is that normal? Seems like it's taking too long, even for the all-pairs calculation.

[fkiraly]

I believe it runs the naive algorithm which scales with "number of time series squared", see https://en.wikipedia.org/wiki/Silhouette_(clustering)#Simplified_Silhouette_and_Medoid_Silhouette.

I may be wrong though, I would also be interested in a definite answer.

So, if 300 time series take about 50sec, then at squared scaling one would expect 20k time series to take 50sec * (20.000 / 300) ** 2 = ca 61 hours.

[fkiraly]

I believe the sample_size argument for the silhouette score exists for that reason, you can subsample at random and this will be a consistent estimator, with variance decreasing linearly in sub-sample size.

[fkiraly]

I think it should be from sktime.dists_kernels.dtw import DtwDistTslearn

yes, thanks, fixed it in the original code.

[justkrismanohar]

@fkiraly seems like its the dist_mat = dist.transform(X) call that is taking most of the time. I separated that call and the silhouette_score(dist_mat, labels, metric="precomputed") to confirm. Not sure what dist.transform(X) is doing but it is taking ages. Any way we can just get the distances used by k_means.fit and just pass that into the silhouette call?

[fkiraly]

Not sure what dist.transform(X) is doing

It's computing the pairwise distance between every pair of time series, so it scales with n-squared.

Any way we can just get the distances used by k_means.fit and just pass that into the silhouette call?

I think that won't work, the silhouette algorithm requires each pairwise distance, afaik.

You could subsample manually though.

[justkrismanohar]

Question: @fkiraly, so without silhouette scoring, how did you all test to ensure the TS clustering gave good results? Maybe I can use that?

[fkiraly]

afaik we do not test performance - philosophically, we leave it up to the implementer to produce a good or bad algorithm, and up to the user to select the one they need. After all, "good" and "bad" strongly depend on scenario, use case, etc.