Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues with the GOSPA metric #999

Open
narykov opened this issue Apr 29, 2024 · 4 comments
Open

Issues with the GOSPA metric #999

narykov opened this issue Apr 29, 2024 · 4 comments

Comments

@narykov
Copy link

narykov commented Apr 29, 2024

I ran into a couple of problems trying to use GOSPA in place of the OSPA metric:

  1. While compute_OSPA_distance returns a scalar, compute_gospa_metric is designed to return a dictionary of values (including that of distance). This leads to problems with visualising the GOSPA errors using built-in tools, as scalar values are anticipated.
  2. In the Metrics example, it is not possible to replace OSPA with GOSPA metric (e.g., via from stonesoup.metricgenerator.ospametric import GOSPAMetric as OSPAMetric) as it leads to issues when trying to extract a timestamp from a numpy array of Particle elements.

Perhaps, some of these issues are known (at least I heard about the second one from @A-acuto), but I couldn't find them explicitly outlined here.

@sdhiscocks
Copy link
Member

I think issue 2. was fixed in #967

@A-acuto
Copy link
Contributor

A-acuto commented Apr 30, 2024

@sdhiscocks Thanks for pointing out that! I was not aware, I found that issue some time ago and I forgot about mentioning before. Glad it is fixed now

@simonrmaskell
Copy link
Collaborator

If we want a common visualisation process that is agnostic to whether we use GOSPA or OSPA, I think there is still an issue here. Do we want the OSPA output to look like GOSPA's or vice versa? I don't think we want to persist the current status quo

@narykov
Copy link
Author

narykov commented May 29, 2024

While compute_OSPA_distance returns a scalar, compute_gospa_metric is designed to return a dictionary of values (including that of distance). This leads to problems with visualising the GOSPA errors using built-in tools, as scalar values are anticipated.

To get any visualisation, I 'hacked' computed_over_time() in GOSPA by putting metric.value = metric.value['distance'] above line 196.

metric.value['switching'] = switching_metric.loss()
metric.value['distance'] = np.power(metric.value['distance']**self.alpha +
metric.value['switching']**self.alpha,
1.0/self.alpha)
gospa_metrics.append(metric)

This way I only access 'distance' metric, which then can be visualised. By replacing 'distance' with 'localisation', 'missed', or 'false', it is (potentially) possible to get access to other information that is particular to GOSPA.

P.S. Visualisation of the GOSPA metric is accessed via title='GOSPA Metric', and of the OSPA metric via title='OSPA distances', which is not very intuitive.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants