Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update benchmarking example #193

Open
jscanass opened this issue Oct 4, 2022 · 1 comment
Open

Update benchmarking example #193

jscanass opened this issue Oct 4, 2022 · 1 comment

Comments

@jscanass
Copy link

jscanass commented Oct 4, 2022

I installed the package as recommend using
pip install carla-recourse
The version installed is carla-recourse==0.0.5 but I am not able to reproduce the benchmarking example. The error happen when importing evaluation module:
Screen Shot 2022-10-04 at 11 27 42

@jscanass jscanass changed the title Update benchmarking examplle Update benchmarking example Oct 7, 2022
@xueyagaga
Copy link

I installed the package as recommend using pip install carla-recourse The version installed is carla-recourse==0.0.5 but I am not able to reproduce the benchmarking example. The error happen when importing evaluation module: Screen Shot 2022-10-04 at 11 27 42

I also meet this BUG. The solution is easy. First, delete these codes (if you don't need the specific results):

import carla.evaluation.catalog as evaluation_catalog
evaluation_measures = [
    evaluation_catalog.YNN(benchmark.mlmodel, {"y": 5, "cf_label": 1}),
    evaluation_catalog.Distance(benchmark.mlmodel),
    evaluation_catalog.SuccessRate(),
    evaluation_catalog.Redundancy(benchmark.mlmodel, {"cf_label": 1}),
    evaluation_catalog.ConstraintViolation(benchmark.mlmodel),
    evaluation_catalog.AvgTime({"time": benchmark.timer}),
]

Second, change results = benchmark.run_benchmark(evaluation_measures) to:
results = benchmark.run_benchmark()
Then you can get all the metrics, such as:
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants