Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Status comparison for diff tables does not take category into account #969

Open
PhilippWendler opened this issue Nov 19, 2023 · 0 comments

Comments

@PhilippWendler
Copy link
Member

If we compute a diff table by comparing the status column (the default mode for diff tables), only the status itself is taken into account, not the associated category. Usually, this is no problem because for a given tuple (task, status) the category is always the same. However, in some scenarios additional categories (correct-unconfirmed) are introduced into the result files (from outside of BenchExec) and then this no longer applies. But the diff table will omit the columns where the category is different but the status is the same.

Given that we provide some support for the category correct-unconfirmed and e.g. compute statistics for it, we should also take it into account for the diff table. Comparing (status, category) by default instead of just status is probably a good default.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

1 participant