CI: lower FER/BER is always OK for regression testing. #185
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Improve PASS rate of the regression testsuite by accepting lower error rates, i.e. lower BER/FER vs. reference is always OK. Signed-off-by: Gerhard Baer <Gerhard.Baer@protonmail.com>
|
I'm not very convinced by this :-/. Normally to avoid this problem, in the GitLab CI, we enabled the retry feature. Otherwise we can increase the required number of errors. The thing is: sometime, when the BER/FER increases it could be because of a bug. |
|
I agree that increasing BER/FER could be due to a bug. There is no change wrt. that case. However, decreasing BER/FER could actually mean an improvement and/or a bug fixed, isn't it? |
|
Sorry it was not clear: I meant even if the BER/FER decreases it could be because of a bug (or a improvement) but it means that something changed in the code and we want to be notified to understand the change. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some AVX2 tests, and even SSE4 tests, randomly fail because the resulting error rates are lower than the reference. IMHO, lower error rates (BER, FER) should be accepted. Therefore, I am suggesting this patch to improve the PASS rate and accept lower BER/FER than reference.
Note: ignoring the lower error rates involve setting
diffto zero. I am not quite sure of the expected behavior wrt. the sensibility computations. Otherwise, forLDPCtests, I need to set sensibility to 2.75 and weak rate to 0.87. Likewise, forTURBO_PRODtests, sensibility need to be increased to 2.75 too. Probably not a big deal in the end, but you get the idea. :)