test(changelog): refactor changelog validation #809
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
codejedi365
force-pushed
the
test/refactor-changelog-validation
branch
5 times, most recently
from
e27c7da
to
475bf52
Compare
codejedi365
force-pushed
the
test/refactor-changelog-validation
branch
2 times, most recently
from
02a438d
to
6f88e0d
Compare
Drop the test related changelog template as that is one more thing to maintain and does not actually match the embedded template provided to the user. Secondly, drop the context unit test as it does not provide value that the template ultimately will fail in other unit tests.
Drop the test related release notes template as that is one more thing to maintain and use the one provided to the user.
codejedi365
force-pushed
the
test/refactor-changelog-validation
branch
from
6f88e0d
to
d6e15fe
Compare
codejedi365
force-pushed
the
test/refactor-changelog-validation
branch
from
d6c2106
to
0347dac
Compare
codejedi365
force-pushed
the
test/refactor-changelog-validation
branch
from
c65ed3b
to
a183229
Compare
|
@relekang, ready for review |
relekang
approved these changes
Mar 18, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Rationale
While I've been working on testing some improvements to the changelog functionality, I stumbled across some logic bugs within the testing framework which were producing false positives. One such example is the regenerate_content from the changelog test.
Refactored some assertions so that the failure is readable and immediately identifiable in the logs rather than obscured.
I identified & fixed that some of the angular commit messages that syntactically had the scope in the wrong spot.
I ensured that changelogs are actually generated (the empty template directory disabled all default changelog creations) both before the tests and then after the tests.
The strategy I employ here when refactoring the testing for changelogs is to identify what should go as a unit test and what is a scenario based system test. Personally, I value system tests way more than unit tests. I believe that unit tests really should just focus on "prove it works" with basic default values. The scenarios should be extensive to try all variations as this will catch the most bugs that our users will actually encounter. I have been burned many times when the unit tests don't actually catch the problems users have. How does this apply to this PR? I removed more parametrize values from unit tests and instead beef up the scenario testing. One reason I think this applies more here is that the outputs are related to changelog and release notes is that these are the outputs the user receives from the entire program which should be vetted thoroughly as opposed to an internal function. If you disagree as I realize I am opinionated on this, then I can adjust accordingly.
How I tested
Related to purpose 3, If you try to execute the test cases on a machine in the Eastern Standard Time (EST) zone after 3PM, all the changelogs tests will fail due to a date mix match. My first assumption was UTC time, however, EST is only 4 or 5 hours off of UTC so at 3PM even UTC isn't on the next day. But if you interpret a timedelta incorrectly from git as PSR was you instead add 5 hours to UTC time, then you find yourself at 1AM (UTC+5) and Ah Ha!