Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed test failures caused by non-deterministic order of style map #1817

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

RugvedB
Copy link

@RugvedB RugvedB commented Nov 27, 2023

Created this PR to fix 19 flaky tests listed as follows:

  1. shouldSerialiseEventWithMultiLineStringArgumentAsAll
  2. shouldSerialiseEventWithObjectArgumentAsAll
  3. shouldSerialiseEventWithMultiLineString
  4. shouldSerialiseEventWithBecauseArgumentNotMatching
  5. shouldSerialiseEventWithBecauseArgumentAsFirst
  6. shouldSerialiseEventWithMultiLineStringArgumentAsFirst
  7. shouldSerialiseEventWithTooFewArguments
  8. shouldSerialiseBinaryForward
  9. shouldSerialiseEventWithObjectArgumentAsFirst
  10. shouldSerialiseEventWithTooManyArguments
  11. shouldSerialiseEventWithBecauseArgumentAsLast
  12. shouldSerialiseEventWithSingleLineStringArgument
  13. shouldSerialiseEventWithThrowable
  14. shouldSerialiseMessageWithException
  15. shouldSerialiseRollUpEventsWithSameCorrelationIdAndNotWarpEventsWithUniqueCorrelationId
  16. shouldSerialiseEventsWithRequest
  17. shouldSerialiseEventsWithoutFields
  18. shouldSerialiseGroupWithSingleEvent
  19. shouldSerialiseGroupWithMultipleEvents

  1. How were these test identified as flaky?
    These tests were identified as flaky by using an open-source research tool named NonDex which is responsible for finding and diagnosing non-deterministic runtime exceptions in Java programs.

  2. What was the error?
    The error was caused by non-deterministic ordering of keys in the style map, which is used to create a part of the json variable structure in the tests (here). The tests were asserting the exact order of keys in the generated JSON, including properties like paddingBottom, whiteSpace, overflow, color, and paddingTop. Since the implementation was using a HashMap to create the style map, the order of keys was not guaranteed, leading to different orders during test execution.

Common Error among all the tests mentioned below - jumbled ordering of style map's keys in the json:

Expected: is "{\n  \"key\" : \"854d6862-2ee2-4251-9586-b259f854a402_log\",\n  \"value\" : {\n    \"description\" : \"07-01 00:51:18.216 TEMPLATE_GENERATED \",\n    \"style\" : {\n      \"paddingBottom\" : \"4px\",\n      \"whiteSpace\" : \"nowrap\",\n      \"overflow\" : \"auto\",\n      \"color\" : \"rgb(241, 186, 27)\",\n      \"paddingTop\" : \"4px\"\n    },\n    \"messageParts\" : [ {\n      \"key\" : \"854d6862-2ee2-4251-9586-b259f854a402_0msg\",\n      \"value\" : \"some random\"\n    }, {\n      \"key\" : \"854d6862-2ee2-4251-9586-b259f854a402_0arg\",\n      \"multiline\" : true,\n      \"argument\" : true,\n      \"value\" : [ \"line one_one\", \"line one_two\", \"line one_three\" ]\n    }, {\n      \"key\" : \"854d6862-2ee2-4251-9586-b259f854a402_1msg\",\n      \"value\" : \"formatted string\"\n    }, {\n      \"key\" : \"854d6862-2ee2-4251-9586-b259f854a402_1arg\",\n      \"multiline\" : true,\n      \"argument\" : true,\n      \"value\" : [ \"line two_one\", \"line two_two\", \"line two_three\" ]\n    } ]\n  }\n}"
     but: was "{\n  \"key\" : \"854d6862-2ee2-4251-9586-b259f854a402_log\",\n  \"value\" : {\n    \"description\" : \"07-01 00:51:18.216 TEMPLATE_GENERATED \",\n    \"style\" : {\n      \"whiteSpace\" : \"nowrap\",\n      \"color\" : \"rgb(241, 186, 27)\",\n      \"paddingTop\" : \"4px\",\n      \"paddingBottom\" : \"4px\",\n      \"overflow\" : \"auto\"\n    },\n    \"messageParts\" : [ {\n      \"key\" : \"854d6862-2ee2-4251-9586-b259f854a402_0msg\",\n      \"value\" : \"some random\"\n    }, {\n      \"key\" : \"854d6862-2ee2-4251-9586-b259f854a402_0arg\",\n      \"multiline\" : true,\n      \"argument\" : true,\n      \"value\" : [ \"line one_one\", \"line one_two\", \"line one_three\" ]\n    }, {\n      \"key\" : \"854d6862-2ee2-4251-9586-b259f854a402_1msg\",\n      \"value\" : \"formatted string\"\n    }, {\n      \"key\" : \"854d6862-2ee2-4251-9586-b259f854a402_1arg\",\n      \"multiline\" : true,\n      \"argument\" : true,\n      \"value\" : [ \"line two_one\", \"line two_two\", \"line two_three\" ]\n    } ]\n  }\n}"
  1. What is the fix?
    To address the non-deterministic order of keys, I replaced the usage of HashMap with LinkedHashMap when creating the style map. Unlike HashMap, LinkedHashMap preserves the order of keys based on their insertion order. By explicitly ordering the keys in the style map according to the expected order, we can ensure consistent ordering during each test run. This PR proposes changing HashMap to LinkedHashMap here and adding elements to style map in the expected order to fix all the listed tests.

You can run the following commands to run the 19 tests using NonDex tool respectively:

mvn -pl mockserver-netty edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.mockserver.dashboard.serializers.DashboardLogEntryDTOSerializerTest#shouldSerialiseEventWithMultiLineStringArgumentAsAll
mvn -pl mockserver-netty edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.mockserver.dashboard.serializers.DashboardLogEntryDTOSerializerTest#shouldSerialiseEventWithObjectArgumentAsAll
mvn -pl mockserver-netty edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.mockserver.dashboard.serializers.DashboardLogEntryDTOSerializerTest#shouldSerialiseEventWithMultiLineString
mvn -pl mockserver-netty edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.mockserver.dashboard.serializers.DashboardLogEntryDTOSerializerTest#shouldSerialiseEventWithBecauseArgumentNotMatching 
mvn -pl mockserver-netty edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.mockserver.dashboard.serializers.DashboardLogEntryDTOSerializerTest#shouldSerialiseEventWithBecauseArgumentAsFirst
mvn -pl mockserver-netty edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.mockserver.dashboard.serializers.DashboardLogEntryDTOSerializerTest#shouldSerialiseEventWithMultiLineStringArgumentAsFirst
mvn -pl mockserver-netty edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.mockserver.dashboard.serializers.DashboardLogEntryDTOSerializerTest#shouldSerialiseEventWithTooFewArguments
mvn -pl mockserver-netty edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.mockserver.dashboard.serializers.DashboardLogEntryDTOSerializerTest#shouldSerialiseBinaryForward
mvn -pl mockserver-netty edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.mockserver.dashboard.serializers.DashboardLogEntryDTOSerializerTest#shouldSerialiseEventWithObjectArgumentAsFirst
mvn -pl mockserver-netty edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.mockserver.dashboard.serializers.DashboardLogEntryDTOSerializerTest#shouldSerialiseEventWithTooManyArguments
mvn -pl mockserver-netty edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.mockserver.dashboard.serializers.DashboardLogEntryDTOSerializerTest#shouldSerialiseEventWithBecauseArgumentAsLast
mvn -pl mockserver-netty edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.mockserver.dashboard.serializers.DashboardLogEntryDTOSerializerTest#shouldSerialiseEventWithSingleLineStringArgument
mvn -pl mockserver-netty edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.mockserver.dashboard.serializers.DashboardLogEntryDTOSerializerTest#shouldSerialiseEventWithThrowable
mvn -pl mockserver-netty edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.mockserver.dashboard.DashboardWebSocketHandlerTest#shouldSerialiseMessageWithException
mvn -pl mockserver-netty edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.mockserver.dashboard.DashboardWebSocketHandlerTest#shouldSerialiseRollUpEventsWithSameCorrelationIdAndNotWarpEventsWithUniqueCorrelationId
mvn -pl mockserver-netty edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.mockserver.dashboard.DashboardWebSocketHandlerTest#shouldSerialiseEventsWithRequest
mvn -pl mockserver-netty edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.mockserver.dashboard.DashboardWebSocketHandlerTest#shouldSerialiseEventsWithoutFields
mvn -pl mockserver-netty edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.mockserver.dashboard.serializers.DashboardLogEntryDTOGroupSerializerTest#shouldSerialiseGroupWithSingleEvent
mvn -pl mockserver-netty edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.mockserver.dashboard.serializers.DashboardLogEntryDTOGroupSerializerTest#shouldSerialiseGroupWithMultipleEvents

Test Environment:

java version 11.0.19
Apache Maven 3.9.5

Kindly let me know if this fix is acceptable.
Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants