[TRACKER] Flaky Tests

[codeadict]

Let's keep track of random test failures here. A stable CI pipeline is the key to having safe releases.:

• Test at vmq_bridge_SUITE:buffer_outgoing_test Is failing with unexpected message mostly in OTP 24. Traceback:

  %%% vmq_bridge_SUITE ==> {{badmatch,{{mqtt_subscribe,4,[{[<<"bridge">>,<<"#">>],0}]},<<48,16,0>>}},
  [{packet,expect_packet,5,
           [{file,"/Users/runner/work/vernemq/vernemq/apps/vmq_commons/src/packet.erl"},
            {line,43}]},
   {vmq_bridge_SUITE,'-buffer_outgoing_test/1-fun-0-',2,
                     [{file,"/Users/runner/work/vernemq/vernemq/apps/vmq_bridge/test/vmq_bridge_SUITE.erl"},
                      {line,344}]},
   {vmq_bridge_SUITE,buffer_outgoing_test,1,
                     [{file,"/Users/runner/work/vernemq/vernemq/apps/vmq_bridge/test/vmq_bridge_SUITE.erl"},
                      {line,350}]},
   {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1783}]},
   {test_server,run_test_case_eval1,6,[{file,"test_server.erl"},{line,1292}]},
   {test_server,run_test_case_eval,9,[{file,"test_server.erl"},{line,1224}]}]}

• In OTP 25 both Linux and Mac there is a timetrap failure in the vmq_swc_SUITE:

   %%% vmq_swc_store_SUITE: .
   %%% vmq_swc_store_SUITE ==> leveldb.partitioned_cluster_test: SKIPPED
   %%% vmq_swc_store_SUITE ==> {tc_auto_skip,
       {failed,
           {vmq_swc_store_SUITE,init_per_testcase,{timetrap_timeout,1800000}}}}
   .......
   [os_mon] memory supervisor port (memsup): Erlang has closed
   [os_mon] cpu supervisor port (cpu_sup): Erlang has closed
   %%% vmq_webhooks_SUITE: .........................................
   ````
  

• Random failure in vmq_retain_SUITE: retain_qos0_clear_test which looks like an indication or a bug in the packet parsing:

   %%% vmq_retain_SUITE ==> mqttv5.retain_qos0_clear_test: FAILED
   %%% vmq_retain_SUITE ==> {{badmatch,more},
    [{packetv5,expect_frame,4,
               [{file,"/home/runner/work/vernemq/vernemq/apps/vmq_commons/src/packetv5.erl"},
                {line,175}]},
     {vmq_retain_SUITE,retain_qos0_clear_test,1,
                       [{file,"/home/runner/work/vernemq/vernemq/apps/vmq_server/test/vmq_retain_SUITE.erl"},
                        {line,156}]},
     {test_server,ts_tc,3,[{file,"test_server.erl"},{line,1784}]},

[codeadict]

@ioolkos - Can you please Pin this issue at the bottom right of it? This way we can keep track of unreliable tests and fix them one by one