Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spec instabilities #1871

Open
mensfeld opened this issue Jan 26, 2024 · 10 comments
Open

Spec instabilities #1871

mensfeld opened this issue Jan 26, 2024 · 10 comments
Assignees
@mensfeld
Labels
internal Things that are needed but do not change any user-facing APIs specs Extra unit or integration specs

Comments

@mensfeld
Copy link
Member

mensfeld commented Jan 26, 2024 •
edited

This PR aggregates the CI server loop failures coming from minor instabilities in specs. They never or almost never occur in actions and only occur when running specs 24/7
@mensfeld mensfeld added specs Extra unit or integration specs internal Things that are needed but do not change any user-facing APIs labels Jan 26, 2024
@mensfeld mensfeld self-assigned this Jan 26, 2024
@mensfeld mensfeld changed the title Spec instability Spec instabilities Jan 27, 2024
@mensfeld
Copy link
Member Author 

I, [2024-02-11T12:02:34.251988 #2160676]  INFO -- : [e86be8d811b9] Consume job for Consumer on 4cd58ae4f63aa6ba/0 finished in 0.1417860984802246ms
D, [2024-02-11T12:02:34.252176 #2160676] DEBUG -- : [9077dc2a3590] Polling messages...
I, [2024-02-11T12:02:34.268193 #2160676]  INFO -- : Stopping Karafka server
I, [2024-02-11T12:02:34.269140 #2160676]  INFO -- : [9077dc2a3590] Polled 2 messages in 16.825072050094604ms
I, [2024-02-11T12:02:34.269586 #2160676]  INFO -- : [e86be8d811b9] Consume job for Consumer on 4cd58ae4f63aa6ba/0 started
I, [2024-02-11T12:02:34.269950 #2160676]  INFO -- : [e86be8d811b9] Consume job for Consumer on 4cd58ae4f63aa6ba/0 finished in 0.2063610553741455ms
I, [2024-02-11T12:02:34.270415 #2160676]  INFO -- : [e86be8d811b9] Shutdown job for Consumer on 4cd58ae4f63aa6ba/0 started
I, [2024-02-11T12:02:34.272363 #2160676]  INFO -- : [e86be8d811b9] Shutdown job for Consumer on 4cd58ae4f63aa6ba/0 finished in 0.0561070442199707ms
I, [2024-02-11T12:02:34.357346 #2160676]  INFO -- : [f4624191acb1] Sync producing of a message to '4cd58ae4f63aa6ba' topic took 10.468673944473267 ms
D, [2024-02-11T12:02:34.357537 #2160676] DEBUG -- : [f4624191acb1] {:partition=>0, :topic=>"4cd58ae4f63aa6ba", :payload=>"1"}
I, [2024-02-11T12:02:34.468338 #2160676]  INFO -- : [f4624191acb1] Sync producing of a message to '4cd58ae4f63aa6ba' topic took 10.382905960083008 ms
D, [2024-02-11T12:02:34.468466 #2160676] DEBUG -- : [f4624191acb1] {:partition=>0, :topic=>"4cd58ae4f63aa6ba", :payload=>"1"}
I, [2024-02-11T12:02:34.579161 #2160676]  INFO -- : [f4624191acb1] Sync producing of a message to '4cd58ae4f63aa6ba' topic took 10.363621950149536 ms
D, [2024-02-11T12:02:34.579307 #2160676] DEBUG -- : [f4624191acb1] {:partition=>0, :topic=>"4cd58ae4f63aa6ba", :payload=>"1"}
I, [2024-02-11T12:02:34.690793 #2160676]  INFO -- : [f4624191acb1] Sync producing of a message to '4cd58ae4f63aa6ba' topic took 7.949773073196411 ms
D, [2024-02-11T12:02:34.690868 #2160676] DEBUG -- : [f4624191acb1] {:partition=>0, :topic=>"4cd58ae4f63aa6ba", :payload=>"1"}
I, [2024-02-11T12:02:34.769552 #2160676]  INFO -- : Stopped Karafka server
I, [2024-02-11T12:02:34.769627 #2160676]  INFO -- : [f4624191acb1] Closing producer
I, [2024-02-11T12:02:34.839359 #2160676]  INFO -- : [f4624191acb1] Closing producer took 69.67589211463928 ms
/home/deploy/karafka/spec/integrations_helper.rb:346:in `assert_equal': {:offsets=>[0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3], :split=>[8]} (AssertionFailedError)
        from /home/deploy/karafka/spec/integrations/pro/consumption/strategies/mom/with_manual_offset_and_error_spec.rb:57:in `<main>'

@mensfeld
Copy link
Member Author 

, [2024-02-21T02:02:38.211973 #21848]  INFO -- : Stopped Karafka server
I, [2024-02-21T02:02:38.212061 #21848]  INFO -- : [1a826764d0ea] Closing producer
I, [2024-02-21T02:02:38.225915 #21848]  INFO -- : [1a826764d0ea] Closing producer took 13.850955999922007 ms
/home/runner/work/karafka/karafka/spec/integrations_helper.rb:363:in `assert_equal': {:iterator=>#<Enumerator: 0..9:cycle>, :done=>[3520, 3540, 3560, 3580, 3600, 3620, 3640, 3660, 3680, 3700], :last=>["9", "19", "29", "39", "49", "59", "69", "79", "89", "99", "4", "14", "24", "34", "44", "54", "64", "74", "84", "94", "7", "17", "27", "37", "47", "57", "67", "77", "87", "97", "6", "16", "26", "36", "46", "56", "66", "76", "86", "96", "5", "15", "25", "35", "45", "55", "65", "75", "85", "95", "1", "11", "21", "31", "41", "51", "61", "71", "81", "91", "3", "13", "23", "33", "43", "53", "63", "73", "83", "93", "0", "10", "20", "30", "40", "50", "60", "70", "80", "90", "2", "12", "22", "32", "42", "52", "62", "72", "82", "92", "8", "18", "28", "38", "48", "58", "68", "78", "88", "98"], :metadata=>["92", "92", "92", "92", "92", "92", "92", "92", "92", "92"]} (AssertionFailedError)
	from /home/runner/work/karafka/karafka/spec/integrations/pro/consumption/transactions/vps/all_ok_current_spec.rb:56:in `<main>'

@mensfeld
Copy link
Member Author

mensfeld commented Mar 1, 2024 

Run bin/integrations --exclude '/pro'
Random seed: 9896
.............................................................................................................................................................................................
[FAILED] swarm/forceful_nodes_restart_graceful_supervisor_stop_spec.rb
Time taken: 120.49680681099994 seconds
Exit code: 2
Exit status: pid 12697 exit 2
I, [2024-03-01T01:39:59.069755 #12699]  INFO -- : [a2e3c58826ce] Sync producing of 1 messages to 1 topics took 11.00485300004948 ms
D, [2024-03-01T01:39:59.069855 #12699] DEBUG -- : [a2e3c58826ce] [{:topic=>"7579ad9412afe443", :payload=>"91aeb5a3-6cd5-4916-87d8-62f2c708b915"}]
I, [2024-03-01T01:39:59.070048 #12699]  INFO -- : [a2e3c58826ce] Closing producer
...
I, [2024-03-01T01:39:59.069755 #12699]  INFO -- : [a2e3c58826ce] Sync producing of 1 messages to 1 topics took 11.00485300004948 ms
D, [2024-03-01T01:39:59.069855 #12699] DEBUG -- : [a2e3c58826ce] [{:topic=>"7579ad9412afe443", :payload=>"91aeb5a3-6cd5-4916-87d8-62f2c708b915"}]
I, [2024-03-01T01:39:59.070048 #12699]  INFO -- : [a2e3c58826ce] Closing producer
I, [2024-03-01T01:39:59.160641 #12699]  INFO -- : [a2e3c58826ce] Closing producer took 90.56184199999552 ms
D, [2024-03-01T01:39:59.181004 #12699] DEBUG -- : Swarm manager starting node with id: 0
I, [2024-03-01T01:39:59.183914 #12746]  INFO -- : Swarm node 12746 forked from 12699
I, [2024-03-01T01:39:59.186003 #12746]  INFO -- : Running in ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux]
I, [2024-03-01T01:39:59.186034 #12746]  INFO -- : Running Karafka 2.3.4 server
I, [2024-03-01T01:39:59.186046 #12746]  INFO -- : See LICENSE and the LGPL-3.0 for licensing details
D, [2024-03-01T01:40:00.183576 #12699] DEBUG -- : Swarm manager checking nodes: 12746
D, [2024-03-01T01:40:01.183935 #12699] DEBUG -- : Swarm manager checking nodes: 12746
E, [2024-03-01T01:40:02.184419 #12699] ERROR -- : Swarm manager detected unhealthy node 12746. Sending TERM signal...
D, [2024-03-01T01:40:02.184503 #12699] DEBUG -- : Swarm manager checking nodes: 12746
I, [2024-03-01T01:40:02.184668 #12746]  INFO -- : Received SIGTERM system signal
I, [2024-03-01T01:40:02.184794 #12746]  INFO -- : Stopping Karafka server
D, [2024-03-01T01:40:03.184726 #12699] DEBUG -- : Swarm manager checking nodes: 12746
D, [2024-03-01T01:40:04.184934 #12699] DEBUG -- : Swarm manager checking nodes: 12746
D, [2024-03-01T01:40:05.185157 #12699] DEBUG -- : Swarm manager checking nodes: 12746
D, [2024-03-01T01:40:06.185360 #12699] DEBUG -- : Swarm manager checking nodes: 12746
E, [2024-03-01T01:40:07.185667 #12699] ERROR -- : Swarm manager detected unresponsive node 12746. Sending KILL signal...
D, [2024-03-01T01:40:07.185742 #12699] DEBUG -- : Swarm manager checking nodes: 12746
I, [2024-03-01T01:40:07.189151 #12699]  INFO -- : Received SIGCHLD system signal
D, [2024-03-01T01:40:07.189302 #12699] DEBUG -- : Swarm manager checking nodes: 12746
D, [2024-03-01T01:40:08.189485 #12699] DEBUG -- : Swarm manager starting node with id: 0
D, [2024-03-01T01:40:08.191225 #12699] DEBUG -- : Swarm manager checking nodes: 12854
I, [2024-03-01T01:40:08.192654 #12854]  INFO -- : Swarm node 12854 forked from 12699
I, [2024-03-01T01:40:08.194251 #12854]  INFO -- : Running in ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux]
I, [2024-03-01T01:40:08.194290 #12854]  INFO -- : Running Karafka 2.3.4 server
I, [2024-03-01T01:40:08.194308 #12854]  INFO -- : See LICENSE and the LGPL-3.0 for licensing details
D, [2024-03-01T01:40:09.191506 #12699] DEBUG -- : Swarm manager checking nodes: 12854
D, [2024-03-01T01:40:10.191768 #12699] DEBUG -- : Swarm manager checking nodes: 12854
E, [2024-03-01T01:40:11.192133 #12699] ERROR -- : Swarm manager detected unhealthy node 12854. Sending TERM signal...
D, [2024-03-01T01:40:11.192271 #12699] DEBUG -- : Swarm manager checking nodes: 12854
I, [2024-03-01T01:40:11.192489 #12854]  INFO -- : Received SIGTERM system signal
I, [2024-03-01T01:40:11.192772 #12854]  INFO -- : Stopping Karafka server
I, [2024-03-01T01:40:11.192617 #12854]  INFO -- : Received SIGTERM system signal
D, [2024-03-01T01:40:12.192470 #12699] DEBUG -- : Swarm manager checking nodes: 12854
D, [2024-03-01T01:40:13.192694 #12699] DEBUG -- : Swarm manager checking nodes: 12854
D, [2024-03-01T01:40:14.192938 #12699] DEBUG -- : Swarm manager checking nodes: 12854
D, [2024-03-01T01:40:15.193173 #12699] DEBUG -- : Swarm manager checking nodes: 12854
E, [2024-03-01T01:40:16.193509 #12699] ERROR -- : Swarm manager detected unresponsive node 12854. Sending KILL signal...
D, [2024-03-01T01:40:16.193610 #12699] DEBUG -- : Swarm manager checking nodes: 12854
I, [2024-03-01T01:40:16.195802 #12699]  INFO -- : Received SIGCHLD system signal
D, [2024-03-01T01:40:16.195963 #12699] DEBUG -- : Swarm manager checking nodes: 12854
D, [2024-03-01T01:40:17.196140 #12699] DEBUG -- : Swarm manager starting node with id: 0
D, [2024-03-01T01:40:17.198034 #12699] DEBUG -- : Swarm manager checking nodes: 12976
I, [2024-03-01T01:40:17.208245 #12699]  INFO -- : Received SIGTERM system signal
I, [2024-03-01T01:40:17.208426 #12699]  INFO -- : Stopping Karafka server
E, [2024-03-01T01:41:58.324978 #12699] ERROR -- : Forceful Karafka server stop
I, [2024-03-01T01:41:58.326820 #12699]  INFO -- : Received SIGCHLD system signal
I, [2024-03-01T01:41:58.425383 #12699]  INFO -- : Stopped Karafka server


..........................................................................................................................................................

@mensfeld
Copy link
Member Author

mensfeld commented Mar 7, 2024 

I, [2024-03-07T06:26:52.735491 #16466]  INFO -- : [849f957cfa52] Shutdown job for Consumer on 0a41f6e8d93bf5f2/0 finished in 0.017302999971434474ms
I, [2024-03-07T06:26:53.443920 #16466]  INFO -- : Stopped Karafka server
I, [2024-03-07T06:26:53.444002 #16466]  INFO -- : [e9008b4ee353] Closing producer
I, [2024-03-07T06:26:53.509218 #16466]  INFO -- : [e9008b4ee353] Closing producer took 65.20574000000488 ms
/home/runner/work/karafka/karafka/spec/integrations_helper.rb:368:in `assert_equal': {"rebalance.partitions_assign"=>[["unique_name_a_c4ca4238a0b9_0", 1709792811.1235788], ["unique_name_b_c81e728d9d4c_1", 1709792812.2057123]], "rebalance.partitions_assigned"=>[["unique_name_a_c4ca4238a0b9_0", 1709792811.1237726], ["unique_name_b_c81e728d9d4c_1", 1709792812.2058632]], :rebalanced=>[true], "rebalance.partitions_revoke"=>[["unique_name_b_c81e728d9d4c_1", 1709792813.2118719], ["unique_name_a_c4ca4238a0b9_0", 1709792813.4411252]], "rebalance.partitions_revoked"=>[["unique_name_b_c81e728d9d4c_1", 1709792813.2125385], ["unique_name_a_c4ca4238a0b9_0", 1709792813.4419413]]} (AssertionFailedError)
	from /home/runner/work/karafka/karafka/spec/integrations_helper.rb:376:in `assert'
	from /home/runner/work/karafka/karafka/spec/integrations/rebalancing/lifecycle_events_spec.rb:79:in `block in <main>'
	from /home/runner/work/karafka/karafka/spec/integrations/rebalancing/lifecycle_events_spec.rb:67:in `each'
	from /home/runner/work/karafka/karafka/spec/integrations/rebalancing/lifecycle_events_spec.rb:67:in `<main>'

@mensfeld
Copy link
Member Author 

Failures:

  1) Rdkafka::Admin#ACL tests #describe_acl create acls and describe the newly created acls
     Failure/Error: expect(describe_acl_report.acls.length).to eq(2)

       expected: 2
            got: 0

       (compared using ==)
     # ./spec/rdkafka/admin_spec.rb:551:in `block (4 levels) in <top (required)>'
     # ./spec/spec_helper.rb:154:in `block (3 levels) in <top (required)>'
     # ./spec/spec_helper.rb:153:in `block (2 levels) in <top (required)>'

@mensfeld
Copy link
Member Author

mensfeld commented Apr 8, 2024 

 /home/runner/work/karafka/karafka/spec/integrations_helper.rb:386:in `assert_equal': {0=>[[360, 0.992173976], [425, 1.270734662], [580, 0.990429862], [645, 0.991673659], [710, 0.990338508], [725, 0.816608926], [735, 0.987613629], [795, 0.999672706], [860, 0.997020233], [925, 0.994553094], [990, 0.992248526], [1055, 0.989964098], [1120, 0.987371535], [1180, 0.999562498], [1245, 0.997058409], [1310, 0.99417685], [1375, 0.992898482], [1440, 0.991848299], [1505, 0.990087855], [1570, 0.988233265], [1635, 0.98771777], [1700, 0.985114093], [1760, 0.999382841], [1825, 0.997103674], [1890, 0.993899132], [1955, 0.993048008], [2020, 0.991879779], [2085, 0.988571856], [2150, 0.987972893], [2215, 0.984743275], [2275, 0.997703351], [2340, 0.993961762], [2405, 0.990902412], [2470, 0.98699265], [2535, 0.985160873], [2595, 0.999377293], [2660, 0.996967191], [2725, 0.995162191], [2790, 0.993499789], [2855, 0.99065317], [2920, 0.988739503], [2985, 0.986760729], [3045, 0.998926756], [3110, 0.997414245], [3175, 0.99842744], [3240, 0.998779862], [3305, 0.997238944], [3370, 0.995226806], [3430, 0.989000656], [3495, 0.985589808]]} (AssertionFailedError)
	from /home/runner/work/karafka/karafka/spec/integrations_helper.rb:394:in `assert'
	from /home/runner/work/karafka/karafka/spec/integrations/pro/consumption/strategies/ftr/exg/with_some_expiring_messages_spec.rb:38:in `<main>'

@mensfeld
Copy link
Member Author 

[FAILED] swarm/forceful_nodes_restart_graceful_supervisor_stop_spec.rb
Time taken: 120.21547053400002 seconds
Exit code: 2
Exit status: pid 15580 exit 2
I, [2024-04-16T16:09:17.368517 #15582]  INFO -- : [585c86a6f524] Sync producing of 1 messages to 1 topics took 8.265069999964908 ms
D, [2024-04-16T16:09:17.368602 #15582] DEBUG -- : [585c86a6f524] [{:topic=>"500c2d6804e70c0a", :payload=>"aa280444-3c2e-4126-b508-e787e431f799"}]
I, [2024-04-16T16:09:17.368747 #15582]  INFO -- : [585c86a6f524] Closing producer
I, [2024-04-16T16:09:17.461504 #15582]  INFO -- : [585c86a6f524] Closing producer took 92.7445300000254 ms
...
I, [2024-04-16T16:09:17.368517 #15582]  INFO -- : [585c86a6f524] Sync producing of 1 messages to 1 topics took 8.265069999964908 ms
D, [2024-04-16T16:09:17.368602 #15582] DEBUG -- : [585c86a6f524] [{:topic=>"500c2d6804e70c0a", :payload=>"aa280444-3c2e-4126-b508-e787e431f799"}]
I, [2024-04-16T16:09:17.368747 #15582]  INFO -- : [585c86a6f524] Closing producer
I, [2024-04-16T16:09:17.461504 #15582]  INFO -- : [585c86a6f524] Closing producer took 92.7445300000254 ms
D, [2024-04-16T16:09:17.480966 #15582] DEBUG -- : Swarm manager starting node with id: 0
I, [2024-04-16T16:09:17.484341 #15616]  INFO -- : Swarm node 15616 forked from 15582
I, [2024-04-16T16:09:17.485748 #15616]  INFO -- : Running in ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux]
I, [2024-04-16T16:09:17.485779 #15616]  INFO -- : Running Karafka 2.4.0.beta2 server
I, [2024-04-16T16:09:17.485796 #15616]  INFO -- : See LICENSE and the LGPL-3.0 for licensing details
D, [2024-04-16T16:09:18.483861 #15582] DEBUG -- : Swarm manager checking nodes: 15616
D, [2024-04-16T16:09:19.484207 #15582] DEBUG -- : Swarm manager checking nodes: 15616
E, [2024-04-16T16:09:20.484587 #15582] ERROR -- : Swarm manager detected unhealthy node 15616. Sending TERM signal...
D, [2024-04-16T16:09:20.484665 #15582] DEBUG -- : Swarm manager checking nodes: 15616
I, [2024-04-16T16:09:20.484815 #15616]  INFO -- : Received SIGTERM system signal
I, [2024-04-16T16:09:20.484931 #15616]  INFO -- : Stopping Karafka server
D, [2024-04-16T16:09:21.484868 #15582] DEBUG -- : Swarm manager checking nodes: 15616
D, [2024-04-16T16:09:22.485066 #15582] DEBUG -- : Swarm manager checking nodes: 15616
D, [2024-04-16T16:09:23.485281 #15582] DEBUG -- : Swarm manager checking nodes: 15616
D, [2024-04-16T16:09:24.485500 #15582] DEBUG -- : Swarm manager checking nodes: 15616
E, [2024-04-16T16:09:25.485793 #15582] ERROR -- : Swarm manager detected unresponsive node 15616. Sending KILL signal...
D, [2024-04-16T16:09:25.485874 #15582] DEBUG -- : Swarm manager checking nodes: 15616
I, [2024-04-16T16:09:25.488320 #15582]  INFO -- : Received SIGCHLD system signal
D, [2024-04-16T16:09:25.488467 #15582] DEBUG -- : Swarm manager checking nodes: 15616
D, [2024-04-16T16:09:26.488658 #15582] DEBUG -- : Swarm manager starting node with id: 0
D, [2024-04-16T16:09:26.491142 #15582] DEBUG -- : Swarm manager checking nodes: 15817
I, [2024-04-16T16:09:26.493572 #15817]  INFO -- : Swarm node 15817 forked from 15582
I, [2024-04-16T16:09:26.494962 #15817]  INFO -- : Running in ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux]
I, [2024-04-16T16:09:26.494994 #15817]  INFO -- : Running Karafka 2.4.0.beta2 server
I, [2024-04-16T16:09:26.495010 #15817]  INFO -- : See LICENSE and the LGPL-3.0 for licensing details
D, [2024-04-16T16:09:27.491400 #15582] DEBUG -- : Swarm manager checking nodes: 15817
D, [2024-04-16T16:09:28.491635 #15582] DEBUG -- : Swarm manager checking nodes: 15817
E, [2024-04-16T16:09:29.492000 #15582] ERROR -- : Swarm manager detected unhealthy node 15817. Sending TERM signal...
D, [2024-04-16T16:09:29.492077 #15582] DEBUG -- : Swarm manager checking nodes: 15817
I, [2024-04-16T16:09:29.492298 #15817]  INFO -- : Received SIGTERM system signal
I, [2024-04-16T16:09:29.492522 #15817]  INFO -- : Stopping Karafka server
I, [2024-04-16T16:09:29.492362 #15817]  INFO -- : Received SIGTERM system signal
D, [2024-04-16T16:09:30.492262 #15582] DEBUG -- : Swarm manager checking nodes: 15817
D, [2024-04-16T16:09:31.492466 #15582] DEBUG -- : Swarm manager checking nodes: 15817
D, [2024-04-16T16:09:32.492684 #15582] DEBUG -- : Swarm manager checking nodes: 15817
D, [2024-04-16T16:09:33.492900 #15582] DEBUG -- : Swarm manager checking nodes: 15817
E, [2024-04-16T16:09:34.493219 #15582] ERROR -- : Swarm manager detected unresponsive node 15817. Sending KILL signal...
D, [2024-04-16T16:09:34.493304 #15582] DEBUG -- : Swarm manager checking nodes: 15817
I, [2024-04-16T16:09:34.495698 #15582]  INFO -- : Received SIGCHLD system signal
D, [2024-04-16T16:09:34.495830 #15582] DEBUG -- : Swarm manager checking nodes: 15817
D, [2024-04-16T16:09:35.495997 #15582] DEBUG -- : Swarm manager starting node with id: 0
D, [2024-04-16T16:09:35.497863 #15582] DEBUG -- : Swarm manager checking nodes: 16009
I, [2024-04-16T16:09:35.508062 #15582]  INFO -- : Received SIGTERM system signal
I, [2024-04-16T16:09:35.508304 #15582]  INFO -- : Stopping Karafka server
E, [2024-04-16T16:11:16.612225 #15582] ERROR -- : Forceful Karafka server stop
I, [2024-04-16T16:11:16.614070 #15582]  INFO -- : Received SIGCHLD system signal
I, [2024-04-16T16:11:16.712620 #15582]  INFO -- : Stopped Karafka server


..............................................................................................

Longest scenarios:

[120.22] swarm/forceful_nodes_restart_graceful_supervisor_stop_spec.rb
[ 71.22] rebalancing/cooperative_sticky_full_spec.rb
[ 58.91] rebalancing/cooperative_sticky_spec.rb
[ 47.41] rebalancing/constant_rebalance_continuity_spec.rb
[ 42.59] rebalancing/lifecycle_events_spec.rb
[ 40.67] consumption/standard_saturation_reliability_spec.rb
[ 40.49] rebalancing/exceeding_max_poll_with_late_commit_spec.rb
[ 40.47] rebalancing/exceeding_max_poll_with_fast_commit_spec.rb
[ 40.47] rebalancing/exceeding_max_poll_interval_spec.rb
[ 40.37] rebalancing/with_static_membership_reconnect_spec.rb


Failed scenarios:

[FAILED] swarm/forceful_nodes_restart_graceful_supervisor_stop_spec.rb

@mensfeld
Copy link
Member Author 


[FAILED] admin/configs_flow_spec.rb
Time taken: 1.7049441710000224 seconds
Exit code: 1
Exit status: pid 4757 exit 1

...

/home/runner/work/karafka/karafka/spec/integrations_helper.rb:384:in `assert_equal': {} (AssertionFailedError)
	from /home/runner/work/karafka/karafka/spec/integrations/admin/configs_flow_spec.rb:62:in `block in <main>'
	from /home/runner/work/karafka/karafka/spec/integrations/admin/configs_flow_spec.rb:54:in `each'
	from /home/runner/work/karafka/karafka/spec/integrations/admin/configs_flow_spec.rb:54:in `<main>'

@mensfeld
Copy link
Member Author 

Random seed: 9253
.
[FAILED] admin/configs_flow_spec.rb
Time taken: 1.8677176129999964 seconds
Exit code: 1
Exit status: pid 4682 exit 1

...

/home/runner/work/karafka/karafka/spec/integrations_helper.rb:384:in `assert_equal': {} (AssertionFailedError)
	from /home/runner/work/karafka/karafka/spec/integrations/admin/configs_flow_spec.rb:62:in `block in <main>'
	from /home/runner/work/karafka/karafka/spec/integrations/admin/configs_flow_spec.rb:54:in `each'
	from /home/runner/work/karafka/karafka/spec/integrations/admin/configs_flow_spec.rb:54:in `<main>'

@mensfeld
Copy link
Member Author

mensfeld commented May 6, 2024 

 I, [2024-05-06T15:10:48.697481 #21462]  INFO -- : Running in ruby 3.0.7p220 (2024-04-23 revision 724a071175) [x86_64-linux]
I, [2024-05-06T15:10:48.697524 #21462]  INFO -- : Running Karafka 2.4.1 server
D, [2024-05-06T15:10:48.799376 #21462] DEBUG -- : [68b589c64d37] Polling messages...
D, [2024-05-06T15:10:49.301815 #21462] DEBUG -- : [68b589c64d37] Polled 0 messages in 502.36189900012687ms
D, [2024-05-06T15:10:49.301924 #21462] DEBUG -- : [68b589c64d37] Polling messages...
D, [2024-05-06T15:10:49.802107 #21462] DEBUG -- : [68b589c64d37] Polled 0 messages in 500.1257820003666ms
D, [2024-05-06T15:10:49.802183 #21462] DEBUG -- : [68b589c64d37] Polling messages...
D, [2024-05-06T15:10:50.304360 #21462] DEBUG -- : [68b589c64d37] Polled 0 messages in 502.1247239999939ms
D, [2024-05-06T15:10:50.304446 #21462] DEBUG -- : [68b589c64d37] Polling messages...
D, [2024-05-06T15:10:50.806687 #21462] DEBUG -- : [68b589c64d37] Polled 0 messages in 502.1822130000219ms
D, [2024-05-06T15:10:50.806799 #21462] DEBUG -- : [68b589c64d37] Polling messages...
D, [2024-05-06T15:10:51.310551 #21462] DEBUG -- : [68b589c64d37] Polled 0 messages in 503.6713979996275ms
D, [2024-05-06T15:10:51.310682 #21462] DEBUG -- : [68b589c64d37] Polling messages...
I, [2024-05-06T15:10:51.806731 #21462]  INFO -- : [68b589c64d37] Polled 1 messages in 495.9900849999394ms
I, [2024-05-06T15:10:51.807247 #21462]  INFO -- : [f0ea31c262eb] Idle job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:51.807495 #21462]  INFO -- : [3ba1fc1f0f58] Post-filtering seeking to message 0 on 96009503c467b5fa/0
I, [2024-05-06T15:10:51.807548 #21462]  INFO -- : [f0ea31c262eb] Idle job for Consumer on 96009503c467b5fa/0 finished in 0.21476099966093898ms
D, [2024-05-06T15:10:51.807625 #21462] DEBUG -- : [68b589c64d37] Polling messages...
I, [2024-05-06T15:10:52.306534 #21462]  INFO -- : [68b589c64d37] Polled 100 messages in 498.85613900003955ms
I, [2024-05-06T15:10:52.307916 #21462]  INFO -- : [2a376259fedf] Consume job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:52.308139 #21462]  INFO -- : [952546f3a922] Consume job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:52.308277 #21462]  INFO -- : [24b454b5f03f] Consume job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:52.308371 #21462]  INFO -- : [f0c935c48a92] Consume job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:52.308479 #21462]  INFO -- : [c691f48f3ea8] Consume job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:52.308554 #21462]  INFO -- : [d761200a6c31] Consume job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:52.308654 #21462]  INFO -- : [f0ea31c262eb] Consume job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:52.308741 #21462]  INFO -- : [9ef1a12588f9] Consume job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:52.308814 #21462]  INFO -- : [f35ab65c9385] Consume job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:52.308902 #21462]  INFO -- : [1edaf841edc0] Consume job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:52.308990 #21462]  INFO -- : [42bd1df19da4] Starting transaction took 0.8847109999042004 ms
I, [2024-05-06T15:10:52.309297 #21462]  INFO -- : [42bd1df19da4] Committing transaction took 0.1228899999987334 ms
I, [2024-05-06T15:10:52.309330 #21462]  INFO -- : [42bd1df19da4] Processing transaction took 1.2421679999679327 ms
I, [2024-05-06T15:10:52.309561 #21462]  INFO -- : [42bd1df19da4] Starting transaction took 0.0879340001847595 ms
I, [2024-05-06T15:10:52.310119 #21462]  INFO -- : [42bd1df19da4] Committing transaction took 0.22056100005283952 ms
I, [2024-05-06T15:10:52.310146 #21462]  INFO -- : [42bd1df19da4] Processing transaction took 0.6802399998996407 ms
I, [2024-05-06T15:10:52.310454 #21462]  INFO -- : [2a376259fedf] Consume job for Consumer on 96009503c467b5fa/0 finished in 2.4547520002815872ms
I, [2024-05-06T15:10:52.310958 #21462]  INFO -- : [952546f3a922] Consume job for Consumer on 96009503c467b5fa/0 finished in 2.7552430003415793ms
I, [2024-05-06T15:10:52.311001 #21462]  INFO -- : [42bd1df19da4] Starting transaction took 0.6440859998110682 ms
I, [2024-05-06T15:10:52.311412 #21462]  INFO -- : [42bd1df19da4] Committing transaction took 0.11015500011853874 ms
I, [2024-05-06T15:10:52.311446 #21462]  INFO -- : [42bd1df19da4] Processing transaction took 1.0927020001690835 ms
I, [2024-05-06T15:10:52.311675 #21462]  INFO -- : [42bd1df19da4] Starting transaction took 0.10036499984562397 ms
I, [2024-05-06T15:10:52.312380 #21462]  INFO -- : [42bd1df19da4] Committing transaction took 0.1382980002090335 ms
I, [2024-05-06T15:10:52.312535 #21462]  INFO -- : [24b454b5f03f] Consume job for Consumer on 96009503c467b5fa/0 finished in 4.20449799997732ms
I, [2024-05-06T15:10:52.312586 #21462]  INFO -- : [42bd1df19da4] Processing transaction took 0.8422390001360327 ms
I, [2024-05-06T15:10:52.312924 #21462]  INFO -- : [42bd1df19da4] Starting transaction took 0.11245999997481704 ms
I, [2024-05-06T15:10:52.313563 #21462]  INFO -- : [f0c935c48a92] Consume job for Consumer on 96009503c467b5fa/0 finished in 5.132726999931037ms
I, [2024-05-06T15:10:52.313889 #21462]  INFO -- : [42bd1df19da4] Committing transaction took 0.11076600011438131 ms
I, [2024-05-06T15:10:52.313912 #21462]  INFO -- : [42bd1df19da4] Processing transaction took 1.1088590000290424 ms
I, [2024-05-06T15:10:52.314137 #21462]  INFO -- : [42bd1df19da4] Starting transaction took 0.11567299999296665 ms
I, [2024-05-06T15:10:52.314799 #21462]  INFO -- : [c691f48f3ea8] Consume job for Consumer on 96009503c467b5fa/0 finished in 6.271780000301078ms
I, [2024-05-06T15:10:52.314892 #21462]  INFO -- : [42bd1df19da4] Committing transaction took 0.13628700003027916 ms
I, [2024-05-06T15:10:52.314913 #21462]  INFO -- : [42bd1df19da4] Processing transaction took 0.9041570001281798 ms
I, [2024-05-06T15:10:52.315133 #21462]  INFO -- : [42bd1df19da4] Starting transaction took 0.12209800002165139 ms
I, [2024-05-06T15:10:52.315685 #21462]  INFO -- : [d761200a6c31] Consume job for Consumer on 96009503c467b5fa/0 finished in 7.078583000227809ms
I, [2024-05-06T15:10:52.315844 #21462]  INFO -- : [42bd1df19da4] Committing transaction took 0.10991899995133281 ms
I, [2024-05-06T15:10:52.315867 #21462]  INFO -- : [42bd1df19da4] Processing transaction took 0.8660099999979138 ms
I, [2024-05-06T15:10:52.317280 #21462]  INFO -- : [42bd1df19da4] Starting transaction took 0.115074000088498 ms
I, [2024-05-06T15:10:52.317562 #21462]  INFO -- : [f0ea31c262eb] Consume job for Consumer on 96009503c467b5fa/0 finished in 8.861404000082985ms
I, [2024-05-06T15:10:52.319382 #21462]  INFO -- : Stopping Karafka server
I, [2024-05-06T15:10:57.317712 #21462]  INFO -- : [42bd1df19da4] Committing transaction took 0.1372859999537468 ms
I, [2024-05-06T15:10:57.317892 #21462]  INFO -- : [42bd1df19da4] Processing transaction took 5000.7371529999655 ms
I, [2024-05-06T15:10:57.346825 #21462]  INFO -- : [42bd1df19da4] Starting transaction took 28.110149999847636 ms
I, [2024-05-06T15:10:57.347300 #21462]  INFO -- : [42bd1df19da4] Committing transaction took 0.14700799994170666 ms
I, [2024-05-06T15:10:57.347327 #21462]  INFO -- : [42bd1df19da4] Processing transaction took 28.629893000004813 ms
I, [2024-05-06T15:10:57.347443 #21462]  INFO -- : [42bd1df19da4] Starting transaction took 0.058588000014424324 ms
I, [2024-05-06T15:10:57.347650 #21462]  INFO -- : [42bd1df19da4] Committing transaction took 0.1243920000270009 ms
I, [2024-05-06T15:10:57.347684 #21462]  INFO -- : [42bd1df19da4] Processing transaction took 0.30618000007234514 ms
I, [2024-05-06T15:10:57.348116 #21462]  INFO -- : [f35ab65c9385] Consume job for Consumer on 96009503c467b5fa/0 finished in 5039.245160000166ms
I, [2024-05-06T15:10:57.349014 #21462]  INFO -- : [9ef1a12588f9] Consume job for Consumer on 96009503c467b5fa/0 finished in 5039.964887000155ms
I, [2024-05-06T15:10:57.353398 #21462]  INFO -- : [1edaf841edc0] Consume job for Consumer on 96009503c467b5fa/0 finished in 5044.454356000293ms
I, [2024-05-06T15:10:57.353647 #21462]  INFO -- : [f0ea31c262eb] Shutdown job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:57.353693 #21462]  INFO -- : [f35ab65c9385] Shutdown job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:57.353740 #21462]  INFO -- : [9ef1a12588f9] Shutdown job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:57.353791 #21462]  INFO -- : [d761200a6c31] Shutdown job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:57.353849 #21462]  INFO -- : [c691f48f3ea8] Shutdown job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:57.353891 #21462]  INFO -- : [f0c935c48a92] Shutdown job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:57.353930 #21462]  INFO -- : [24b454b5f03f] Shutdown job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:57.353968 #21462]  INFO -- : [952546f3a922] Shutdown job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:57.354002 #21462]  INFO -- : [2a376259fedf] Shutdown job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:57.354043 #21462]  INFO -- : [1edaf841edc0] Shutdown job for Consumer on 96009503c467b5fa/0 started
I, [2024-05-06T15:10:58.355447 #21462]  INFO -- : [f35ab65c9385] Shutdown job for Consumer on 96009503c467b5fa/0 finished in 1001.276533999946ms
I, [2024-05-06T15:10:58.355869 #21462]  INFO -- : [f0ea31c262eb] Shutdown job for Consumer on 96009503c467b5fa/0 finished in 1001.7610849998891ms
I, [2024-05-06T15:10:58.356294 #21462]  INFO -- : [d761200a6c31] Shutdown job for Consumer on 96009503c467b5fa/0 finished in 1002.071821999969ms
I, [2024-05-06T15:10:58.356708 #21462]  INFO -- : [c691f48f3ea8] Shutdown job for Consumer on 96009503c467b5fa/0 finished in 1002.4621400001924ms
I, [2024-05-06T15:10:58.357101 #21462]  INFO -- : [f0c935c48a92] Shutdown job for Consumer on 96009503c467b5fa/0 finished in 1002.8256740001962ms
I, [2024-05-06T15:10:58.357492 #21462]  INFO -- : [24b454b5f03f] Shutdown job for Consumer on 96009503c467b5fa/0 finished in 1003.194656000007ms
I, [2024-05-06T15:10:58.357878 #21462]  INFO -- : [952546f3a922] Shutdown job for Consumer on 96009503c467b5fa/0 finished in 1003.5551409998443ms
I, [2024-05-06T15:10:58.358246 #21462]  INFO -- : [2a376259fedf] Shutdown job for Consumer on 96009503c467b5fa/0 finished in 1003.90468100016ms
I, [2024-05-06T15:10:58.358630 #21462]  INFO -- : [9ef1a12588f9] Shutdown job for Consumer on 96009503c467b5fa/0 finished in 1004.4541889999527ms
I, [2024-05-06T15:10:58.359007 #21462]  INFO -- : [1edaf841edc0] Shutdown job for Consumer on 96009503c467b5fa/0 finished in 1004.6327109998092ms
I, [2024-05-06T15:10:59.048883 #21462]  INFO -- : Stopped Karafka server
I, [2024-05-06T15:10:59.048983 #21462]  INFO -- : [42bd1df19da4] Closing producer
I, [2024-05-06T15:10:59.072864 #21462]  INFO -- : [42bd1df19da4] Closing producer took 23.879759000148624 ms
/home/runner/work/karafka/karafka/spec/integrations_helper.rb:384:in `assert_equal': {:iterator=>#<Enumerator: 0..9:cycle>, :done=>[3520, 3540, 3560, 3580, 3600, 3620, 3640, 3660, 3680, 3700], :last=>["8", "18", "28", "38", "48", "58", "68", "78", "88", "98", "7", "17", "27", "37", "47", "57", "67", "77", "87", "97", "6", "16", "26", "36", "46", "56", "66", "76", "86", "96", "5", "15", "25", "35", "45", "55", "65", "75", "85", "95", "4", "14", "24", "34", "44", "54", "64", "74", "84", "94", "3", "13", "23", "33", "43", "53", "63", "73", "83", "93", "2", "12", "22", "32", "42", "52", "62", "72", "82", "92", "0", "10", "20", "30", "40", "50", "60", "70", "80", "90", "9", "19", "29", "39", "49", "59", "69", "79", "89", "99", "1", "11", "21", "31", "41", "51", "61", "71", "81", "91"], :metadata=>["99", "99", "99", "99", "99", "99", "99", "99", "99", "99"]} (AssertionFailedError)
	from /home/runner/work/karafka/karafka/spec/integrations/pro/consumption/transactions/vps/all_ok_current_spec.rb:56:in `<main>'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mensfeld mensfeld

Labels
internal Things that are needed but do not change any user-facing APIs specs Extra unit or integration specs
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mensfeld