Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Topic Subscription exponential backoff not working properly (too many retries) #2235

Open
bpatrzyk opened this issue Apr 1, 2024 · 3 comments
Open

Topic Subscription exponential backoff not working properly (too many retries) #2235

bpatrzyk opened this issue Apr 1, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@bpatrzyk
Copy link

bpatrzyk commented Apr 1, 2024

Description

What I am trying to do is set up a backend that subscribes to a Topic and in case of an error retries connecting indefinitely.
For testing I set the max attempts to a large number and I pointed the client to a non-existent local node (so that the connection fails each time)

const topicQuery = new TopicMessageQuery()
    .setTopicId(topicId).setMaxAttempts(Number.MAX_VALUE).setMaxBackoff(20000);

What I noticed is that the number of retries is rising unexpectedly fast. For debugging I added timestamp in the library TopicMessageQuery.js file in the MirrorChannel.makeServerStreamRequest function error handler:

console.warn(
                            `${(new Date()).toISOString()} - Error subscribing to topic ${
                                this._topicId != null
                                    ? this._topicId.toString()
                                    : "UNKNOWN"
                            } during attempt ${
                                this._attempt
                            }. Waiting ${delay} ms before next attempt: ${message}`,
                        );

As well as a console log in the timeout handler in the same function:

                        setTimeout(() => {
                            console.log(`${(new Date()).toISOString()} - Fire timeout for attempt ${attempt} (delay ${delay})`)
                            this._makeServerStreamRequest(client);
                        }, delay);

Please see the attached log.

After roughly 3 minutes there are about a 1000 retries with the max backoff time of 20s. I would expect it to be maximum of about 15-20 retries for these parameters.

What I noticed is that the error handler is executed twice for each error. Once with an Error class error and the second time with an object of the following form:

{
  code: 14,
  details: 'No connection established',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} },
  progress: 'PROCESSED'
}

I suppose this duplicated error handler execution leads to the unexpected exponential rise in the number of retries.

Could you please check this issue? It would also be great if you could add support for setting an indefinite number of max retries (perhaps a null or -1 value?)

Steps to reproduce

  1. set up an empty node project (npm init)
  2. install hashgraph sdk version v2.40.0
  3. create the following file index.mjs
import {AccountId, Client, AccountBalanceQuery, TopicMessageQuery} from '@hashgraph/sdk';
import 'dotenv/config'

const localnodeaddress = "localhost";


const node = {};
node[`${localnodeaddress}:50211`] = new AccountId(3)
const client = Client.forNetwork(node).setMirrorNetwork(`${localnodeaddress}:5600`).setMaxExecutionTime(null);

const topicId = "0.0.1002"

// Subscribe to the topic
const topicQuery = new TopicMessageQuery()
    .setTopicId(topicId).setMaxAttempts(Number.MAX_VALUE).setMaxBackoff(20000);

topicQuery    .subscribe(client, null, (message) => {
        let messageAsString = Buffer.from(message.contents, "utf8").toString();
        console.log(
            `${message.consensusTimestamp.toDate()} Received: ${messageAsString}`
        );
    });
  1. make sure that THERE IS NO localnode running at localhost:5600
  2. run the code node index.mjs

Additional context

2024-04-01T09:32:53.966Z - Error subscribing to topic 0.0.3378616 during attempt 0. Waiting 250 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:32:53.967Z - Error subscribing to topic 0.0.3378616 during attempt 1. Waiting 500 ms before next attempt: No connection established
2024-04-01T09:32:54.217Z - Fire timeout for attempt 1 (delay 250)
2024-04-01T09:32:54.221Z - Error subscribing to topic 0.0.3378616 during attempt 2. Waiting 1000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:32:54.221Z - Error subscribing to topic 0.0.3378616 during attempt 3. Waiting 2000 ms before next attempt: No connection established
2024-04-01T09:32:54.467Z - Fire timeout for attempt 2 (delay 500)
2024-04-01T09:32:54.469Z - Error subscribing to topic 0.0.3378616 during attempt 4. Waiting 4000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:32:54.469Z - Error subscribing to topic 0.0.3378616 during attempt 5. Waiting 8000 ms before next attempt: No connection established
2024-04-01T09:32:55.221Z - Fire timeout for attempt 3 (delay 1000)
2024-04-01T09:32:55.222Z - Error subscribing to topic 0.0.3378616 during attempt 6. Waiting 16000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:32:55.222Z - Error subscribing to topic 0.0.3378616 during attempt 7. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:32:56.220Z - Fire timeout for attempt 4 (delay 2000)
2024-04-01T09:32:56.223Z - Error subscribing to topic 0.0.3378616 during attempt 8. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:32:56.223Z - Error subscribing to topic 0.0.3378616 during attempt 9. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:32:58.470Z - Fire timeout for attempt 5 (delay 4000)
2024-04-01T09:32:58.474Z - Error subscribing to topic 0.0.3378616 during attempt 10. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:32:58.474Z - Error subscribing to topic 0.0.3378616 during attempt 11. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:02.469Z - Fire timeout for attempt 6 (delay 8000)
2024-04-01T09:33:02.471Z - Error subscribing to topic 0.0.3378616 during attempt 12. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:02.471Z - Error subscribing to topic 0.0.3378616 during attempt 13. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:11.222Z - Fire timeout for attempt 7 (delay 16000)
2024-04-01T09:33:11.226Z - Error subscribing to topic 0.0.3378616 during attempt 14. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:11.226Z - Error subscribing to topic 0.0.3378616 during attempt 15. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:15.223Z - Fire timeout for attempt 8 (delay 20000)
2024-04-01T09:33:15.224Z - Error subscribing to topic 0.0.3378616 during attempt 16. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:15.224Z - Error subscribing to topic 0.0.3378616 during attempt 17. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:16.223Z - Fire timeout for attempt 9 (delay 20000)
2024-04-01T09:33:16.223Z - Error subscribing to topic 0.0.3378616 during attempt 18. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:16.224Z - Error subscribing to topic 0.0.3378616 during attempt 19. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:16.224Z - Fire timeout for attempt 10 (delay 20000)
2024-04-01T09:33:16.224Z - Error subscribing to topic 0.0.3378616 during attempt 20. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:16.224Z - Error subscribing to topic 0.0.3378616 during attempt 21. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:18.474Z - Fire timeout for attempt 11 (delay 20000)
2024-04-01T09:33:18.476Z - Error subscribing to topic 0.0.3378616 during attempt 22. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:18.476Z - Error subscribing to topic 0.0.3378616 during attempt 23. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:18.476Z - Fire timeout for attempt 12 (delay 20000)
2024-04-01T09:33:18.477Z - Error subscribing to topic 0.0.3378616 during attempt 24. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:18.477Z - Error subscribing to topic 0.0.3378616 during attempt 25. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:22.470Z - Fire timeout for attempt 13 (delay 20000)
2024-04-01T09:33:22.471Z - Error subscribing to topic 0.0.3378616 during attempt 26. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:22.472Z - Error subscribing to topic 0.0.3378616 during attempt 27. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:22.472Z - Fire timeout for attempt 14 (delay 20000)
2024-04-01T09:33:22.472Z - Error subscribing to topic 0.0.3378616 during attempt 28. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:22.472Z - Error subscribing to topic 0.0.3378616 during attempt 29. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:31.225Z - Fire timeout for attempt 15 (delay 20000)
2024-04-01T09:33:31.226Z - Fire timeout for attempt 16 (delay 20000)
2024-04-01T09:33:34.551Z - Error subscribing to topic 0.0.3378616 during attempt 30. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:34.551Z - Error subscribing to topic 0.0.3378616 during attempt 31. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:34.551Z - Error subscribing to topic 0.0.3378616 during attempt 32. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:34.551Z - Error subscribing to topic 0.0.3378616 during attempt 33. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:35.224Z - Fire timeout for attempt 17 (delay 20000)
2024-04-01T09:33:35.226Z - Fire timeout for attempt 18 (delay 20000)
2024-04-01T09:33:35.227Z - Error subscribing to topic 0.0.3378616 during attempt 34. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:35.227Z - Error subscribing to topic 0.0.3378616 during attempt 35. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:35.227Z - Error subscribing to topic 0.0.3378616 during attempt 36. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:35.227Z - Error subscribing to topic 0.0.3378616 during attempt 37. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:36.223Z - Fire timeout for attempt 19 (delay 20000)
2024-04-01T09:33:36.225Z - Error subscribing to topic 0.0.3378616 during attempt 38. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:36.225Z - Error subscribing to topic 0.0.3378616 during attempt 39. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:36.225Z - Fire timeout for attempt 20 (delay 20000)
2024-04-01T09:33:36.226Z - Error subscribing to topic 0.0.3378616 during attempt 40. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:36.226Z - Error subscribing to topic 0.0.3378616 during attempt 41. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:36.227Z - Fire timeout for attempt 21 (delay 20000)
2024-04-01T09:33:36.227Z - Error subscribing to topic 0.0.3378616 during attempt 42. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:36.227Z - Error subscribing to topic 0.0.3378616 during attempt 43. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:36.227Z - Fire timeout for attempt 22 (delay 20000)
2024-04-01T09:33:36.228Z - Error subscribing to topic 0.0.3378616 during attempt 44. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:36.228Z - Error subscribing to topic 0.0.3378616 during attempt 45. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:38.475Z - Fire timeout for attempt 23 (delay 20000)
2024-04-01T09:33:38.476Z - Error subscribing to topic 0.0.3378616 during attempt 46. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:38.476Z - Error subscribing to topic 0.0.3378616 during attempt 47. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:38.476Z - Fire timeout for attempt 24 (delay 20000)
2024-04-01T09:33:38.477Z - Error subscribing to topic 0.0.3378616 during attempt 48. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:38.477Z - Error subscribing to topic 0.0.3378616 during attempt 49. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:38.477Z - Fire timeout for attempt 25 (delay 20000)
2024-04-01T09:33:38.477Z - Error subscribing to topic 0.0.3378616 during attempt 50. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:38.477Z - Error subscribing to topic 0.0.3378616 during attempt 51. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:38.479Z - Fire timeout for attempt 26 (delay 20000)
2024-04-01T09:33:38.480Z - Error subscribing to topic 0.0.3378616 during attempt 52. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:38.480Z - Error subscribing to topic 0.0.3378616 during attempt 53. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:42.472Z - Fire timeout for attempt 27 (delay 20000)
2024-04-01T09:33:42.473Z - Error subscribing to topic 0.0.3378616 during attempt 54. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:42.473Z - Error subscribing to topic 0.0.3378616 during attempt 55. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:42.473Z - Fire timeout for attempt 28 (delay 20000)
2024-04-01T09:33:42.474Z - Error subscribing to topic 0.0.3378616 during attempt 56. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:42.474Z - Error subscribing to topic 0.0.3378616 during attempt 57. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:42.474Z - Fire timeout for attempt 29 (delay 20000)
2024-04-01T09:33:42.474Z - Error subscribing to topic 0.0.3378616 during attempt 58. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:42.474Z - Error subscribing to topic 0.0.3378616 during attempt 59. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:42.475Z - Fire timeout for attempt 30 (delay 20000)
2024-04-01T09:33:42.475Z - Error subscribing to topic 0.0.3378616 during attempt 60. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:42.475Z - Error subscribing to topic 0.0.3378616 during attempt 61. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:54.551Z - Fire timeout for attempt 31 (delay 20000)
2024-04-01T09:33:54.552Z - Error subscribing to topic 0.0.3378616 during attempt 62. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:54.552Z - Error subscribing to topic 0.0.3378616 during attempt 63. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:54.552Z - Fire timeout for attempt 32 (delay 20000)
2024-04-01T09:33:54.553Z - Error subscribing to topic 0.0.3378616 during attempt 64. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:54.553Z - Error subscribing to topic 0.0.3378616 during attempt 65. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:54.553Z - Fire timeout for attempt 33 (delay 20000)
2024-04-01T09:33:54.554Z - Error subscribing to topic 0.0.3378616 during attempt 66. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:54.554Z - Error subscribing to topic 0.0.3378616 during attempt 67. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:54.554Z - Fire timeout for attempt 34 (delay 20000)
2024-04-01T09:33:54.555Z - Error subscribing to topic 0.0.3378616 during attempt 68. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:54.555Z - Error subscribing to topic 0.0.3378616 during attempt 69. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:55.227Z - Fire timeout for attempt 35 (delay 20000)
2024-04-01T09:33:55.228Z - Error subscribing to topic 0.0.3378616 during attempt 70. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:55.228Z - Error subscribing to topic 0.0.3378616 during attempt 71. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:55.228Z - Fire timeout for attempt 36 (delay 20000)
2024-04-01T09:33:55.228Z - Error subscribing to topic 0.0.3378616 during attempt 72. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:55.228Z - Error subscribing to topic 0.0.3378616 during attempt 73. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:55.229Z - Fire timeout for attempt 37 (delay 20000)
2024-04-01T09:33:55.229Z - Error subscribing to topic 0.0.3378616 during attempt 74. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:55.229Z - Error subscribing to topic 0.0.3378616 during attempt 75. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:55.229Z - Fire timeout for attempt 38 (delay 20000)
2024-04-01T09:33:55.229Z - Error subscribing to topic 0.0.3378616 during attempt 76. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:55.229Z - Error subscribing to topic 0.0.3378616 during attempt 77. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:56.225Z - Fire timeout for attempt 39 (delay 20000)
2024-04-01T09:33:56.226Z - Error subscribing to topic 0.0.3378616 during attempt 78. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:56.226Z - Error subscribing to topic 0.0.3378616 during attempt 79. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:56.226Z - Fire timeout for attempt 40 (delay 20000)
2024-04-01T09:33:56.228Z - Error subscribing to topic 0.0.3378616 during attempt 80. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:56.228Z - Error subscribing to topic 0.0.3378616 during attempt 81. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:56.228Z - Fire timeout for attempt 41 (delay 20000)
2024-04-01T09:33:56.229Z - Error subscribing to topic 0.0.3378616 during attempt 82. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:56.229Z - Error subscribing to topic 0.0.3378616 during attempt 83. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:56.229Z - Fire timeout for attempt 42 (delay 20000)
2024-04-01T09:33:56.229Z - Error subscribing to topic 0.0.3378616 during attempt 84. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:56.229Z - Error subscribing to topic 0.0.3378616 during attempt 85. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:56.231Z - Fire timeout for attempt 43 (delay 20000)
2024-04-01T09:33:56.231Z - Error subscribing to topic 0.0.3378616 during attempt 86. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:56.232Z - Error subscribing to topic 0.0.3378616 during attempt 87. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:56.232Z - Fire timeout for attempt 44 (delay 20000)
2024-04-01T09:33:56.232Z - Error subscribing to topic 0.0.3378616 during attempt 88. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:56.232Z - Error subscribing to topic 0.0.3378616 during attempt 89. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:56.232Z - Fire timeout for attempt 45 (delay 20000)
2024-04-01T09:33:56.232Z - Error subscribing to topic 0.0.3378616 during attempt 90. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:56.232Z - Error subscribing to topic 0.0.3378616 during attempt 91. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:56.233Z - Fire timeout for attempt 46 (delay 20000)
2024-04-01T09:33:56.233Z - Error subscribing to topic 0.0.3378616 during attempt 92. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:56.233Z - Error subscribing to topic 0.0.3378616 during attempt 93. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:58.476Z - Fire timeout for attempt 47 (delay 20000)
2024-04-01T09:33:58.477Z - Error subscribing to topic 0.0.3378616 during attempt 94. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:58.477Z - Error subscribing to topic 0.0.3378616 during attempt 95. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:58.477Z - Fire timeout for attempt 48 (delay 20000)
2024-04-01T09:33:58.478Z - Error subscribing to topic 0.0.3378616 during attempt 96. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:58.478Z - Error subscribing to topic 0.0.3378616 during attempt 97. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:58.478Z - Fire timeout for attempt 49 (delay 20000)
2024-04-01T09:33:58.479Z - Error subscribing to topic 0.0.3378616 during attempt 98. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:58.479Z - Error subscribing to topic 0.0.3378616 during attempt 99. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:58.479Z - Fire timeout for attempt 50 (delay 20000)
2024-04-01T09:33:58.479Z - Error subscribing to topic 0.0.3378616 during attempt 100. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:58.479Z - Error subscribing to topic 0.0.3378616 during attempt 101. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:58.480Z - Fire timeout for attempt 51 (delay 20000)
2024-04-01T09:33:58.480Z - Error subscribing to topic 0.0.3378616 during attempt 102. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:58.481Z - Error subscribing to topic 0.0.3378616 during attempt 103. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:58.481Z - Fire timeout for attempt 52 (delay 20000)
2024-04-01T09:33:58.481Z - Error subscribing to topic 0.0.3378616 during attempt 104. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:58.481Z - Error subscribing to topic 0.0.3378616 during attempt 105. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:58.483Z - Fire timeout for attempt 53 (delay 20000)
2024-04-01T09:33:58.483Z - Error subscribing to topic 0.0.3378616 during attempt 106. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:58.483Z - Error subscribing to topic 0.0.3378616 during attempt 107. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:33:58.483Z - Fire timeout for attempt 54 (delay 20000)
2024-04-01T09:33:58.484Z - Error subscribing to topic 0.0.3378616 during attempt 108. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:33:58.484Z - Error subscribing to topic 0.0.3378616 during attempt 109. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:02.472Z - Fire timeout for attempt 55 (delay 20000)
2024-04-01T09:34:02.472Z - Error subscribing to topic 0.0.3378616 during attempt 110. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:02.473Z - Error subscribing to topic 0.0.3378616 during attempt 111. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:02.473Z - Fire timeout for attempt 56 (delay 20000)
2024-04-01T09:34:02.473Z - Error subscribing to topic 0.0.3378616 during attempt 112. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:02.473Z - Error subscribing to topic 0.0.3378616 during attempt 113. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:02.474Z - Fire timeout for attempt 57 (delay 20000)
2024-04-01T09:34:02.475Z - Error subscribing to topic 0.0.3378616 during attempt 114. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:02.475Z - Error subscribing to topic 0.0.3378616 during attempt 115. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:02.475Z - Fire timeout for attempt 58 (delay 20000)
2024-04-01T09:34:02.476Z - Error subscribing to topic 0.0.3378616 during attempt 116. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:02.476Z - Error subscribing to topic 0.0.3378616 during attempt 117. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:02.476Z - Fire timeout for attempt 59 (delay 20000)
2024-04-01T09:34:02.476Z - Error subscribing to topic 0.0.3378616 during attempt 118. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:02.476Z - Error subscribing to topic 0.0.3378616 during attempt 119. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:02.476Z - Fire timeout for attempt 60 (delay 20000)
2024-04-01T09:34:02.477Z - Error subscribing to topic 0.0.3378616 during attempt 120. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:02.477Z - Error subscribing to topic 0.0.3378616 during attempt 121. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:02.477Z - Fire timeout for attempt 61 (delay 20000)
2024-04-01T09:34:02.478Z - Error subscribing to topic 0.0.3378616 during attempt 122. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:02.478Z - Error subscribing to topic 0.0.3378616 during attempt 123. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:02.478Z - Fire timeout for attempt 62 (delay 20000)
2024-04-01T09:34:02.478Z - Error subscribing to topic 0.0.3378616 during attempt 124. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:02.478Z - Error subscribing to topic 0.0.3378616 during attempt 125. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:14.552Z - Fire timeout for attempt 63 (delay 20000)
2024-04-01T09:34:14.553Z - Error subscribing to topic 0.0.3378616 during attempt 126. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:14.553Z - Error subscribing to topic 0.0.3378616 during attempt 127. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:14.554Z - Fire timeout for attempt 64 (delay 20000)
2024-04-01T09:34:14.554Z - Error subscribing to topic 0.0.3378616 during attempt 128. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:14.554Z - Error subscribing to topic 0.0.3378616 during attempt 129. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:14.554Z - Fire timeout for attempt 65 (delay 20000)
2024-04-01T09:34:14.555Z - Error subscribing to topic 0.0.3378616 during attempt 130. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:14.555Z - Error subscribing to topic 0.0.3378616 during attempt 131. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:14.555Z - Fire timeout for attempt 66 (delay 20000)
2024-04-01T09:34:14.556Z - Error subscribing to topic 0.0.3378616 during attempt 132. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:14.556Z - Error subscribing to topic 0.0.3378616 during attempt 133. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:14.557Z - Fire timeout for attempt 67 (delay 20000)
2024-04-01T09:34:14.558Z - Error subscribing to topic 0.0.3378616 during attempt 134. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:14.558Z - Error subscribing to topic 0.0.3378616 during attempt 135. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:14.558Z - Fire timeout for attempt 68 (delay 20000)
2024-04-01T09:34:14.559Z - Error subscribing to topic 0.0.3378616 during attempt 136. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:14.559Z - Error subscribing to topic 0.0.3378616 during attempt 137. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:14.559Z - Fire timeout for attempt 69 (delay 20000)
2024-04-01T09:34:14.559Z - Error subscribing to topic 0.0.3378616 during attempt 138. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:14.559Z - Error subscribing to topic 0.0.3378616 during attempt 139. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:14.559Z - Fire timeout for attempt 70 (delay 20000)
2024-04-01T09:34:14.560Z - Error subscribing to topic 0.0.3378616 during attempt 140. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:14.560Z - Error subscribing to topic 0.0.3378616 during attempt 141. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:15.228Z - Fire timeout for attempt 71 (delay 20000)
2024-04-01T09:34:15.229Z - Error subscribing to topic 0.0.3378616 during attempt 142. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:15.229Z - Error subscribing to topic 0.0.3378616 during attempt 143. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:15.229Z - Fire timeout for attempt 72 (delay 20000)
2024-04-01T09:34:15.229Z - Error subscribing to topic 0.0.3378616 during attempt 144. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:15.230Z - Error subscribing to topic 0.0.3378616 during attempt 145. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:15.230Z - Fire timeout for attempt 73 (delay 20000)
2024-04-01T09:34:15.230Z - Error subscribing to topic 0.0.3378616 during attempt 146. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:15.230Z - Error subscribing to topic 0.0.3378616 during attempt 147. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:15.230Z - Fire timeout for attempt 74 (delay 20000)
2024-04-01T09:34:15.231Z - Error subscribing to topic 0.0.3378616 during attempt 148. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:15.231Z - Error subscribing to topic 0.0.3378616 during attempt 149. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:15.231Z - Fire timeout for attempt 75 (delay 20000)
2024-04-01T09:34:15.232Z - Error subscribing to topic 0.0.3378616 during attempt 150. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:15.232Z - Error subscribing to topic 0.0.3378616 during attempt 151. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:15.232Z - Fire timeout for attempt 76 (delay 20000)
2024-04-01T09:34:15.232Z - Error subscribing to topic 0.0.3378616 during attempt 152. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:15.232Z - Error subscribing to topic 0.0.3378616 during attempt 153. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:15.232Z - Fire timeout for attempt 77 (delay 20000)
2024-04-01T09:34:15.232Z - Error subscribing to topic 0.0.3378616 during attempt 154. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:15.233Z - Error subscribing to topic 0.0.3378616 during attempt 155. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:15.233Z - Fire timeout for attempt 78 (delay 20000)
2024-04-01T09:34:15.233Z - Error subscribing to topic 0.0.3378616 during attempt 156. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:15.234Z - Error subscribing to topic 0.0.3378616 during attempt 157. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:16.226Z - Fire timeout for attempt 79 (delay 20000)
2024-04-01T09:34:16.227Z - Error subscribing to topic 0.0.3378616 during attempt 158. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:16.228Z - Error subscribing to topic 0.0.3378616 during attempt 159. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:16.228Z - Fire timeout for attempt 80 (delay 20000)
2024-04-01T09:34:16.228Z - Error subscribing to topic 0.0.3378616 during attempt 160. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:16.228Z - Error subscribing to topic 0.0.3378616 during attempt 161. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:16.228Z - Fire timeout for attempt 81 (delay 20000)
2024-04-01T09:34:16.228Z - Error subscribing to topic 0.0.3378616 during attempt 162. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:16.228Z - Error subscribing to topic 0.0.3378616 during attempt 163. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:16.228Z - Fire timeout for attempt 82 (delay 20000)
2024-04-01T09:34:16.229Z - Error subscribing to topic 0.0.3378616 during attempt 164. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:16.229Z - Error subscribing to topic 0.0.3378616 during attempt 165. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:16.229Z - Fire timeout for attempt 83 (delay 20000)
2024-04-01T09:34:16.229Z - Error subscribing to topic 0.0.3378616 during attempt 166. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:16.229Z - Error subscribing to topic 0.0.3378616 during attempt 167. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:16.229Z - Fire timeout for attempt 84 (delay 20000)
2024-04-01T09:34:16.229Z - Error subscribing to topic 0.0.3378616 during attempt 168. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:16.229Z - Error subscribing to topic 0.0.3378616 during attempt 169. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:16.229Z - Fire timeout for attempt 85 (delay 20000)
2024-04-01T09:34:16.230Z - Error subscribing to topic 0.0.3378616 during attempt 170. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:16.230Z - Error subscribing to topic 0.0.3378616 during attempt 171. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:16.230Z - Fire timeout for attempt 86 (delay 20000)
2024-04-01T09:34:16.230Z - Error subscribing to topic 0.0.3378616 during attempt 172. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:16.230Z - Error subscribing to topic 0.0.3378616 during attempt 173. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:16.231Z - Fire timeout for attempt 87 (delay 20000)
2024-04-01T09:34:16.232Z - Error subscribing to topic 0.0.3378616 during attempt 174. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:16.232Z - Error subscribing to topic 0.0.3378616 during attempt 175. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:16.232Z - Fire timeout for attempt 88 (delay 20000)
2024-04-01T09:34:16.233Z - Error subscribing to topic 0.0.3378616 during attempt 176. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:16.233Z - Error subscribing to topic 0.0.3378616 during attempt 177. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:16.233Z - Fire timeout for attempt 89 (delay 20000)
2024-04-01T09:34:16.233Z - Error subscribing to topic 0.0.3378616 during attempt 178. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:16.233Z - Error subscribing to topic 0.0.3378616 during attempt 179. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:16.234Z - Fire timeout for attempt 90 (delay 20000)
2024-04-01T09:34:16.236Z - Error subscribing to topic 0.0.3378616 during attempt 180. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:16.236Z - Error subscribing to topic 0.0.3378616 during attempt 181. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:16.236Z - Fire timeout for attempt 91 (delay 20000)
2024-04-01T09:34:16.237Z - Error subscribing to topic 0.0.3378616 during attempt 182. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:16.237Z - Error subscribing to topic 0.0.3378616 during attempt 183. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:16.238Z - Fire timeout for attempt 92 (delay 20000)
2024-04-01T09:34:16.239Z - Error subscribing to topic 0.0.3378616 during attempt 184. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:16.239Z - Error subscribing to topic 0.0.3378616 during attempt 185. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:16.239Z - Fire timeout for attempt 93 (delay 20000)
2024-04-01T09:34:16.240Z - Error subscribing to topic 0.0.3378616 during attempt 186. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:16.240Z - Error subscribing to topic 0.0.3378616 during attempt 187. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:16.240Z - Fire timeout for attempt 94 (delay 20000)
2024-04-01T09:34:16.240Z - Error subscribing to topic 0.0.3378616 during attempt 188. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:16.240Z - Error subscribing to topic 0.0.3378616 during attempt 189. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:18.478Z - Fire timeout for attempt 95 (delay 20000)
2024-04-01T09:34:18.478Z - Error subscribing to topic 0.0.3378616 during attempt 190. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:18.478Z - Error subscribing to topic 0.0.3378616 during attempt 191. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:18.478Z - Fire timeout for attempt 96 (delay 20000)
2024-04-01T09:34:18.479Z - Error subscribing to topic 0.0.3378616 during attempt 192. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:18.479Z - Error subscribing to topic 0.0.3378616 during attempt 193. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:18.479Z - Fire timeout for attempt 97 (delay 20000)
2024-04-01T09:34:18.479Z - Error subscribing to topic 0.0.3378616 during attempt 194. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:18.479Z - Error subscribing to topic 0.0.3378616 during attempt 195. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:18.479Z - Fire timeout for attempt 98 (delay 20000)
2024-04-01T09:34:18.480Z - Error subscribing to topic 0.0.3378616 during attempt 196. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:18.480Z - Error subscribing to topic 0.0.3378616 during attempt 197. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:18.480Z - Fire timeout for attempt 99 (delay 20000)
2024-04-01T09:34:18.481Z - Error subscribing to topic 0.0.3378616 during attempt 198. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:18.482Z - Error subscribing to topic 0.0.3378616 during attempt 199. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:18.482Z - Fire timeout for attempt 100 (delay 20000)
2024-04-01T09:34:18.482Z - Error subscribing to topic 0.0.3378616 during attempt 200. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:18.482Z - Error subscribing to topic 0.0.3378616 during attempt 201. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:18.482Z - Fire timeout for attempt 101 (delay 20000)
2024-04-01T09:34:18.483Z - Error subscribing to topic 0.0.3378616 during attempt 202. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:18.483Z - Error subscribing to topic 0.0.3378616 during attempt 203. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:18.483Z - Fire timeout for attempt 102 (delay 20000)
2024-04-01T09:34:18.484Z - Error subscribing to topic 0.0.3378616 during attempt 204. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:18.484Z - Error subscribing to topic 0.0.3378616 during attempt 205. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:18.484Z - Fire timeout for attempt 103 (delay 20000)
2024-04-01T09:34:18.484Z - Error subscribing to topic 0.0.3378616 during attempt 206. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:18.484Z - Error subscribing to topic 0.0.3378616 during attempt 207. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:18.485Z - Fire timeout for attempt 104 (delay 20000)
2024-04-01T09:34:18.485Z - Error subscribing to topic 0.0.3378616 during attempt 208. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:18.485Z - Error subscribing to topic 0.0.3378616 during attempt 209. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:18.486Z - Fire timeout for attempt 105 (delay 20000)
2024-04-01T09:34:18.487Z - Error subscribing to topic 0.0.3378616 during attempt 210. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:18.487Z - Error subscribing to topic 0.0.3378616 during attempt 211. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:18.487Z - Fire timeout for attempt 106 (delay 20000)
2024-04-01T09:34:18.488Z - Error subscribing to topic 0.0.3378616 during attempt 212. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:18.488Z - Error subscribing to topic 0.0.3378616 during attempt 213. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:18.488Z - Fire timeout for attempt 107 (delay 20000)
2024-04-01T09:34:18.488Z - Error subscribing to topic 0.0.3378616 during attempt 214. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:18.488Z - Error subscribing to topic 0.0.3378616 during attempt 215. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:18.488Z - Fire timeout for attempt 108 (delay 20000)
2024-04-01T09:34:18.489Z - Error subscribing to topic 0.0.3378616 during attempt 216. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:18.489Z - Error subscribing to topic 0.0.3378616 during attempt 217. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:18.489Z - Fire timeout for attempt 109 (delay 20000)
2024-04-01T09:34:18.490Z - Error subscribing to topic 0.0.3378616 during attempt 218. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:18.490Z - Error subscribing to topic 0.0.3378616 during attempt 219. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:18.490Z - Fire timeout for attempt 110 (delay 20000)
2024-04-01T09:34:18.490Z - Error subscribing to topic 0.0.3378616 during attempt 220. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:18.490Z - Error subscribing to topic 0.0.3378616 during attempt 221. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:22.473Z - Fire timeout for attempt 111 (delay 20000)
2024-04-01T09:34:22.474Z - Error subscribing to topic 0.0.3378616 during attempt 222. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:22.474Z - Error subscribing to topic 0.0.3378616 during attempt 223. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:22.474Z - Fire timeout for attempt 112 (delay 20000)
2024-04-01T09:34:22.474Z - Error subscribing to topic 0.0.3378616 during attempt 224. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:22.474Z - Error subscribing to topic 0.0.3378616 during attempt 225. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:22.475Z - Fire timeout for attempt 113 (delay 20000)
2024-04-01T09:34:22.475Z - Error subscribing to topic 0.0.3378616 during attempt 226. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:22.475Z - Error subscribing to topic 0.0.3378616 during attempt 227. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:22.475Z - Fire timeout for attempt 114 (delay 20000)
2024-04-01T09:34:22.475Z - Error subscribing to topic 0.0.3378616 during attempt 228. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:22.475Z - Error subscribing to topic 0.0.3378616 during attempt 229. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:22.475Z - Fire timeout for attempt 115 (delay 20000)
2024-04-01T09:34:22.476Z - Error subscribing to topic 0.0.3378616 during attempt 230. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:22.476Z - Error subscribing to topic 0.0.3378616 during attempt 231. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:22.476Z - Fire timeout for attempt 116 (delay 20000)
2024-04-01T09:34:22.476Z - Error subscribing to topic 0.0.3378616 during attempt 232. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:22.476Z - Error subscribing to topic 0.0.3378616 during attempt 233. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:22.476Z - Fire timeout for attempt 117 (delay 20000)
2024-04-01T09:34:22.477Z - Error subscribing to topic 0.0.3378616 during attempt 234. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:22.477Z - Error subscribing to topic 0.0.3378616 during attempt 235. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:22.477Z - Fire timeout for attempt 118 (delay 20000)
2024-04-01T09:34:22.477Z - Error subscribing to topic 0.0.3378616 during attempt 236. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:22.477Z - Error subscribing to topic 0.0.3378616 during attempt 237. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:22.477Z - Fire timeout for attempt 119 (delay 20000)
2024-04-01T09:34:22.479Z - Error subscribing to topic 0.0.3378616 during attempt 238. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:22.479Z - Error subscribing to topic 0.0.3378616 during attempt 239. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:22.479Z - Fire timeout for attempt 120 (delay 20000)
2024-04-01T09:34:22.479Z - Error subscribing to topic 0.0.3378616 during attempt 240. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:22.479Z - Error subscribing to topic 0.0.3378616 during attempt 241. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:22.479Z - Fire timeout for attempt 121 (delay 20000)
2024-04-01T09:34:22.479Z - Error subscribing to topic 0.0.3378616 during attempt 242. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:22.479Z - Error subscribing to topic 0.0.3378616 during attempt 243. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:22.479Z - Fire timeout for attempt 122 (delay 20000)
2024-04-01T09:34:22.480Z - Error subscribing to topic 0.0.3378616 during attempt 244. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:22.480Z - Error subscribing to topic 0.0.3378616 during attempt 245. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:22.481Z - Fire timeout for attempt 123 (delay 20000)
2024-04-01T09:34:22.481Z - Error subscribing to topic 0.0.3378616 during attempt 246. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:22.481Z - Error subscribing to topic 0.0.3378616 during attempt 247. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:22.481Z - Fire timeout for attempt 124 (delay 20000)
2024-04-01T09:34:22.482Z - Error subscribing to topic 0.0.3378616 during attempt 248. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:22.482Z - Error subscribing to topic 0.0.3378616 during attempt 249. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:22.482Z - Fire timeout for attempt 125 (delay 20000)
2024-04-01T09:34:22.482Z - Error subscribing to topic 0.0.3378616 during attempt 250. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:22.482Z - Error subscribing to topic 0.0.3378616 during attempt 251. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:22.482Z - Fire timeout for attempt 126 (delay 20000)
2024-04-01T09:34:22.482Z - Error subscribing to topic 0.0.3378616 during attempt 252. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:22.483Z - Error subscribing to topic 0.0.3378616 during attempt 253. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:34.554Z - Fire timeout for attempt 127 (delay 20000)
2024-04-01T09:34:34.555Z - Error subscribing to topic 0.0.3378616 during attempt 254. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:34.555Z - Error subscribing to topic 0.0.3378616 during attempt 255. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:34.555Z - Fire timeout for attempt 128 (delay 20000)
2024-04-01T09:34:34.555Z - Error subscribing to topic 0.0.3378616 during attempt 256. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:34.556Z - Error subscribing to topic 0.0.3378616 during attempt 257. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:34.556Z - Fire timeout for attempt 129 (delay 20000)
2024-04-01T09:34:34.556Z - Error subscribing to topic 0.0.3378616 during attempt 258. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:34.556Z - Error subscribing to topic 0.0.3378616 during attempt 259. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:34.556Z - Fire timeout for attempt 130 (delay 20000)
2024-04-01T09:34:34.557Z - Error subscribing to topic 0.0.3378616 during attempt 260. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:34.557Z - Error subscribing to topic 0.0.3378616 during attempt 261. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:34.557Z - Fire timeout for attempt 131 (delay 20000)
2024-04-01T09:34:34.558Z - Error subscribing to topic 0.0.3378616 during attempt 262. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:34.558Z - Error subscribing to topic 0.0.3378616 during attempt 263. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:34.558Z - Fire timeout for attempt 132 (delay 20000)
2024-04-01T09:34:34.558Z - Error subscribing to topic 0.0.3378616 during attempt 264. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:34.558Z - Error subscribing to topic 0.0.3378616 during attempt 265. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:34.558Z - Fire timeout for attempt 133 (delay 20000)
2024-04-01T09:34:34.558Z - Error subscribing to topic 0.0.3378616 during attempt 266. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:34.558Z - Error subscribing to topic 0.0.3378616 during attempt 267. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:34.559Z - Fire timeout for attempt 134 (delay 20000)
2024-04-01T09:34:34.559Z - Error subscribing to topic 0.0.3378616 during attempt 268. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:34.559Z - Error subscribing to topic 0.0.3378616 during attempt 269. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:34.559Z - Fire timeout for attempt 135 (delay 20000)
2024-04-01T09:34:34.559Z - Error subscribing to topic 0.0.3378616 during attempt 270. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:34.559Z - Error subscribing to topic 0.0.3378616 during attempt 271. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:34.559Z - Fire timeout for attempt 136 (delay 20000)
2024-04-01T09:34:34.560Z - Error subscribing to topic 0.0.3378616 during attempt 272. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:34.560Z - Error subscribing to topic 0.0.3378616 during attempt 273. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:34.560Z - Fire timeout for attempt 137 (delay 20000)
2024-04-01T09:34:34.560Z - Error subscribing to topic 0.0.3378616 during attempt 274. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:34.560Z - Error subscribing to topic 0.0.3378616 during attempt 275. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:34.562Z - Fire timeout for attempt 138 (delay 20000)
2024-04-01T09:34:34.562Z - Error subscribing to topic 0.0.3378616 during attempt 276. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:34.562Z - Error subscribing to topic 0.0.3378616 during attempt 277. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:34.562Z - Fire timeout for attempt 139 (delay 20000)
2024-04-01T09:34:34.564Z - Error subscribing to topic 0.0.3378616 during attempt 278. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:34.564Z - Error subscribing to topic 0.0.3378616 during attempt 279. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:34.564Z - Fire timeout for attempt 140 (delay 20000)
2024-04-01T09:34:34.564Z - Error subscribing to topic 0.0.3378616 during attempt 280. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:34.564Z - Error subscribing to topic 0.0.3378616 during attempt 281. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:34.564Z - Fire timeout for attempt 141 (delay 20000)
2024-04-01T09:34:34.565Z - Error subscribing to topic 0.0.3378616 during attempt 282. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:34.565Z - Error subscribing to topic 0.0.3378616 during attempt 283. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:34.565Z - Fire timeout for attempt 142 (delay 20000)
2024-04-01T09:34:34.565Z - Error subscribing to topic 0.0.3378616 during attempt 284. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:34.565Z - Error subscribing to topic 0.0.3378616 during attempt 285. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:35.228Z - Fire timeout for attempt 143 (delay 20000)
2024-04-01T09:34:35.229Z - Error subscribing to topic 0.0.3378616 during attempt 286. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:35.229Z - Error subscribing to topic 0.0.3378616 during attempt 287. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:35.229Z - Fire timeout for attempt 144 (delay 20000)
2024-04-01T09:34:35.230Z - Error subscribing to topic 0.0.3378616 during attempt 288. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:35.230Z - Error subscribing to topic 0.0.3378616 during attempt 289. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:35.230Z - Fire timeout for attempt 145 (delay 20000)
2024-04-01T09:34:35.230Z - Error subscribing to topic 0.0.3378616 during attempt 290. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:35.230Z - Error subscribing to topic 0.0.3378616 during attempt 291. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:35.230Z - Fire timeout for attempt 146 (delay 20000)
2024-04-01T09:34:35.230Z - Error subscribing to topic 0.0.3378616 during attempt 292. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:35.230Z - Error subscribing to topic 0.0.3378616 during attempt 293. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:35.232Z - Fire timeout for attempt 147 (delay 20000)
2024-04-01T09:34:35.232Z - Error subscribing to topic 0.0.3378616 during attempt 294. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:35.233Z - Error subscribing to topic 0.0.3378616 during attempt 295. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:35.233Z - Fire timeout for attempt 148 (delay 20000)
2024-04-01T09:34:35.233Z - Error subscribing to topic 0.0.3378616 during attempt 296. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:35.233Z - Error subscribing to topic 0.0.3378616 during attempt 297. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:35.233Z - Fire timeout for attempt 149 (delay 20000)
2024-04-01T09:34:35.233Z - Error subscribing to topic 0.0.3378616 during attempt 298. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:35.233Z - Error subscribing to topic 0.0.3378616 during attempt 299. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:35.234Z - Fire timeout for attempt 150 (delay 20000)
2024-04-01T09:34:35.234Z - Error subscribing to topic 0.0.3378616 during attempt 300. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:35.234Z - Error subscribing to topic 0.0.3378616 during attempt 301. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:35.234Z - Fire timeout for attempt 151 (delay 20000)
2024-04-01T09:34:35.234Z - Error subscribing to topic 0.0.3378616 during attempt 302. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:35.234Z - Error subscribing to topic 0.0.3378616 during attempt 303. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:35.234Z - Fire timeout for attempt 152 (delay 20000)
2024-04-01T09:34:35.235Z - Error subscribing to topic 0.0.3378616 during attempt 304. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:35.235Z - Error subscribing to topic 0.0.3378616 during attempt 305. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:35.235Z - Fire timeout for attempt 153 (delay 20000)
2024-04-01T09:34:35.235Z - Error subscribing to topic 0.0.3378616 during attempt 306. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:35.235Z - Error subscribing to topic 0.0.3378616 during attempt 307. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:35.235Z - Fire timeout for attempt 154 (delay 20000)
2024-04-01T09:34:35.236Z - Error subscribing to topic 0.0.3378616 during attempt 308. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:35.236Z - Error subscribing to topic 0.0.3378616 during attempt 309. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:35.236Z - Fire timeout for attempt 155 (delay 20000)
2024-04-01T09:34:35.236Z - Error subscribing to topic 0.0.3378616 during attempt 310. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:35.236Z - Error subscribing to topic 0.0.3378616 during attempt 311. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:35.236Z - Fire timeout for attempt 156 (delay 20000)
2024-04-01T09:34:35.236Z - Error subscribing to topic 0.0.3378616 during attempt 312. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:35.236Z - Error subscribing to topic 0.0.3378616 during attempt 313. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:35.238Z - Fire timeout for attempt 157 (delay 20000)
2024-04-01T09:34:35.238Z - Error subscribing to topic 0.0.3378616 during attempt 314. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established
2024-04-01T09:34:35.238Z - Error subscribing to topic 0.0.3378616 during attempt 315. Waiting 20000 ms before next attempt: No connection established
2024-04-01T09:34:35.238Z - Fire timeout for attempt 158 (delay 20000)
2024-04-01T09:34:35.238Z - Error subscribing to topic 0.0.3378616 during attempt 316. Waiting 20000 ms before next attempt: 14 UNAVAILABLE: No connection established

Hedera network

other

Version

v2.40.0

Operating system

Linux
@bpatrzyk bpatrzyk added the bug Something isn't working label Apr 1, 2024
@agadzhalov
Copy link
Contributor

We've managed to reproduce the problem and we want to double check on the use case and on the impact of the issue

  1. Is this a blocking problem? If yes how exactly?
  2. Why you are testing against a stopped local-node? May be in case of network disconnection?
  3. For example you are trying to subscribe to a Topic, your internet connection goes down (or was down at the first place) and then we would witness the unexpected exponential rise in the number of retries?
  4. Is there any other use cases a stopped local-node is required?

@bpatrzyk

@bpatrzyk
Copy link
Author

bpatrzyk commented Apr 17, 2024 •
edited

Thank you for addressing the issue.
I see that using a stopped (or non-existent) local node for testing caused some confusion, sorry.

Let me explain our usecase. We have a backend deployed to an on-prem server that subscribes to a topic. We noticed that from time to time (say, every 2-3 days) the backend disconnects from the topic and it does not reconnect automatically (it tries 10 times and fails) - so our backend requires a manual restart to re-subscribe to a topic. We suspect that the initial disconnect is caused by temporary problem with the Internet connection.
What I tried to do is increasing the max number of reconnect attempts but I experienced the aforementioned problems with exponential rise of retries.

  1. Is this a blocking problem? If yes how exactly?

It is a blocking problem. We built a workaround to monitor the topic subscription and re-create it if necessary but it is not a good solution and we would prefer it if the reconnect could be handled by the library automatically.

  1. Why you are testing against a stopped local-node? May be in case of network disconnection?

It was easy to test the reconnects in a repetitive and isolated environment (without dropping my network access) that way. It does not have anything to do with our production setup. In staging and prod environments we use testnet and mainnet.

  1. For example you are trying to subscribe to a Topic, your internet connection goes down (or was down at the first place) and then we would witness the unexpected exponential rise in the number of retries?

When the max attempts is set to a large number and:

  1. there is no Internet connection when we first subscribe to a topic or
  2. the internet connection goes down after we successfully subscribe to a topic
    then we experience the unexpected exponential rise in the number of retires
  1. Is there any other use cases a stopped local-node is required?

No, not at all. We only used it for reproducing this issue. We are typically using testnet and mainnet for development and production respectively.

@agadzhalov
Copy link
Contributor

Thank you for the detailed explanation, understanding the use case is a key part from understanding the issue in deep details. At this point we've only managed to reproduce it, but we will need additional time for debugging and fixing it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bpatrzyk @agadzhalov