stack.sh tendermint node network error

[ezreal1997]

Describe the bug
After downloading ./stack.sh script from master branch and running it, I'm getting error for tendermint container exited.

To Reproduce
Just try running ./stack.sh

Expected behavior
The ./stack.sh script is deploying docker version of BigchainDB network with STACK_SIZE number of nodes including tendermint nodes.

Logs or terminal output
Here are the logs of tendermint containers:

Desktop (please complete the following information):

• Distribution: Ubuntu 18.04

[Davsen39]

@rayzui i have been having the same problem for 3 months but still locked and nobody answers to me. If you find a solution please write it here :)

[ezreal1997]

@zzappie Could u plz help us with this error？

[linpinn]

Hi, I am also stuck at a similar error. I've been trying to change some part that I think is relevant but none of them fix the "connection reset by peer" issue.
FYI: Ubuntu 18.04 LTS, Tendermint[0.31.5], BigchainDB[2.2.2]

First, somehow, the port in "bigchaindb/pkg/configuration/roles/bigchaindb/defaults/main.yml" is
"tendermint_abci_port: 45558"

But tendermint, if I'm not mistaken, is looking for 26658 in "bigchaindb/pkg/scripts/tm_start"
" --proxy_app="tcp://"$BIGCHAINDB_DOCKER_NAME$_ITEM":26658"
So, I change the port in main.yaml to 26658.

Next, I fix the node_key.json part in the "tm_config_gen" file because I saw that there is only one node_key where it is supposed to be 4. (And that fixed the node_key not found error)

But then I got another error about missing priv_validator_state.json and prov_validator_key.json. I found that the script should handle creating this file from this source. https://sourcegraph.com/github.com/tendermint/tendermint@aeb6cc475e14c9b4c4455111f6dfb54672507006/-/raw/UPGRADING.md

Here is what I do to the tm_config_gen and tm_start file

The last thing I did was eliminating the "no such host" error by swapping the ansible sequence to start Bigchaindb before Tendermint in "bigchaindb-start.ymal"

I'm not sure if all the things I did was correct but it fixed some errors in the log.

After all of the attempts, I do "docker exec -i -t bigchaindb1 /bin/sh" to check if bigchaindb was listening to the 26658 port. I use "netstat -ptan | grep LISTEN". I found that it was listening.

Now, I have no idea what went wrong. Please advise.

[davie0]

Hello!
@linpinn big thanks for investigation and providing fixes.
Strangely It worked without these changes last time I checked... see #2707
I suspect It doesn't work yet because of greenlet api change.
Now I get this in bigchaindb logs:

<frozen importlib._bootstrap>:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
<frozen importlib._bootstrap>:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
<frozen importlib._bootstrap>:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
<frozen importlib._bootstrap>:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
<frozen importlib._bootstrap>:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
<frozen importlib._bootstrap>:219: RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject
[2021-01-13 12:25:22] [DEBUG] (bigchaindb.config_utils) System already configured, skipping autoconfiguration (MainProcess - pid: 1)

It could be fixed by upgrading bigchaindb-abci gevent dependency to "gevent>=20.9" and hopefully it will go away. I'm testing this out

[linpinn]

Hi!
@zzappie, Thank you for your reply :)
I saw the issue #2707 and tried the solution you mention in the conversation but it didn't fix the problem for me.

So, do you know what causes the error? Does upgrading "gevent" works?

[davie0]

I've upgraded "gevent" in the stack-fix branch and applied fixes you provided. All docker images are up now and the "binary incompatibility" issue went away. But I get internal error when creating transactions I haven't yet debugged.

You can try changing branch from "master" to "stack-fix" in stack.sh and see whether it works for you.

[linpinn]

Hello @zzappie
Using the branch "stack-fix" fixes the connection problem and tendermint docker seems to work fine. But I think tendermint nodes have problem connecting to each other according to the logs.