Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Error: recovered: Block (unix) time must never be empty or negative nstack:ngoroutine 252403 on block 15488803 #1115

Closed
taariq opened this issue Apr 5, 2024 · 5 comments
Assignees
Labels
bug Something isn't working

Comments

@taariq
Copy link
Contributor

taariq commented Apr 5, 2024

What is happening?

Section description Provide as much context as you can. Give as much context as you can to make it easier for the developers to figure what is happening.

Unable to get past block 15488803. Error output:

mainnet-validator palomad[1999487]: {"level":"error","module":"server","err":"recovered: Block (unix) time must never be empty or negative \nstack:\ngoroutine 252403 [running]:\nruntime/debug.Stack()\n\t/opt/hostedtoolcache/go/1.21.8/x64/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/cosmos/cosmos-sdk/baseapp.NewBaseApp.newDefaultRecoveryMiddleware.func5({0x2d03080?, 0x3bfcee0})\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/recovery.go:74 +0x26\ngithub.com/cosmos/cosmos-sdk/baseapp.NewBaseApp.newDefaultRecoveryMiddleware.newRecoveryMiddleware.func7({0x2d03080?, 0x3bfcee0?})\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/recovery.go:42 +0x2d\ngithub.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x2d03080, 0x3bfcee0}, 0x0?)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/recovery.go:31 +0x2f\ngithub.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x2d03080, 0x3bfcee0}, 0x0?)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/recovery.go:36 +0x53\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx.func1()\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/baseapp.go:836 +0x136\npanic({0x2d03080?, 0x3bfcee0?})\n\t/opt/hostedtoolcache/go/1.21.8/x64/src/runtime/panic.go:920 +0x270\ngithub.com/CosmWasm/wasmd/x/wasm/types.NewEnv({{0x3c3f5f8, 0x5aefe00}, {0x3c5b680, 0xc00c251a80}, {{0x0, 0x0}, {0xc002301700, 0x9}, 0x0, {0x0, ...}, ...}, ...}, ...)\n\t/home/runner/go/pkg/mod/github.com/!cosm!wasm/wasmd@v0.50.0/x/wasm/types/types.go:278 +0x398\ngithub.com/CosmWasm/wasmd/x/wasm/keeper.Keeper.execute({{0x3c04ea0, 0xc000708ae0}, {0x3c7d068, 0xc002303cb0}, {0x3c1d4d0, 0xc00150be00}, {0x3c06340, 0xc002344720}, {0x3c04aa0, 0xc0009ca918}, ...}, ...)\n\t/home/runner/go/pkg/mod/github.com/!cosm!wasm/wasmd@v0.50.0/x/wasm/keeper/keeper.go:395 +0x6f0\ngithub.com/CosmWasm/wasmd/x/wasm/keeper.msgServer.ExecuteContract({0x4122da?}, {0x3c40058, 0xc00cf1a7b0}, 0xc00cedc1e0)\n\t/home/runner/go/pkg/mod/github.com/!cosm!wasm/wasmd@v0.50.0/x/wasm/keeper/msg_server.go:124 +0x1ba\ngithub.com/CosmWasm/wasmd/x/wasm/types._Msg_ExecuteContract_Handler.func1({0x3c40058, 0xc00cf1a7b0}, {0x3278580?, 0xc00cedc1e0})\n\t/home/runner/go/pkg/mod/github.com/!cosm!wasm/wasmd@v0.50.0/x/wasm/types/tx.pb.go:2264 +0x72\ngithub.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).registerMsgServiceHandler.func2.1({0x3c3f198, 0xc00cf0b180}, {0xc00eb6b500?, 0x411da5?}, 0x358?, 0xc00e5b5170)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/msg_service_router.go:175 +0x93\ngithub.com/CosmWasm/wasmd/x/wasm/types.Msg_ExecuteContract_Handler({0x324f3c0?, 0xc0001b11e0}, {0x3c3f198, 0xc00cf0b180}, 0x361beb0, 0xc00c276260)\n\t/home/runner/go/pkg/mod/github.com/!cosm!wasm/wasmd@v0.50.0/x/wasm/types/tx.pb.go:2266 +0x135\ngithub.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).registerMsgServiceHandler.func2({{0x3c3f5f8, 0x5aefe00}, {0x3c5b680, 0xc00c251a80}, {{0x0, 0x0}, {0xc002301700, 0x9}, 0x0, {0x0, ...}, ...}, ...}, ...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/msg_service_router.go:198 +0x363\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(, {{0x3c3f5f8, 0x5aefe00}, {0x3c5b680, 0xc00c251a80}, {{0x0, 0x0}, {0xc002301700, 0x9}, 0x0, ...}, ...}, ...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/baseapp.go:1009 +0x1e6\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx(0xc0015fc480, 0x2, {0xc000cd1980, 0x168, 0x180})\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/baseapp.go:947 +0x116c\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Simulate(...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/test_helpers.go:28\ngithub.com/cosmos/cosmos-sdk/x/auth/tx.txServer.Simulate({{{0x0, 0x0, 0x0}, {0x3c637a0, 0xc005c0aa50}, 0x0, {0xc000a33d30, 0x9}, {0x3c7d068, 0xc000ec39b0}, ...}, ...}, ...)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/x/auth/tx/service.go:98 +0x129\ngithub.com/cosmos/cosmos-sdk/types/tx._Service_Simulate_Handler.func1({0x3c40058, 0xc00cee2300}, {0x31ed560?, 0xc00c209580})\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/types/tx/service.pb.go:1443 +0x72\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func1({0x3c40058, 0xc00c0ed500}, {0x31ed560, 0xc00c209580}, 0xc00c2095a0?, 0xc00cf49d58)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/grpcserver.go:70 +0x458\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func2.ChainUnaryServer.func2.1({0x3c40058?, 0xc00c0ed500?}, {0x31ed560?, 0xc00c209580?})\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48 +0x45\ngithub.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x3c40058?, 0xc00c0ed500?}, {0x31ed560?, 0xc00c209580?}, 0xc00cf49d58?, 0x0?)\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/recovery/interceptors.go:33 +0xb0\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func2.ChainUnaryServer.func2({0x3c40058, 0xc00c0ed500}, {0x31ed560, 0xc00c209580}, 0xc00c2095a0, 0x414645?)\n\t/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:53 +0x135\ngithub.com/cosmos/cosmos-sdk/types/tx._Service_Simulate_Handler({0x31b7ac0?, 0xc00099ad20}, {0x3c40058, 0xc00c0ed500}, 0xc00c0ca600, 0xc00c0ed530)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/types/tx/service.pb.go:1445 +0x135\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func2({0x31b7ac0, 0xc00099ad20}, {0x3c40058, 0xc00c0ed500}, 0x32c7180?, 0xc00c0898c0?)\n\t/home/runner/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.50.5/baseapp/grpcserver.go:84 +0x112\ngoogle.golang.org/grpc.(*Server).processUnaryRPC(0xc00157fa00, {0x3c40058, 0xc00c0ed470}, {0x3c578c0, 0xc006737040}, 0xc00c0898c0, 0xc0066fea20, 0xc00520cee0, 0x0)\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.0/server.go:1383 +0xe03\ngoogle.golang.org/grpc.(*Server).handleStream(0xc00157fa00, {0x3c578c0, 0xc006737040}, 0xc00c0898c0)\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.0/server.go:1794 +0x100c\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1()\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.0/server.go:1027 +0x8b\ncreated by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 1207\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.0/server.go:1038 +0x135\n: panic","time":"2024-04-05T20:45:26Z","message":"panic recovered in runTx"}

Paloma and pigeon versions and logs

Section description Write down paloma version. Write down pigeon version. Copy and paste pigeon config file as well as relevant ENV variables.

{WRITE YOUR ANSWER HERE}
Paloma

How to reproduce?

Section description Please write detailed steps of what you were doing for this bug to appear.

What is the expected behaviour?

Section description If you know, please write down what is the expected behaviour. If you don't know, that's ok. We can have a discussion in comments.

Cosmwasm time must be properly set in all contracts whose state is validated.

@taariq
Copy link
Contributor Author

taariq commented Apr 18, 2024

Fixed with #1132

@taariq taariq closed this as completed Apr 18, 2024
@verabehr verabehr reopened this May 13, 2024
@verabehr
Copy link
Collaborator

@taariq
Copy link
Contributor Author

taariq commented May 13, 2024

Via the Cosmwasm Discord: https://discord.com/channels/737637324434833438/737640672680607764/1238178635463004192

For us, it manifested from consensus issues, since we had wasm executing in the AnteHandler. Consensus headers are what set that time value, and the default val is 0
If you are executing wasm in the antehandler then it's probably the same issue

@taariq
Copy link
Contributor Author

taariq commented May 20, 2024

@verabehr let's put a tracking ticket in Volume for this one.

@verabehr
Copy link
Collaborator

This could no longer be produced locally or on the lcd mainnet machine

I did see a Block (unix) time must never be empty or negative error at first, but it only happened when the address book was empty. After copying the mainnet address book, it started normally

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Done
Development

No branches or pull requests

5 participants