Always meet "failed to open WAL" #31950
Replies: 4 comments 3 replies
-
"failed to open WAL" is thrown by etcd, which means the wal files of etcd are cracked. If you are using docker-compose to deploy milvus, the data paths of etcd/minio/milvus are mounted to a local folder named "volumes".
So, if no sufficient disk space for this path, milvus and the dependencies will run into trouble. Sometimes data files could be cracked when there is no disk space to write. Double-check the disk space, and delete the "volume" folder entirely. Start a new milvus agian. |
Beta Was this translation helpful? Give feedback.
-
seems to be a bug of etcd 3.5.5 @LoveEachDay can we upgrade to etcd 3.5.13? |
Beta Was this translation helpful? Give feedback.
-
and we need to figure out a way to fix if this shows |
Beta Was this translation helpful? Give feedback.
-
I installed Milvus v2.3.12 in a k8s environment with standalone mode. And meet "failed to open WAL" error after several days running. I try to reinstall a new server and import data to it for the first time, however this error occurs again.
The whole message is as follows:
{"level":"fatal","ts":"2024-04-07T06:28:25.789Z","caller":"etcdserver/storage.go:95","msg":"failed to open WAL","error":"fileutil: file already locked","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver.readWAL\n\t/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.5/etcdserver/storage.go:95\ngo.etcd.io/etcd/server/v3/etcdserver.restartNode\n\t/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.5/etcdserver/raft.go:527\ngo.etcd.io/etcd/server/v3/etcdserver.NewServer\n\t/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.5/etcdserver/server.go:534\ngo.etcd.io/etcd/server/v3/embed.StartEtcd\n\t/go/pkg/mod/go.etcd.io/etcd/server/v3@v3.5.5/embed/etcd.go:243\ngithub.com/milvus-io/milvus/pkg/util/etcd.InitEtcdServer.func1\n\t/go/src/github.com/milvus-io/milvus/pkg/util/etcd/etcd_server.go:52\nsync.(*Once).doSlow\n\t/usr/local/go/src/sync/once.go:74\nsync.(*Once).Do\n\t/usr/local/go/src/sync/once.go:65\ngithub.com/milvus-io/milvus/pkg/util/etcd.InitEtcdServer\n\t/go/src/github.com/milvus-io/milvus/pkg/util/etcd/etcd_server.go:37\ngithub.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:327\ngithub.com/milvus-io/milvus/cmd/milvus.(*run).execute\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:47\ngithub.com/milvus-io/milvus/cmd/milvus.RunMilvus\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60\nmain.main\n\t/go/src/github.com/milvus-io/milvus/cmd/main.go:95\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
Anyone can help with this ? What is the root cause and is there some way to fix this ? Thanks very much.
Beta Was this translation helpful? Give feedback.
All reactions