etcd watch失败时，内存泄漏

[hauerwu]

Issue description

当etcd watch失败，goroutine退出时，会在defer func中将当前GenericStore加入到storeNeedReInit中

每两分钟会调用一次ReInit，重新将storeNeedReInit slice中的GenericStore初始化，又回调回到上面出问题的代码，再次加入storeNeedReInit
storeNeedReInit并没有清理的地方，会一直增大

Expected behavior

就算etcd异常，manager-api进程内存使用也不能持续增长

How to Reproduce

1.将etcd打断点断住，模拟etcd返回超时
2.反复在dashboard中进行操作
3.放开etcd断点
4.manager-api进程内存使用持续增长

Screenshots

No response

Environment

• apisix version (cmd: apisix version):
• OS (cmd: uname -a):
• OpenResty / Nginx version (cmd: nginx -V or openresty -V):
• etcd version, if have (cmd: run etcd --version):
• apisix-dashboard version, if have:
• Browser version, if have:

Additional context

No response

[hauerwu]

apisix version (cmd: apisix version):
3.8.0
OS (cmd: uname -a):
Linux apisix 5.15.0-92-generic #102~20.04.1-Ubuntu SMP Mon Jan 15 13:09:14 UTC 2024 x86_64 GNU/Linux

OpenResty / Nginx version (cmd: nginx -V or openresty -V):
nginx version: openresty/1.21.4.2
built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
built with OpenSSL 3.2.0 23 Nov 2023
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_RUNTIME_VER=1.1.1 -DNGX_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_HTTP_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl3/include' --add-module=../ngx_devel_kit-0.3.2 --add-module=../echo-nginx-module-0.63 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../srcache-nginx-module-0.33 --add-module=../ngx_lua-0.10.25 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.34 --add-module=../array-var-nginx-module-0.06 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.9 --add-module=../ngx_stream_lua-0.0.13 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/openresty/wasmtime-c-api/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl3/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl3/lib' --add-module=/tmp/tmp.JO0RQ1uMXH/openresty-1.21.4.2/../mod_dubbo-1.0.2 --add-module=/tmp/tmp.JO0RQ1uMXH/openresty-1.21.4.2/../ngx_multi_upstream_module-1.1.1 --add-module=/tmp/tmp.JO0RQ1uMXH/openresty-1.21.4.2/../apisix-nginx-module-1.15.0 --add-module=/tmp/tmp.JO0RQ1uMXH/openresty-1.21.4.2/../apisix-nginx-module-1.15.0/src/stream --add-module=/tmp/tmp.JO0RQ1uMXH/openresty-1.21.4.2/../apisix-nginx-module-1.15.0/src/meta --add-module=/tmp/tmp.JO0RQ1uMXH/openresty-1.21.4.2/../wasm-nginx-module-0.6.5 --add-module=/tmp/tmp.JO0RQ1uMXH/openresty-1.21.4.2/../lua-var-nginx-module-v0.5.3 --add-module=/tmp/tmp.JO0RQ1uMXH/openresty-1.21.4.2/../grpc-client-nginx-module-v0.4.4 --add-module=/tmp/tmp.JO0RQ1uMXH/openresty-1.21.4.2/../lua-resty-events-0.2.0 --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module

etcd version, if have (cmd: run etcd --version):
etcd Version: 3.5.9
Git SHA: bdbbde998
Go Version: go1.19.9
Go OS/Arch: linux/amd64

apisix-dashboard version, if have:
Version : 3.0.1
GitHash : 397c0cb

Browser version, if have:
Chrome
版本 120.0.6099.199（正式版本） (arm64)

[hauerwu]

测试环境长期运行，内存消耗21GB

[hanqingwu]

I have to the test .

1. start apisix with etcd normal , store gorouting work normal
   INFO store/store.go:391 watch start from gorouting 71 watch BasePath /apisix/routes

2. stop etcd to mock execption
   

3. restart etcd

4. etcdchecker timer trigger reinit
   etcd connection recovered, but after several connection losses, reinitializing stores, times: 20
   And all store Init Again
   
   INFO store/store.go:391 watch start from gorouting 341 watch BasePath /apisix/routes
   But now old store gorouting still working , I add some log in
   

5. I put a route in etcd , then watch 2 events from two different goroutings old and new
   

[hanqingwu]

When I repeat 2-5 then I got more and more new watch gorouting and old gorouting coexist