Possible issue with Buddy on Windows

[donhardman]

Bug Description:

We seem to have a peculiar issue while using Buddy on Windows. We should try to replicate it on another machine and validate if it's a localized issue or a more widespread problem.

Due to the changes we've made in how we work with mysqldump, we should use the following Manticore search version:

root@dev:/workdir# searchd --version
Manticore 6.2.13 8850df782@24042916 dev (columnar 2.2.5 478fff2@240417) (secondary 2.2.5 478fff2@240417) (knn 2.2.5 478fff2@240417)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2024, Manticore Software LTD (https://manticoresearch.com)

To reproduce the issue, we should run the restoration with mysqldump:

mysql -h0 -P9306 < bench.sql

P.S. The file is available on dev2 at /tmp/bench.tar.gz

If we can insert everything without encountering timeout errors in the log, that's great (probably a local issue). However, if we're unable to insert and consistently encounter timeouts, resulting in a failed restoration, we should investigate what's causing the problem.

I tried to reproduce it on Linux, but I couldn't succeed; it worked fine.

Manticore Search Version:

6.2.13 8850df782@24042916 dev

Operating System Version:

Windows

Have you tried the latest development version?

• Yes

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

• Task estimated
• Specification created, reviewed, and approved
• Implementation completed
• Tests developed
• Documentation updated
• Documentation proofread
• Changelog updated

[tomatolog]

what to expect on this ticket?" could you provide error message from the ?

mysql -h0 -P9306 < bench.sql

could you upload searchd.log after issue happened?

[tomatolog]

seems the dump is invalid as I got error right on the start

gh2116>mysql -h 127.0.0.1 -P9315 < bench.sql
ERROR 1064 (42000) at line 37: KNN error: data has 0 values, index 'vector' needs 960 values

the daemon version is

starting daemon version '6.2.13 50c41a730@24042214 dev (columnar 2.2.5 478fff2@240417) (secondary 2.2.5 478fff2@240417) (knn 2.2.5 478fff2@240417)' ...

[tomatolog]

seems I reproduced this issue - the Buddy error at the daemon log

rt: table bench: optimized progressive chunk(s) 26 ( left 24 ) in 1s 363ms
[BUDDY] Fatal error: Uncaught ErrorException: file_get_contents(http://host.docker.internal:9316/sql?mode=raw): Failed to open stream: Connection timed out in /buddy/vendor/manticoresoftware/buddy-core/src/ManticoreSearch/Client.php:130
[BUDDY] Stack trace:
[BUDDY] #0 [internal function]: buddy_error_handler(2, 'file_get_conten...', '/buddy/vendor/m...', 130)
[BUDDY] #1 /buddy/vendor/manticoresoftware/buddy-core/src/ManticoreSearch/Client.php(130): file_get_contents('http://host.doc...', false, Resource id #804)
[BUDDY] #2 /buddy/vendor/manticoresoftware/buddy-core/src/ManticoreSearch/Client.php(216): Manticoresearch\Buddy\Core\ManticoreSearch\Client->sendRequest('query=SHOW+TABL...')
[BUDDY] #3 /buddy/src/Plugin/Sharding/State.php(152): Manticoresearch\Buddy\Core\ManticoreSearch\Client->hasTable('sharding_state')
[BUDDY] #4 /buddy/src/Plugin/Sharding/Operator.php(161): Manticoresearch\Buddy\Base\Plugin\Sharding\State->isActive()
[BUDDY] #5 /buddy/src/Plugin/Sharding/Processor.php(53): Manticoresearch\Buddy\Base\Plugin\Sharding\Operator->hasSharding()
[BUDDY] #6 /buddy/vendor/manticoresoftware/buddy-core/src/Process/BaseProcessor.php(85): Manticoresearch\Buddy\Base\Plugin\Sharding\Processor->ping()
[BUDDY] #7 /buddy/vendor/manticoresoftware/buddy-core/src/Process/Process.php(150): Manticoresearch\Buddy\Core\Process\BaseProcessor->Manticoresearch\Buddy\Core\Process\{closure}()
[BUDDY] #8 [internal function]: Manticoresearch\Buddy\Core\Process\Process::Manticoresearch\Buddy\Core\Process\{closure}(Object(Swoole\Process))
[BUDDY] #9 /buddy/src/Network/Server.php(259): Swoole\Server->start()
[BUDDY] #10 /buddy/src/main.php(158): Manticoresearch\Buddy\Base\Network\Server->start()
[BUDDY] #11 {main}
[BUDDY]   thrown in /buddy/vendor/manticoresoftware/buddy-core/src/ManticoreSearch/Client.php on line 130
[BUDDY] [2024-04-30 16:53:28 $1.0]      WARNING Server::check_worker_exit_status(): worker(pid=10, id=3) abnormal exit, status=255, signal=0

and the error message at the cli

gh2116>mysql -h 127.0.0.1 -P9315 < bench.sql
ERROR 1064 (42000) at line 1663: file_get_contents(http://host.docker.internal:9316/sql?mode=raw): Failed to open stream: Connection timed out

it inserted

RelWithDebInfo>mysql -h 127.0.0.1 -P 9315 -vvv -e "show table bench status like 'indexed%'"
--------------
show table bench status like 'indexed%'
--------------

+-------------------+--------+
| Variable_name     | Value  |
+-------------------+--------+
| indexed_documents | 195120 |

[tomatolog]

if I disable sharding at the buddy then

mysql -h 127.0.0.1 -P 9315 -vvv -e "DISABLE BUDDY PLUGIN 'manticoresoftware/buddy-plugin-sharding'"
Query OK, 0 rows affected (0.09 sec)

I got the same error around 40k docs posted

mysql -h 127.0.0.1 -P9315 < bench.sql
ERROR 1064 (42000) at line 420: file_get_contents(http://host.docker.internal:9316/sql?mode=raw): Failed to open stream: Connection timed out

however now daemon log has no any traces of the buddy error

[tomatolog]

I restarted buddy with --debug option
disabled sharding

mysql -h 127.0.0.1 -P 9315 -vvv -e "DISABLE BUDDY PLUGIN 'manticoresoftware/buddy-plugin-sharding'"
Query OK, 0 rows affected (0.09 sec)

and at the daemon console I see that buddy pauses during processing the mysql -h 127.0.0.1 -P9315 < bench.sql
and if the pause is long then buddy finish processing with the error

Here is a searchd.log with buddy output and rimestamps

[Wed May  1 09:50:05.683 2024] [8020] [BUDDY] [6632104da354a9.80708571] Plugin: InsertValues 
[Wed May  1 09:50:05.684 2024] [8020] [BUDDY] [6632104da354a9.80708571] InsertValues payload: {"table":"bench","values":["143704","'0.128100,0.083900,0.082000,0.106600,0.075700,0.035700,0.056600,0.046900,0.087500,0.050400, 
[Wed May  1 09:50:05.762 2024] [8020] [BUDDY] [6632104da354a9.80708571] response data: {"version":2,"type":"sql response","message":[{"total":120,"error":"","warning":""}],"error_code":200} 
[Wed May  1 09:50:05.777 2024] [8020] [BUDDY] [6632104dbb6422.63448936] request data: {"type":"unknown sql request","error":"KNN error: data has 0 values, index 'vector' needs 960 values","version":2,"user":"ODBC", 
[Wed May  1 09:50:05.777 2024] [8020] [BUDDY] [6632104dbb6422.63448936] Plugin: InsertValues 
[Wed May  1 09:50:05.777 2024] [8020] [BUDDY] [6632104dbb6422.63448936] InsertValues payload: {"table":"bench","values":["140176","'0.082000,0.067500,0.081300,0.079100,0.063500,0.050800,0.095800,0.063500,0.056000,0.077300, 
[Wed May  1 09:51:05.879 2024] [8020] [BUDDY] [6632104dbb6422.63448936] response data: {"version":2,"type":"sql response","message":{"error":"file_get_contents(http:\/\/host.docker.internal:9316\/sql?mode=raw): Failed to open stream: Connection timed out"},"error_code":0} 

from the daemon timestamp I see if InsertValues payload: ... to response data: ... took 1 second buddy returns error to client.

It is unclear what point of the error or what stack of the error in the buddy - someone should debug that in the buddy.

[sanikolaev]

@tomatolog pls check if you still can reproduce it in Window after manticoresoftware/manticoresearch-buddy#274 (comment) is fixed.

[tomatolog]

can not reproduce the issue on windows as got error

mysql -h0 -P9306 < bench.sql
ERROR 1064 (42000) at line 37: KNN error: data has 0 values, index 'vector' needs 960 values

on data from dev2 at /tmp/bench.tar.gz

I use buddy from the master branch 2963d15e2f0004f90fda72f35389ec0be57f901e and daemon from the master too

Manticore 6.2.13 6e43e69c2@24051510 dev (columnar 2.2.5 1d29b35@240514) (secondary 2.2.5 1d29b35@240514) (knn 2.2.5 1d29b35@240514)

now sure how to get further

[tomatolog]

I use fixed dump from /home/snikolaev/issue_56/bench_dump.tgz on dev2 and all data inserted fine on windows however I see no any activity related to buddy during that posting.
I also tried to restart daemon with buddy disabled in the daemon config and able to post that data fine.

Not sure how it might be related to initial description.

[donhardman]

There's a chance we fixed it after the latest updates. To confirm it's resolved, let's run a simple loop test that executes some queries multiple times, like show version or show queries. This will ensure the requests go to Buddy and include an internal call to Manticore.

If it works without any errors we encountered before, we can close the issue.

[tomatolog]

seems work fine at the master head of the daemon and head of the buddy source