Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Help] CPU utilization remains consistently high after complex hybrid searches utilizing MT mode. #4489

Open
m3ngyang opened this issue Feb 22, 2024 · 3 comments
Open

[Help] CPU utilization remains consistently high after complex hybrid searches utilizing MT mode. #4489

m3ngyang opened this issue Feb 22, 2024 · 3 comments
Assignees
@nafraf
Labels
stale

Comments

@m3ngyang
Copy link

Describe the bug

Hello, I experimented with Redisearch using REDISEARCH_MT_BUILD=1, conducted some intricate hybrid searches (condition+vector similarity), and subsequently noticed that latency was significantly high while the CPU usage remained elevated, even after halting the requests.

Condition: ((@tag1:{1|2})(-@tag2:{3|4|5|6})(-@tag1:{7|8|9|10}))

Here is my redis.conf

# MODULES
loadmodule /usr/lib/redis/modules/redisearch.so TIMEOUT 600 MT_MODE MT_MODE_FULL WORKER_THREADS 7
loadmodule /usr/lib/redis/modules/rejson.so

# Security
protected-mode no

# NETWORK

# Close the connection after a client is idle for N seconds (0 to disable)
timeout 3600

# THREADED I/O
io-threads 4
# Redis will use I/O threads to perform read operations,
# which can improve performance by allowing Redis to read data from multiple clients simultaneously.
io-threads-do-reads yes
image

I checked the redis.log

35:C 22 Feb 2024 16:26:44.236 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
35:C 22 Feb 2024 16:26:44.236 * Redis version=7.2.0, bits=64, commit=00000000, modified=0, pid=35, just started
35:C 22 Feb 2024 16:26:44.236 * Configuration loaded
35:M 22 Feb 2024 16:26:44.237 * monotonic clock: POSIX clock_gettime
35:M 22 Feb 2024 16:26:44.238 * Running mode=standalone, port=8081.
35:M 22 Feb 2024 16:26:44.241 * <search> Redis version found by RedisSearch : 7.2.0 - oss
35:M 22 Feb 2024 16:26:44.241 * <search> RediSearch version 2.8.11 (Git=HEAD-652096f2)
35:M 22 Feb 2024 16:26:44.241 * <search> Low level api version 1 initialized successfully
35:M 22 Feb 2024 16:26:44.248 * <search> concurrent writes: OFF, gc: ON, prefix min length: 2, prefix max expansions: 200, query timeout (ms): 600, timeout policy: return, cursor read size: 1000, cursor max idle (ms): 300000, max doctable size: 1000000, max number of search results:  1000000, search pool size: 20, index pool size: 8,
35:M 22 Feb 2024 16:26:44.249 - <search> Creating background thread-0
35:M 22 Feb 2024 16:26:44.249 - <search> Thread pool of size 1 created successfully
35:M 22 Feb 2024 16:26:44.249 - <search> Creating background thread-0
35:M 22 Feb 2024 16:26:44.249 - <search> Thread pool of size 1 created successfully
35:M 22 Feb 2024 16:26:44.249 * <search> Initialized thread pools!
35:M 22 Feb 2024 16:26:44.249 - <search> Creating background thread-0
35:M 22 Feb 2024 16:26:44.249 - <search> Creating background thread-1
35:M 22 Feb 2024 16:26:44.249 - <search> Creating background thread-2
35:M 22 Feb 2024 16:26:44.249 - <search> Creating background thread-3
35:M 22 Feb 2024 16:26:44.249 - <search> Creating background thread-4
35:M 22 Feb 2024 16:26:44.249 - <search> Creating background thread-5
35:M 22 Feb 2024 16:26:44.250 - <search> Creating background thread-6
35:M 22 Feb 2024 16:26:44.250 - <search> Thread pool of size 7 created successfully
35:M 22 Feb 2024 16:26:44.250 * <search> Created workers threadpool of size 7
35:M 22 Feb 2024 16:26:44.250 - <search> threadpool contains 1 privileged threads that always prefer running queries when possible
35:M 22 Feb 2024 16:26:44.250 * <search> Enabled role change notification
...
35:M 22 Feb 2024 17:07:40.814 . <search> Thread-4 is running iteration
35:M 22 Feb 2024 17:07:41.184 . <search> Thread-1 is running iteration
35:M 22 Feb 2024 17:07:41.509 . <search> Thread-3 is running iteration
35:M 22 Feb 2024 17:07:42.509 - Accepted 10.129.117.5:24398
35:M 22 Feb 2024 17:07:42.626 . <search> Thread-5 is running iteration
35:M 22 Feb 2024 17:07:43.107 . <search> Thread-6 is running iteration
35:M 22 Feb 2024 17:07:43.128 . <search> Thread-0 is running iteration
35:M 22 Feb 2024 17:07:43.128 - DB 0: 1537791 keys (0 volatile) in 2097152 slots HT.
35:M 22 Feb 2024 17:07:43.128 . 12 clients connected (0 replicas), 6858411736 bytes in use
35:M 22 Feb 2024 17:07:43.792 . <search> Thread-0 is running iteration
35:M 22 Feb 2024 17:07:43.792 . <search> All threads are idle
35:M 22 Feb 2024 17:07:43.931 . <search> Thread-2 is running iteration
35:M 22 Feb 2024 17:07:44.510 . <search> Thread-4 is running iteration
35:M 22 Feb 2024 17:07:45.102 . <search> Thread-1 is running iteration
35:M 22 Feb 2024 17:07:45.317 . <search> Thread-3 is running iteration
35:M 22 Feb 2024 17:07:48.166 - DB 0: 1537791 keys (0 volatile) in 2097152 slots HT.
35:M 22 Feb 2024 17:07:48.166 . 12 clients connected (0 replicas), 6857249000 bytes in use
35:M 22 Feb 2024 17:07:53.202 - DB 0: 1537791 keys (0 volatile) in 2097152 slots HT.
35:M 22 Feb 2024 17:07:53.202 . 12 clients connected (0 replicas), 6857208056 bytes in use
35:M 22 Feb 2024 17:07:57.506 - Accepted 10.129.117.5:29940
35:M 22 Feb 2024 17:07:58.214 . <search> Thread-0 is running iteration
35:M 22 Feb 2024 17:07:58.214 . <search> All threads are idle
35:M 22 Feb 2024 17:07:58.240 - DB 0: 1537791 keys (0 volatile) in 2097152 slots HT.
35:M 22 Feb 2024 17:07:58.240 . 12 clients connected (0 replicas), 6857208056 bytes in use
35:M 22 Feb 2024 17:08:03.277 - DB 0: 1537791 keys (0 volatile) in 2097152 slots HT.
35:M 22 Feb 2024 17:08:03.277 . 12 clients connected (0 replicas), 6857208056 bytes in use
35:M 22 Feb 2024 17:08:06.452 . <search> Thread-0 is running iteration
35:M 22 Feb 2024 17:08:06.452 . <search> All threads are idle
35:M 22 Feb 2024 17:08:08.318 - DB 0: 1537791 keys (0 volatile) in 2097152 slots HT.
35:M 22 Feb 2024 17:08:08.318 . 12 clients connected (0 replicas), 6857208072 bytes in use
35:M 22 Feb 2024 17:08:12.506 - Accepted 10.129.117.5:35458
35:M 22 Feb 2024 17:08:13.356 - DB 0: 1537791 keys (0 volatile) in 2097152 slots HT.
35:M 22 Feb 2024 17:08:13.356 . 12 clients connected (0 replicas), 6857208072 bytes in use
35:M 22 Feb 2024 17:08:16.550 . <search> Thread-0 is running iteration
35:M 22 Feb 2024 17:08:16.550 . <search> All threads are idle
35:M 22 Feb 2024 17:08:18.397 - DB 0: 1537791 keys (0 volatile) in 2097152 slots HT.
35:M 22 Feb 2024 17:08:18.398 . 12 clients connected (0 replicas), 6857208056 bytes in use
35:M 22 Feb 2024 17:08:20.793 . <search> Thread-0 is running iteration
35:M 22 Feb 2024 17:08:20.793 . <search> All threads are idle
35:M 22 Feb 2024 17:08:23.437 - DB 0: 1537791 keys (0 volatile) in 2097152 slots HT.
35:M 22 Feb 2024 17:08:23.438 . 12 clients connected (0 replicas), 6857208056 bytes in use
35:M 22 Feb 2024 17:08:27.508 - Accepted 10.129.117.5:40966

It appears that threads 1 to 6 are occupied at the moment, yet there are currently no inquiries. Could anyone assist me with this issue?

To Reproduce
Steps to reproduce the behavior:

  1. ...
  2. ....

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

  • OS: ubuntu 20.04 OS with Redisearch container.
  • CPU model:
  • Version/branch: 2.8.11

Additional context
Add any other context about the problem here.
@m3ngyang m3ngyang added the bug label Feb 22, 2024
@alonre24
Copy link
Collaborator

Hey @m3ngyang,
First, REDISEARCH_MT_BUILD=1 is an internal build flag, and we do not recommend using it when building RediSearch from source. The multi-threading feature is currently supported only in redis enterprise.
Can you try running again without MT flags and see if CPU usage remains high while there are no requests to process? Also, I suggest running redis-cli monitor to validate that there are no requests.

@alonre24 alonre24 removed the bug label Feb 27, 2024
@m3ngyang
Copy link
Author

Ok, thx @alonre24
let me try that.

@m3ngyang m3ngyang changed the title [BUG] CPU utilization remains consistently high after complex hybrid searches utilizing MT mode. [Help] CPU utilization remains consistently high after complex hybrid searches utilizing MT mode. Feb 28, 2024
@github-actions GitHub Actions
Copy link

This issue is stale because it has been open for 60 days with no activity.

@github-actions github-actions bot added the stale label Apr 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nafraf nafraf

Labels
stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@m3ngyang @nafraf @alonre24