Meilisearch CPU utilization crazy high - Keeps becoming unresponsive.

[lrojas94]

First off - Hello again! I asked a question before and was really happy with you guys' support <3

So, here's another one for you.... We're on Meilisearch 1.6.
Basically, the problem is our instance keeps becoming unavailable quite regularly, specially after one of these crazy CPU spikes (We don't have a CPU limit for it). The pod it's in is a Digital Ocean 8 vCPU/32GB (7,500 IOPS/ 300 MB/s Throughput). The white-spots on the table after the spikes are times it's either reboot itself, gone down or is completely unavailable.

From what we can tell, it seems related to really high search usage during our peak hours, although we don't have a particular metric that can tell this sadly. We have a big data set (about 70GB of data) divided among multiple indexes, but our searchable attributes (I believe) are setup so it's fairly performant - quite minimal.

We have tried disabling our indexing jobs (Which did nothing at all).
Another weird thing is that - when indexing - Meili won't use more than 5GB RAM. Stop indexing and it jumps to 24gb (our max atm). Seems weird, thought it should be noted, although I can't tell how these relate since the CPU spikes / issues happen regardless of whether or not the indexing queue is empty.

On the above point - here's how the RAM chart looks like (No new queued jobs since the 20:00 mark or so)

Any help would be appreciated!

[Kerollmops]

Hey @lrojas94 👋

Thank you very much for this report. I want to ask a bunch of questions beforehand:

• What are the settings of your indexes?
• How many documents are requested by the queries? Can you show a typical search query sent during the spikes?
• Why are you splitting your dataset into indexes in the same instance?

Thank you very much for helping me debug this.

[lrojas94]

Thanks for the reply! So, first off, let me get to your questions 1 at a time:

Why are you splitting your dataset into indexes in the same instance?
It's different data-sets per database entities basically. So we have an index for each of them. The reason we have them on a single instance is to avoid having X amount of instances/pods of course. It was my understanding this was OK overall. The data on each index is completely different to be clear.

How many documents are requested by the queries? Can you show a typical search query sent during the spikes?
We're requesting 51 items per request. Search query can be anything, so this is an example:

{
   "queries":[
      {
         "q":"dragon ball z",
         "indexUid":"models_v5",
         "facets":[
            "category.name",
            "checkpointType",
            "fileFormats",
            "lastVersionAtUnix",
            "tags.name",
            "type",
            "user.username",
            "version.baseModel"
         ],
         "attributesToHighlight":[
            
         ],
         "highlightPreTag":"__ais-highlight__",
         "highlightPostTag":"__/ais-highlight__",
         "limit":51,
         "offset":0,
         "filter":[
            [
               "\"type\"=\"LORA\""
            ]
         ]
      },
      {
         "q":"dragon ball z",
         "indexUid":"models_v5",
         "facets":[
            "type"
         ],
         "attributesToHighlight":[
            
         ],
         "highlightPreTag":"__ais-highlight__",
         "highlightPostTag":"__/ais-highlight__",
         "limit":1,
         "offset":0
      }
   ]
}

What are the settings of your indexes?
Settings vary a bit in terms of searchable attributes, filterable attributes, sortable attributes and whatnot, but this being one of the most "complete" ones should hopefully serve as a good example.

{
    "displayedAttributes": [
        "*"
    ],
    "searchableAttributes": [
        "name",
        "user.username",
        "hashes",
        "triggerWords"
    ],
    "filterableAttributes": [
        "canGenerate",
        "category.name",
        "checkpointType",
        "fileFormats",
        "hashes",
        "lastVersionAtUnix",
        "nsfw",
        "status",
        "tags.name",
        "type",
        "user.username",
        "version.baseModel",
        "versions.hashes"
    ],
    "sortableAttributes": [
        "createdAt",
        "metrics.collectedCount",
        "metrics.commentCount",
        "metrics.downloadCount",
        "metrics.favoriteCount",
        "metrics.rating",
        "metrics.ratingCount",
        "metrics.tippedAmountCount",
        "metrics.weightedRating"
    ],
    "rankingRules": [
        "sort",
        "attribute",
        "metrics.weightedRating:desc",
        "words",
        "proximity",
        "exactness"
    ],
    "stopWords": [],
    "nonSeparatorTokens": [],
    "separatorTokens": [],
    "dictionary": [],
    "synonyms": {},
    "distinctAttribute": null,
    "proximityPrecision": "byWord",
    "typoTolerance": {
        "enabled": true,
        "minWordSizeForTypos": {
            "oneTypo": 12,
            "twoTypos": 16
        },
        "disableOnWords": [],
        "disableOnAttributes": []
    },
    "faceting": {
        "maxValuesPerFacet": 100,
        "sortFacetValuesBy": {
            "*": "alpha"
        }
    },
    "pagination": {
        "maxTotalHits": 1000
    }
}

[lrojas94]

@Kerollmops sorry about the ping - but since it's been a few weeks, wondering if there's anything we can improve on here?

[Kerollmops]

Hey @lrojas94 👋

Sorry for the delay. I would like to know a little bit more about your use case before jumping to conclusions. What's the average size of your documents and most of all, what are the search spikes you experience?

[...] it seems related to really high search usage during our peak hours, although we don't have a particular metric that can tell this sadly.

That's sad to hear because we could determine the memory spike you got from the search spike you experienced.

We are thinking about different fixes to this issue:

• Reduce the number of current searches at a point to precisely the number of threads.
• Stream the search responses.

The former makes sense because search requests are CPU intensive, so only being able to process a single request by CPU is the most efficient approach. Queuing the newly arriving search requests must also be done using a FIFO approach. Combine this with a cutoff that forces the engine to degrade the quality of a search result when it takes too long to compute.

However, the latter is much more complex to implement and will not reduce the memory used during search but only the one used to generate the JSON response body.

Would the former solution fit your needs?