Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Version: 0.24.1: Out of memory #1686

Open
developerEinpix opened this issue Apr 23, 2024 · 3 comments
Open

Version: 0.24.1: Out of memory #1686

developerEinpix opened this issue Apr 23, 2024 · 3 comments

Comments

@developerEinpix
Copy link

developerEinpix commented Apr 23, 2024 •
edited

I'm new to the typesense. We are using the Linux server and we have installed the typesense over there (Self hosted) with the HA with 3 nodes.

Initially, we do have 8GB RAM and 4 vcpus. All the memories was occupied by the typesense and we got the out of memory error from the server.

Can you please suggest/help me to come out this situation?

We have increased the memory to 16GB still we get the same problem. In 2-3 hours it occupied all the server memory.

{
    "system_cpu1_active_percentage": "0.00",
    "system_cpu2_active_percentage": "20.00",
    "system_cpu3_active_percentage": "9.09",
    "system_cpu4_active_percentage": "0.00",
    "system_cpu_active_percentage": "5.00",
    "system_disk_total_bytes": "71926022144",
    "system_disk_used_bytes": "26616872960",
    "system_memory_total_bytes": "16514195456",
    "system_memory_used_bytes": "11204046848",
    "system_network_received_bytes": "0",
    "system_network_sent_bytes": "0",
    "typesense_memory_active_bytes": "1984081920",
    "typesense_memory_allocated_bytes": "1813526688",
    "typesense_memory_fragmentation_ratio": "0.09",
    "typesense_memory_mapped_bytes": "2060288000",
    "typesense_memory_metadata_bytes": "46520176",
    "typesense_memory_resident_bytes": "1984081920",
    "typesense_memory_retained_bytes": "469401600"
}

This is the result of the matric json from the leader.

Below is the sample logs from the followers:

image

It would be great if you could help me to solve this issue.

Thanks in advance.

Let me know if any other details you need from my side.

Schema which we are using:

{
  "name": "documents",
  "fields": [
    {
      "name": "sortable_id",
      "type": "int32",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "title",
      "type": "string",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": ""
    },
    {
      "name": "reporter_id",
      "type": "int32",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "reporter_name",
      "type": "string",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "category_id",
      "type": "string",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": ""
    },
    {
      "name": "tags",
      "type": "string[]",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": ""
    },
    {
      "name": "severity",
      "type": "string",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": ""
    },
    {
      "name": "is_reopened",
      "type": "bool",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "is_waiting_for_approval",
      "type": "bool",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "reporter_user_groups",
      "type": "string[]",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": ""
    },
    {
      "name": "assignees",
      "type": "string[]",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": ""
    },
    {
      "name": "assignees_name",
      "type": "string",
      "facet": false,
      "optional": true,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "assignees_user_groups",
      "type": "string[]",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": ""
    },
    {
      "name": "user_groups",
      "type": "string[]",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": ""
    },
    {
      "name": "responsible_id",
      "type": "int32",
      "facet": true,
      "optional": true,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "responsible_user_groups",
      "type": "string[]",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": ""
    },
    {
      "name": "watchers",
      "type": "string[]",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": ""
    },
    {
      "name": "venue_id",
      "type": "int32",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "venue_name",
      "type": "string",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "organization_id",
      "type": "int32",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "search_keywords",
      "type": "string",
      "facet": false,
      "optional": true,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": ""
    },
    {
      "name": "client_id",
      "type": "string",
      "facet": true,
      "optional": true,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": ""
    },
    {
      "name": "has_category",
      "type": "bool",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "has_assignee",
      "type": "bool",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "has_tags",
      "type": "bool",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "is_shared",
      "type": "bool",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "is_scheduled",
      "type": "bool",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "is_periodic",
      "type": "bool",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "activity_groups",
      "type": "string[]",
      "facet": true,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": ""
    },
    {
      "name": "work_log_users",
      "type": "string[]",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": ""
    },
    {
      "name": "has_work_log",
      "type": "bool",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "owner_organization_id",
      "type": "int32",
      "facet": true,
      "optional": true,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "assignee_organization_id",
      "type": "int32",
      "facet": true,
      "optional": true,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "reporter_organization_id",
      "type": "int32",
      "facet": true,
      "optional": true,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "status_order",
      "type": "string",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "done_date",
      "type": "int64",
      "facet": false,
      "optional": true,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "deadline",
      "type": "int64",
      "facet": false,
      "optional": true,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "resolution_deadline",
      "type": "int64",
      "facet": false,
      "optional": true,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "reaction_deadline",
      "type": "int64",
      "facet": false,
      "optional": true,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "scheduled_end_date_time",
      "type": "int64",
      "facet": false,
      "optional": true,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "scheduled_start_date_time",
      "type": "int64",
      "facet": false,
      "optional": true,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "deadline_order_by",
      "type": "int32",
      "facet": false,
      "optional": true,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "created",
      "type": "int64",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "updated",
      "type": "int64",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "is_deleted",
      "type": "bool",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": true,
      "infix": false,
      "locale": ""
    },
    {
      "name": "status",
      "type": "string",
      "facet": false,
      "optional": false,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": ""
    },
    {
      "name": "location_map_id",
      "type": "string",
      "facet": true,
      "optional": true,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": ""
    },
    {
      "name": "equipment_id",
      "type": "string",
      "facet": true,
      "optional": true,
      "index": true,
      "sort": false,
      "infix": false,
      "locale": ""
    }
  ],
  "default_sorting_field": "sortable_id",
  "enable_nested_fields": false,
  "symbols_to_index": [],
  "token_separators": []
}
@developerEinpix developerEinpix changed the title Version: 0.24.1: Memory exceed issue Version: 0.24.1: Out of memory Apr 23, 2024
@jasonbosco
Copy link
Member

Memory usage is completely dependent on the amount of data you've indexed in Typesense.

Here's a detailed article on how to estimate RAM usage for your dataset: https://typesense.org/docs/guide/system-requirements.html#choosing-ram

Also, v0.24.1 is pretty old at this point. I would recommend using v26

@developerEinpix
Copy link
Author

developerEinpix commented Apr 24, 2024 •
edited

@jasonbosco Thanks for the quick response.

From the above metrics.json I can see that it is taking 1984081920 bytes -> 1.9GB, right?

I'm not sure if this is relevant data of the memory usages but I've doubt if it has the huge amount only for 0.7M records.

image

And memory increased time by time. Let's say If i restart all the nodes then initially it took 4-5 GB of RAM and in next 4-5 hour it increased to 16GB.

Any help would save my day as we have the issue in production. When we have the same things on the previous server without HA it was working without any error with the same amount of the data.

One more thing I don't understand that why system_network_sent_bytes and system_network_received_bytes always 0 for first 2 nodes. Only 3rd node have the values for these 2 keys

@jasonbosco
Copy link
Member

From the above metrics.json I can see that it is taking 1984081920 bytes -> 1.9GB, right?

That's correct. So Typesense is actually only using about 2GB of RAM.

Something else running on the same server seems to be the one that's taking up the rest of the memory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jasonbosco @developerEinpix