MOD-5977 MOD-5866: Fix inverted index memory counting #4218

nafraf · 2023-12-17T05:13:27Z

Describe the changes in the pull request
Improve the report of memory used by inverted index

Which issues this PR fixes

MOD-5977
MOD-5866

(done through MOD-6515)

Main objects this PR modified

Buffer functions return the number of bytes that were allocated or released.
The size of Inverted index structure is counted when the structure is allocated and released.
fork_gc counts the bytes allocated if a new block is added.
The memory occupied by the buffers in the inverted size index is reported based on its capacity, because it is the amount of memory allocated.

Mark if applicable

This PR introduces API changes
This PR introduces serialization changes

codecov · 2023-12-17T06:19:24Z

Codecov Report

Attention: Patch coverage is 86.27451% with 14 lines in your changes are missing coverage. Please review.

Project coverage is 86.04%. Comparing base (78409fd) to head (be4de5e).

Files	Patch %	Lines
src/qint.c	69.23%	4 Missing ⚠️
src/numeric_index.c	84.21%	3 Missing ⚠️
src/redis_index.c	66.66%	3 Missing ⚠️
src/fork_gc.c	88.88%	2 Missing ⚠️
src/inverted_index.c	87.50%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4218      +/-   ##
==========================================
+ Coverage   85.98%   86.04%   +0.06%     
==========================================
  Files         190      190              
  Lines       34517    34531      +14     
==========================================
+ Hits        29678    29713      +35     
+ Misses       4839     4818      -21

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

filipecosta90 · 2024-01-22T14:13:38Z

Automated performance analysis summary

This comment was automatically generated given there is performance data available.

In summary:

Detected a total of 31 stable tests between versions.
Detected a total of 8 highly unstable benchmarks.
Detected a total of 2 improvements above the improvement water line.
Detected a total of 2 regressions bellow the regression water line 5.0.

You can check a comparison in detail via the grafana link

Comparison between master and nafraf_fix-inv-idx-mem-report.

Time Period from 30 days ago. (environment used: oss-standalone)

Test Case	Baseline master (median obs. +- std.dev)	Comparison nafraf_fix-inv-idx-mem-report (median obs. +- std.dev)	% change (higher-better)	Note
ftsb-10K-enwiki_abstract-hashes-fulltext-sortby	68 +- 0.8% (7 datapoints)	69	1.1%	No Change
ftsb-10K-enwiki_abstract-hashes-term-prefix	7195 +- 2.5% (7 datapoints)	7363	2.3%	No Change
ftsb-10K-enwiki_abstract-hashes-term-suffix	2028 +- 3.9% (7 datapoints)	1952	-3.8%	potential REGRESSION
ftsb-10K-enwiki_abstract-hashes-term-suffix-withsuffixtrie	70092 +- 5.2% (7 datapoints)	64813	-7.5%	waterline=5.2%. REGRESSION
ftsb-10K-enwiki_abstract-hashes-term-wildcard	11959 +- 2.6% (7 datapoints)	11649	-2.6%	No Change
ftsb-10K-enwiki_pages-hashes-fulltext-mixed_simple-1word-query_write_1_to_read_20.yml	1261 +- 2.5% (7 datapoints)	1228	-2.6%	No Change
ftsb-10K-enwiki_pages-hashes-load	49096 +- 4.2% (7 datapoints)	47232	-3.8%	potential REGRESSION
ftsb-1K-enwiki_abstract-hashes-term-contains	1718 +- 3.3% (7 datapoints)	1663	-3.2%	potential REGRESSION
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query	346 +- 13.3% UNSTABLE (7 datapoints)	384	11.0%	UNSTABLE (very high variance)
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query-non-sortable	41 +- 5.4% (7 datapoints)	37	-9.0%	waterline=5.4%. REGRESSION
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query@100_ops_sec	100 +- 0.0% (7 datapoints)	100	-0.0%	No Change
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query	2666 +- 2.7% (7 datapoints)	2695	1.1%	No Change
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query-non-sortable	1015 +- 11.5% UNSTABLE (7 datapoints)	1028	1.3%	UNSTABLE (very high variance)
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query@100_ops_sec	100 +- 0.0% (7 datapoints)	100	0.0%	No Change
ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query	977 +- 10.3% UNSTABLE (7 datapoints)	1065	9.0%	UNSTABLE (very high variance)
ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query@100_ops_sec	100 +- 0.0% (7 datapoints)	100	0.0%	No Change
ftsb-1M-enwiki_abstract-hashes-load	22411 +- 2.7% (7 datapoints)	21428	-4.4%	potential REGRESSION
ftsb-1M-nyc_taxis-ftadd-load	20884 +- 2.8% (7 datapoints)	20746	-0.7%	No Change
ftsb-1M-nyc_taxis-hashes-load	21892 +- 2.2% (7 datapoints)	21548	-1.6%	No Change
search-aggregate-post-filter-simple.yml	93421 +- 4.7% (7 datapoints)	99338	6.3%	IMPROVEMENT
search-filtering-tag-numeric	184 +- 6.4% (7 datapoints)	176	-4.1%	waterline=6.4%. potential REGRESSION
search-filtering-tag-numeric-filter-pipeline	19133 +- 2.1% (7 datapoints)	18437	-3.6%	potential REGRESSION
search-ftsb-10K-enwiki_abstract-hashes-term-withoutsuffix-trie	40375 +- 3.0% (7 datapoints)	38543	-4.5%	potential REGRESSION
search-ftsb-10K-enwiki_abstract-hashes-term-withsuffix-trie	38511 +- 3.0% (7 datapoints)	39850	3.5%	potential IMPROVEMENT
search-ftsb-1700K-docs-union-iterators-q3	6.2 +- 2.6% (7 datapoints)	6.20	0.2%	No Change
search-ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query-non-sortable@50_ops_sec	50 +- 10.7% UNSTABLE (7 datapoints)	50	0.2%	UNSTABLE (very high variance)
search-ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query-non-sortable@100_ops_sec	100 +- 0.0% (7 datapoints)	100	0.0%	No Change
search-ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query-non-sortable	151 +- 4.2% (7 datapoints)	145	-3.9%	potential REGRESSION
search-ftsb-370K-docs-union-iterators-q4	6.7 +- 0.8% (7 datapoints)	6.70	-0.6%	No Change
search-ftsb-5200K-docs-union-iterators-q1	0.62 +- 0.9% (3 datapoints)	0.62	0.0%
search-ftsb-5500K-docs-union-iterators-q2	0.93	0.92	-1.1%	No Change
search-geo	186 +- 2.9% (7 datapoints)	188	1.2%	No Change
search-high-cardinality-negation-term-baseline	20 +- 4.5% (4 datapoints)	20	0.6%	No Change
search-high-cardinality-negation-term-comparison_union_all_other_terms	N/A	N/A	0.0%	waterline=7.3%.
search-numeric	3563 +- 28.8% UNSTABLE (7 datapoints)	3393	-4.8%	UNSTABLE (very high variance)
search-numeric-optimize	9058 +- 1.7% (7 datapoints)	8925	-1.5%	No Change
search-numeric-sortby	3012 +- 33.2% UNSTABLE (7 datapoints)	3352	11.3%	UNSTABLE (very high variance)
search-numeric-sortby-desc	3037 +- 11.6% UNSTABLE (7 datapoints)	3347	10.2%	UNSTABLE (very high variance)
search-numeric-sortby-desc-optimize	33 +- 35.3% UNSTABLE (7 datapoints)	56	68.0%	UNSTABLE (very high variance)
search-numeric-sortby-optimize	19 +- 1.9% (4 datapoints)	20	3.2%	potential IMPROVEMENT
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-fulltext-filter	474 +- 3.7% (7 datapoints)	482	1.6%	No Change
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-numeric-filter	105 +- 8.1% (7 datapoints)	105	0.3%	waterline=8.1%. No Change
vecsim-arxiv-titles-384-angular-filters-m16-ef-128-tag-filter	54957 +- 4.5% (7 datapoints)	59901	9.0%	IMPROVEMENT

oshadmi

Few comments up til now

src/buffer.h

src/buffer.c

src/tag_index.c

src/redis_index.c

src/fork_gc.h

tests/cpptests/index_utils.cpp

src/numeric_index.c

This reverts commit c97b2c2.

oshadmi

💪🏼 Very nice!!!

github-actions · 2024-05-16T15:17:10Z

Backport failed for 2.8, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 2.8
git worktree add -d .worktree/backport-4218-to-2.8 origin/2.8
cd .worktree/backport-4218-to-2.8
git switch --create backport-4218-to-2.8
git cherry-pick -x 67567aa66e993e89271508470a15cafafd52706a 226856fb412f877da9cb09b57e22d7853a5ae111 2a691a67a6af20d8d646b59bb5cd4c37e61e5af9 b2f8f2f08eafe4cfe659c20f826e1a297f21f4d0 e17caf4422a13c0cebd68bd2da1ed99689214adb 4fad0280c0e5c0c2cec773c33c7d018cc1c95fda 9a49a337c2ae10802fbddac045b9c9048856fcff 14c478665fb1e3d88d74cd3661402fbd35318d54 9feef665a196b8ecbf965fc61f9d4508c03d094c a55eaba5f4f47d7a85ee2cd9c6d2efa1b4103848 a254ed0baeb2d2299808b87beb6b30ab54ae519a b98f1c79934ffaf5a4630604958b972fdea79ffa 2c3156c0d7f0d6f110bd9d96ad7b8d7f559f29d0 425a28ba4cc49365794ab4a5651c3e7038153653 0c2a6454026242f7bda6fdaf61b6adeabe30adc1 892b00d12001c4dbb8111ea410a13bac0a167aeb 247a553f29c32f6495db77c24e56c45a2dd16887 b259a08d079950fd7584b43e27d9870e78758cf5 a2fd3e38439b306d92506cc3c88a4e5948dce10b fefe7ea29c16e5be49bc259031be808347c33263 d995029a866f26382dfc8a7e5c2b88d4e19f2a23 7c17bf3a3ca7defc512622fb3e3a70ffa06aa248 ee9428ff55786bd6e603dcbf21cb91d6cf0ed2e4 df705d25c2e895d88ca76f5217b4bbf9588e33b6 69fd9d5b853d416eeb3bc6d456ecaaf3b8421126 c172a5bab7ebc1fd8ff88502b07fb97652293aff 58a72f86418882dfe5cb2e10f5308896258bd4d4 06112cc5367d024e0ce84b1809d21c7f880c5143 4425fc11cf3b151d0efda960ea9fbc2f69624bf5 f17b5cababfd4afb163e6f8479d76fef3b3d627f f7b32386040f17763192011cb9bcaf72a6eb1df8 36a6545fb20e0338264e490bb1252fdee21e5dbe 7706bf24f89f591a3d6e1b506c93630c6933026a 737038e516a44bb304c61ab3e82da4111c0b5081 e9ffebe1ea8d55429509f0d371abdf9ea4b1d851 500ff1e9f30372ecf4caa1b0b1aa8d56d29668e4 32515dd26c1c730e8c902bd5ede17cf3b2383942 c97b2c24a6ec509597b21a69f33c916bc281b6a3 4e598d6011a785e765b15b055d75c8c37d0d87aa 3947e88add090bec1dad1a8e54d624eae86c82da 13fa9f07d16aa6d842babdfc4f9c15ceb3336385 d0649c12051403d694c5da73e690e810af9d2a12 b9bb98530c02fab3a8cf1009da970a6ff91bc1ce 62785b9bdbc443f4dc37c3ce08c8cea297e3e160 1f51c932c7e19409e2f389f574bd59aa28d8ec2f c1f2804674e6126525ef96a66a6587b2660fbc67 a1f43d46f634acd3e26514ced025529be63442d0 4253d18413b45ac41e4b8cc29102720f9bd6912f 697ba763df8bcaa9dcf4098f6628a267c57aac91 c8f597604f0c746d2251e4801fd6bbf560964e58

github-actions · 2024-05-16T15:17:11Z

Backport failed for 2.10, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 2.10
git worktree add -d .worktree/backport-4218-to-2.10 origin/2.10
cd .worktree/backport-4218-to-2.10
git switch --create backport-4218-to-2.10
git cherry-pick -x 67567aa66e993e89271508470a15cafafd52706a 226856fb412f877da9cb09b57e22d7853a5ae111 2a691a67a6af20d8d646b59bb5cd4c37e61e5af9 b2f8f2f08eafe4cfe659c20f826e1a297f21f4d0 e17caf4422a13c0cebd68bd2da1ed99689214adb 4fad0280c0e5c0c2cec773c33c7d018cc1c95fda 9a49a337c2ae10802fbddac045b9c9048856fcff 14c478665fb1e3d88d74cd3661402fbd35318d54 9feef665a196b8ecbf965fc61f9d4508c03d094c a55eaba5f4f47d7a85ee2cd9c6d2efa1b4103848 a254ed0baeb2d2299808b87beb6b30ab54ae519a b98f1c79934ffaf5a4630604958b972fdea79ffa 2c3156c0d7f0d6f110bd9d96ad7b8d7f559f29d0 425a28ba4cc49365794ab4a5651c3e7038153653 0c2a6454026242f7bda6fdaf61b6adeabe30adc1 892b00d12001c4dbb8111ea410a13bac0a167aeb 247a553f29c32f6495db77c24e56c45a2dd16887 b259a08d079950fd7584b43e27d9870e78758cf5 a2fd3e38439b306d92506cc3c88a4e5948dce10b fefe7ea29c16e5be49bc259031be808347c33263 d995029a866f26382dfc8a7e5c2b88d4e19f2a23 7c17bf3a3ca7defc512622fb3e3a70ffa06aa248 ee9428ff55786bd6e603dcbf21cb91d6cf0ed2e4 df705d25c2e895d88ca76f5217b4bbf9588e33b6 69fd9d5b853d416eeb3bc6d456ecaaf3b8421126 c172a5bab7ebc1fd8ff88502b07fb97652293aff 58a72f86418882dfe5cb2e10f5308896258bd4d4 06112cc5367d024e0ce84b1809d21c7f880c5143 4425fc11cf3b151d0efda960ea9fbc2f69624bf5 f17b5cababfd4afb163e6f8479d76fef3b3d627f f7b32386040f17763192011cb9bcaf72a6eb1df8 36a6545fb20e0338264e490bb1252fdee21e5dbe 7706bf24f89f591a3d6e1b506c93630c6933026a 737038e516a44bb304c61ab3e82da4111c0b5081 e9ffebe1ea8d55429509f0d371abdf9ea4b1d851 500ff1e9f30372ecf4caa1b0b1aa8d56d29668e4 32515dd26c1c730e8c902bd5ede17cf3b2383942 c97b2c24a6ec509597b21a69f33c916bc281b6a3 4e598d6011a785e765b15b055d75c8c37d0d87aa 3947e88add090bec1dad1a8e54d624eae86c82da 13fa9f07d16aa6d842babdfc4f9c15ceb3336385 d0649c12051403d694c5da73e690e810af9d2a12 b9bb98530c02fab3a8cf1009da970a6ff91bc1ce 62785b9bdbc443f4dc37c3ce08c8cea297e3e160 1f51c932c7e19409e2f389f574bd59aa28d8ec2f c1f2804674e6126525ef96a66a6587b2660fbc67 a1f43d46f634acd3e26514ced025529be63442d0 4253d18413b45ac41e4b8cc29102720f9bd6912f 697ba763df8bcaa9dcf4098f6628a267c57aac91 c8f597604f0c746d2251e4801fd6bbf560964e58

--------- Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> (cherry picked from commit 959ed1a)

…4668)

…4669) * MOD-5977 MOD-5866: Fix inverted index memory counting (#4218) --------- Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> (cherry picked from commit 959ed1a) * Add type conversion in tests

nafraf added 2 commits December 17, 2023 00:12

WIP: Fix inverted index memory counting

67567aa

Fix test:testIssue1184

226856f

nafraf added 10 commits December 17, 2023 07:43

Fix buffer.c, buffer.h line ending

2a691a6

Fix qint.c line ending

b2f8f2f

Merge branch 'master' into nafraf_fix-inv-idx-mem-report

101d04d

Fix NumericRangeNode_Free()

e17caf4

Fix InvertedIndex_Free_Internal()

4fad028

Fix InvertedIndex_MemUsage(): Use block cap

9a49a33

Fix applyNumIdx(): invertedIndexSize += nbytesAdded

14c4786

TagIndex: Count bytes of inverted index

9feef66

test_stats: Test memory after drop text and tag

a55eaba

Merge branch 'master' into nafraf_fix-inv-idx-mem-report

53e7810

nafraf changed the title ~~Fix inverted index memory counting~~ MOD-5977: Fix inverted index memory counting Dec 20, 2023

nafraf added 7 commits December 22, 2023 14:40

trigger flow sanitizer

a254ed0

Fix test_cpp_forkgc tests

b98f1c7

Merge branch 'master' into nafraf_fix-inv-idx-mem-report

958ade7

Fix expected inverted_sz_mb for TAG and TEXT

2c3156c

Fix test_number:testOverrides

425a28b

Remove InvertedIndex_Free_Internal()

0c2a645

Remove casting

892b00d

oshadmi marked this pull request as ready for review December 27, 2023 13:16

nafraf and others added 3 commits December 27, 2023 11:32

Merge branch 'master' into nafraf_fix-inv-idx-mem-report

2344b62

Merge branch 'master' into nafraf_fix-inv-idx-mem-report

4262718

Merge branch 'master' into nafraf_fix-inv-idx-mem-report

e9c245d

oshadmi added the action:run-benchmark label Jan 22, 2024

Merge branch 'master' into nafraf_fix-inv-idx-mem-report

776ae2d

oshadmi reviewed Jan 24, 2024

View reviewed changes

nafraf changed the title ~~MOD-5977: Fix inverted index memory counting~~ MOD-5977 MOD-5866: Fix inverted index memory counting Jan 24, 2024

nafraf and others added 9 commits April 16, 2024 09:40

Test sizeof(t_fieldMask) first

b9bb985

use typedef __uint128_t t_fieldMask; for arm64

62785b9

Fix IndexTest, testHugeSpec

1f51c93

Fix arch_int_bits()

c1f2804

debug: print platform.machine()

a1f43d4

Fix arch_int_bits(): arm64

4253d18

Revert "test memsize on mac"

697ba76

This reverts commit c97b2c2.

Disable flow-tem tests

c8f5976

Merge branch 'master' into nafraf_fix-inv-idx-mem-report

dc17b2d

nafraf added action:run-benchmark and removed action:run-benchmark labels Apr 16, 2024

nafraf added 4 commits April 22, 2024 13:15

Merge branch 'master' into nafraf_fix-inv-idx-mem-report

4603f10

Merge branch 'master' into nafraf_fix-inv-idx-mem-report

7dd2884

Merge branch 'master' into nafraf_fix-inv-idx-mem-report

c42a948

Merge branch 'master' into nafraf_fix-inv-idx-mem-report

be4de5e

oshadmi approved these changes May 16, 2024

View reviewed changes

nafraf added this pull request to the merge queue May 16, 2024

Merged via the queue into master with commit 959ed1a May 16, 2024
31 checks passed

nafraf deleted the nafraf_fix-inv-idx-mem-report branch May 16, 2024 15:16

nafraf added a commit that referenced this pull request May 23, 2024

MOD-5977 MOD-5866: Fix inverted index memory counting (#4218)

f2b4564

--------- Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> (cherry picked from commit 959ed1a)

nafraf mentioned this pull request May 23, 2024

[2.10] MOD-5977 MOD-5866: Fix inverted index memory counting (#4218) #4668

Merged

nafraf added a commit that referenced this pull request May 23, 2024

MOD-5977 MOD-5866: Fix inverted index memory counting (#4218)

aae9446

--------- Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> (cherry picked from commit 959ed1a)

nafraf mentioned this pull request May 23, 2024

[2.8] MOD-5977 MOD-5866: Fix inverted index memory counting (#4218) #4669

Merged

github-merge-queue bot pushed a commit that referenced this pull request May 24, 2024

[2.10] MOD-5977 MOD-5866: Fix inverted index memory counting (#4218) (#…

e417057

…4668)

nafraf mentioned this pull request May 27, 2024

fixed unreported mem allocations and deallocations #3731

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MOD-5977 MOD-5866: Fix inverted index memory counting #4218

MOD-5977 MOD-5866: Fix inverted index memory counting #4218

nafraf commented Dec 17, 2023 •

edited by oshadmi

codecov bot commented Dec 17, 2023 •

edited

filipecosta90 commented Jan 22, 2024 •

edited by fcostaoliveira

oshadmi left a comment

oshadmi left a comment

github-actions bot commented May 16, 2024

github-actions bot commented May 16, 2024

MOD-5977 MOD-5866: Fix inverted index memory counting #4218

MOD-5977 MOD-5866: Fix inverted index memory counting #4218

Conversation

nafraf commented Dec 17, 2023 • edited by oshadmi

codecov bot commented Dec 17, 2023 • edited

Codecov Report

filipecosta90 commented Jan 22, 2024 • edited by fcostaoliveira

Automated performance analysis summary

Comparison between master and nafraf_fix-inv-idx-mem-report.

oshadmi left a comment

Choose a reason for hiding this comment

oshadmi left a comment

Choose a reason for hiding this comment

github-actions bot commented May 16, 2024

github-actions bot commented May 16, 2024

nafraf commented Dec 17, 2023 •

edited by oshadmi

codecov bot commented Dec 17, 2023 •

edited

filipecosta90 commented Jan 22, 2024 •

edited by fcostaoliveira