Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tag table storage optimization? #131

Open
ihard opened this issue May 15, 2023 · 1 comment
Open

Tag table storage optimization? #131

ihard opened this issue May 15, 2023 · 1 comment

Comments

@ihard
Copy link

ihard commented May 15, 2023

On my installation, I observe a large increase in the tags table, as the number of metrics with tags has greatly increased.
The space occupied is somewhat confusing - tables with data take up less and less space over time, since data is aggregated. Tagged tables don't shrink over time and continue to take up a lot of space.
Is there some kind of optimization option here to reduce the space occupied by a table with tags?
Simply deleting old partitions affects the graphs because the metrics have a different set of labels (pod name).
My current tables with partitions:

197001 points_reverse 2.23 KiB
197204 points_reverse 862.00 B
197410 points_reverse 1.33 KiB
202008 points_reverse 5.28 GiB
202009 points_reverse 4.34 GiB
202010 points_reverse 4.86 GiB
202011 points_reverse 4.08 GiB
202012 points_reverse 5.03 GiB
202101 points_reverse 6.28 GiB
202102 points_reverse 6.00 GiB
202103 points_reverse 7.38 GiB
202104 points_reverse 7.48 GiB
202105 points_reverse 8.11 GiB
202106 points_reverse 8.81 GiB
202107 points_reverse 9.88 GiB
202108 points_reverse 10.23 GiB
202109 points_reverse 11.52 GiB
202110 points_reverse 12.60 GiB
202111 points_reverse 11.32 GiB
202112 points_reverse 16.06 GiB
202201 points_reverse 16.02 GiB
202202 points_reverse 9.67 GiB
202203 points_reverse 10.71 GiB
202204 points_reverse 10.52 GiB
202205 points_reverse 10.41 GiB
202206 points_reverse 10.28 GiB
202207 points_reverse 12.89 GiB
202208 points_reverse 12.36 GiB
202209 points_reverse 15.02 GiB
202210 points_reverse 30.05 GiB
202211 points_reverse 14.27 GiB
202212 points_reverse 17.36 GiB
202301 points_reverse 14.47 GiB
202302 points_reverse 47.93 GiB
202303 points_reverse 56.18 GiB
202304 points_reverse 327.05 GiB
202305 points_reverse 308.23 GiB
202309 points_reverse 74.55 MiB
203909 points_reverse 559.00 B
205405 points_reverse 651.00 B
206506 points_reverse 633.00 B
209712 points_reverse 544.00 B
210307 points_reverse 826.00 B
197001 tagged 2.47 KiB
202109 tagged 27.08 GiB
202110 tagged 29.56 GiB
202111 tagged 26.85 GiB
202112 tagged 36.39 GiB
202201 tagged 37.79 GiB
202202 tagged 24.79 GiB
202203 tagged 30.32 GiB
202204 tagged 31.29 GiB
202205 tagged 38.24 GiB
202206 tagged 64.44 GiB
202207 tagged 82.27 GiB
202208 tagged 64.82 GiB
202209 tagged 87.59 GiB
202210 tagged 120.57 GiB
202211 tagged 128.04 GiB
202212 tagged 175.31 GiB
202301 tagged 139.91 GiB
202302 tagged 164.22 GiB
202303 tagged 197.57 GiB
202304 tagged 217.57 GiB
202305 tagged 125.93 GiB
202309 tagged 10.29 KiB
197001 index 1.89 KiB
197002 index 52.99 GiB
197204 index 911.00 B
197410 index 1.30 KiB
202008 index 2.28 GiB
202009 index 1.58 GiB
202010 index 1.85 GiB
202011 index 1.13 GiB
202012 index 2.00 GiB
202101 index 2.43 GiB
202102 index 2.49 GiB
202103 index 3.07 GiB
202104 index 2.99 GiB
202105 index 3.08 GiB
202106 index 3.12 GiB
202107 index 3.49 GiB
202108 index 3.17 GiB
202109 index 3.65 GiB
202110 index 3.14 GiB
202111 index 2.24 GiB
202112 index 3.43 GiB
202201 index 2.99 GiB
202202 index 2.91 GiB
202203 index 2.90 GiB
202204 index 3.12 GiB
202205 index 2.73 GiB
202206 index 2.67 GiB
202207 index 3.39 GiB
202208 index 3.03 GiB
202209 index 2.88 GiB
202210 index 2.89 GiB
202211 index 2.49 GiB
202212 index 2.89 GiB
202301 index 2.55 GiB
202302 index 3.32 GiB
202303 index 3.83 GiB
202304 index 5.22 GiB
202305 index 4.09 GiB
202309 index 14.58 KiB
203909 index 532.00 B
205405 index 641.00 B
206506 index 674.00 B
209712 index 517.00 B
210307 index 911.00 B
@msaf1980
Copy link
Collaborator

I have some plans about this. But it's not too simple. Tagged table has structure like full-text reverse index (but more simple), it's has huge size. If you can restrict all queries for use name tag - you can write only this tag wih tune ignored-tagged-metrics parameter

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants