Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce transparent_decompress_chunk flakiness #6550

Draft
wants to merge 32 commits into
base: main
Choose a base branch
from
Draft

Reduce transparent_decompress_chunk flakiness #6550

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
b479e08
Reduce transparent_decompress_chunk flakiness
akuzm Jan 19, 2024
2563983
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Jan 19, 2024
b70ebd9
refs
akuzm Jan 19, 2024
d3228b9
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Jan 24, 2024
260b2bc
add order by to remove flakiness
akuzm Jan 24, 2024
f3b857c
reference transparent_decompress_chunk-* ordered_append-* ordered_app…
akuzm Jan 24, 2024
0501724
reference transparent_decompress_chunk-* ordered_append-* ordered_app…
akuzm Jan 24, 2024
5c2fb57
fixes...
akuzm Jan 24, 2024
01422d4
fixes
akuzm Jan 24, 2024
7dd8697
better selectivity estimates
akuzm Jan 24, 2024
8529781
fixes
akuzm Jan 24, 2024
5e9e450
Account for filters in DecompressChunk row estimates
akuzm Jan 24, 2024
ee4147f
refs
akuzm Jan 24, 2024
b710c3e
stable mapc test
akuzm Jan 24, 2024
87101d2
reference transparent_decompression_ordered_index-* merge_append_part…
akuzm Jan 25, 2024
c744dfd
reference transparent_decompression_ordered_index-* merge_append_part…
akuzm Jan 25, 2024
a8f83e7
reference transparent_decompression_ordered_index-* merge_append_part…
akuzm Jan 25, 2024
b822a4e
reference transparent_decompress_chunk-*
akuzm Jan 25, 2024
713d4e3
reference transparent_decompress_chunk-*
akuzm Jan 25, 2024
a7ddc1f
reference transparent_decompress_chunk-*
akuzm Jan 25, 2024
9826897
Merge remote-tracking branch 'akuzm/filter-size' into HEAD
akuzm Jan 25, 2024
72645e9
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Jan 25, 2024
8dd8196
fix the row count estimates again
akuzm Jan 25, 2024
ce2b7cd
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Jan 25, 2024
6a2190a
remove debug
akuzm Jan 25, 2024
afcbe74
size fixes
akuzm Jan 26, 2024
4be8f38
Merge remote-tracking branch 'akuzm/filter-size' into HEAD
akuzm Jan 26, 2024
b7c7dc4
uncontroversial test refs
akuzm Jan 26, 2024
97d5046
workaround for merge append cost problem
akuzm Jan 29, 2024
eb52026
Merge remote-tracking branch 'origin/main' into HEAD
akuzm Jan 29, 2024
2571bd8
comment
akuzm Jan 29, 2024
c27f73d
more ref changes
akuzm Jan 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
8 changes: 4 additions & 4 deletions tsl/test/shared/expected/decompress_join.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,17 +18,17 @@ INNER JOIN metrics_compressed m ON t.time = m.time AND t.device_id = m.device_id
LIMIT 1;
QUERY PLAN
Limit (actual rows=1 loops=1)
-> Hash Join (actual rows=1 loops=1)
Hash Cond: ((m_1."time" = t."time") AND (m_1.device_id = t.device_id))
-> Nested Loop (actual rows=1 loops=1)
Join Filter: ((t."time" = m_1."time") AND (t.device_id = m_1.device_id))
Rows Removed by Join Filter: 431804
-> Append (actual rows=43181 loops=1)
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk m_1 (actual rows=17990 loops=1)
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=20 loops=1)
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk m_2 (actual rows=25190 loops=1)
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=30 loops=1)
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk m_3 (actual rows=1 loops=1)
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=1 loops=1)
-> Hash (actual rows=10 loops=1)
Buckets: 1024 Batches: 1
-> Materialize (actual rows=10 loops=43181)
-> Seq Scan on compressed_join_temp t (actual rows=10 loops=1)
(13 rows)

Expand Down
144 changes: 94 additions & 50 deletions tsl/test/shared/expected/ordered_append-15.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2208,15 +2208,23 @@ QUERY PLAN
-> Custom Scan (ChunkAppend) on metrics_compressed (actual rows=1 loops=1)
Order: metrics_compressed."time" DESC
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (actual rows=1 loops=1)
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (actual rows=1 loops=1)
Index Cond: (device_id = 1)
-> Sort (actual rows=1 loops=1)
Sort Key: compress_hyper_X_X_chunk._ts_meta_sequence_num
Sort Method: quicksort
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=6 loops=1)
Filter: (device_id = 1)
Rows Removed by Filter: 24
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (never executed)
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (never executed)
Index Cond: (device_id = 1)
-> Sort (never executed)
Sort Key: compress_hyper_X_X_chunk._ts_meta_sequence_num
-> Seq Scan on compress_hyper_X_X_chunk (never executed)
Filter: (device_id = 1)
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (never executed)
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (never executed)
Index Cond: (device_id = 1)
(12 rows)
-> Sort (never executed)
Sort Key: compress_hyper_X_X_chunk._ts_meta_sequence_num
-> Seq Scan on compress_hyper_X_X_chunk (never executed)
Filter: (device_id = 1)
(20 rows)

-- test RECORD in targetlist
:PREFIX
Expand All @@ -2234,15 +2242,23 @@ QUERY PLAN
-> Custom Scan (ChunkAppend) on metrics_compressed (actual rows=1 loops=1)
Order: metrics_compressed."time" DESC
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (actual rows=1 loops=1)
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (actual rows=1 loops=1)
Index Cond: (device_id = 1)
-> Sort (actual rows=1 loops=1)
Sort Key: compress_hyper_X_X_chunk._ts_meta_sequence_num
Sort Method: quicksort
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=6 loops=1)
Filter: (device_id = 1)
Rows Removed by Filter: 24
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (never executed)
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (never executed)
Index Cond: (device_id = 1)
-> Sort (never executed)
Sort Key: compress_hyper_X_X_chunk._ts_meta_sequence_num
-> Seq Scan on compress_hyper_X_X_chunk (never executed)
Filter: (device_id = 1)
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (never executed)
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (never executed)
Index Cond: (device_id = 1)
(13 rows)
-> Sort (never executed)
Sort Key: compress_hyper_X_X_chunk._ts_meta_sequence_num
-> Seq Scan on compress_hyper_X_X_chunk (never executed)
Filter: (device_id = 1)
(21 rows)

-- test sort column not in targetlist
:PREFIX
Expand Down Expand Up @@ -2281,12 +2297,21 @@ QUERY PLAN
-> Merge Append (actual rows=1 loops=1)
Sort Key: _hyper_X_X_chunk.device_id
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (actual rows=1 loops=1)
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (actual rows=1 loops=1)
-> Sort (actual rows=1 loops=1)
Sort Key: compress_hyper_X_X_chunk.device_id
Sort Method: quicksort
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=20 loops=1)
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (actual rows=1 loops=1)
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (actual rows=1 loops=1)
-> Sort (actual rows=1 loops=1)
Sort Key: compress_hyper_X_X_chunk.device_id
Sort Method: quicksort
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=30 loops=1)
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (actual rows=1 loops=1)
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (actual rows=1 loops=1)
(9 rows)
-> Sort (actual rows=1 loops=1)
Sort Key: compress_hyper_X_X_chunk.device_id
Sort Method: quicksort
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=30 loops=1)
(18 rows)

-- time column must be primary sort order
:PREFIX
Expand All @@ -2305,21 +2330,24 @@ QUERY PLAN
-> Sort (actual rows=1 loops=1)
Sort Key: compress_hyper_X_X_chunk.device_id, compress_hyper_X_X_chunk._ts_meta_sequence_num DESC
Sort Method: quicksort
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (actual rows=8 loops=1)
Index Cond: (device_id = ANY ('{1,2}'::integer[]))
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=8 loops=1)
Filter: (device_id = ANY ('{1,2}'::integer[]))
Rows Removed by Filter: 12
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (actual rows=1 loops=1)
-> Sort (actual rows=1 loops=1)
Sort Key: compress_hyper_X_X_chunk.device_id, compress_hyper_X_X_chunk._ts_meta_sequence_num DESC
Sort Method: quicksort
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (actual rows=12 loops=1)
Index Cond: (device_id = ANY ('{1,2}'::integer[]))
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=12 loops=1)
Filter: (device_id = ANY ('{1,2}'::integer[]))
Rows Removed by Filter: 18
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (actual rows=1 loops=1)
-> Sort (actual rows=1 loops=1)
Sort Key: compress_hyper_X_X_chunk.device_id, compress_hyper_X_X_chunk._ts_meta_sequence_num DESC
Sort Method: quicksort
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (actual rows=12 loops=1)
Index Cond: (device_id = ANY ('{1,2}'::integer[]))
(21 rows)
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=12 loops=1)
Filter: (device_id = ANY ('{1,2}'::integer[]))
Rows Removed by Filter: 18
(24 rows)

-- test equality constraint on ORDER BY prefix
-- currently not optimized
Expand Down Expand Up @@ -2355,20 +2383,30 @@ FROM :TEST_TABLE
WHERE device_id IN (1, 2)
ORDER BY time ASC;
QUERY PLAN
Sort (actual rows=27348 loops=1)
Sort Key: _hyper_X_X_chunk."time"
Sort Method: quicksort
-> Append (actual rows=27348 loops=1)
Custom Scan (ChunkAppend) on metrics_compressed (actual rows=27348 loops=1)
Order: metrics_compressed."time"
-> Sort (actual rows=7196 loops=1)
Sort Key: _hyper_X_X_chunk."time"
Sort Method: quicksort
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (actual rows=7196 loops=1)
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (actual rows=8 loops=1)
Index Cond: (device_id = ANY ('{1,2}'::integer[]))
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=8 loops=1)
Filter: (device_id = ANY ('{1,2}'::integer[]))
Rows Removed by Filter: 12
-> Sort (actual rows=10076 loops=1)
Sort Key: _hyper_X_X_chunk."time"
Sort Method: quicksort
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (actual rows=10076 loops=1)
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (actual rows=12 loops=1)
Index Cond: (device_id = ANY ('{1,2}'::integer[]))
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=12 loops=1)
Filter: (device_id = ANY ('{1,2}'::integer[]))
Rows Removed by Filter: 18
-> Sort (actual rows=10076 loops=1)
Sort Key: _hyper_X_X_chunk."time"
Sort Method: quicksort
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (actual rows=10076 loops=1)
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (actual rows=12 loops=1)
Index Cond: (device_id = ANY ('{1,2}'::integer[]))
(13 rows)
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=12 loops=1)
Filter: (device_id = ANY ('{1,2}'::integer[]))
Rows Removed by Filter: 18
(23 rows)

-- queries without ORDER BY shouldnt use ordered append
:PREFIX
Expand Down Expand Up @@ -2719,15 +2757,18 @@ QUERY PLAN
-> Result (actual rows=27348 loops=1)
-> Append (actual rows=27348 loops=1)
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (actual rows=7196 loops=1)
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (actual rows=8 loops=1)
Index Cond: (device_id = ANY ('{1,2}'::integer[]))
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=8 loops=1)
Filter: (device_id = ANY ('{1,2}'::integer[]))
Rows Removed by Filter: 12
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (actual rows=10076 loops=1)
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (actual rows=12 loops=1)
Index Cond: (device_id = ANY ('{1,2}'::integer[]))
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=12 loops=1)
Filter: (device_id = ANY ('{1,2}'::integer[]))
Rows Removed by Filter: 18
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (actual rows=10076 loops=1)
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (actual rows=12 loops=1)
Index Cond: (device_id = ANY ('{1,2}'::integer[]))
(15 rows)
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=12 loops=1)
Filter: (device_id = ANY ('{1,2}'::integer[]))
Rows Removed by Filter: 18
(18 rows)

-- test query with ORDER BY date_trunc
:PREFIX
Expand Down Expand Up @@ -2790,15 +2831,18 @@ QUERY PLAN
-> Result (actual rows=27348 loops=1)
-> Append (actual rows=27348 loops=1)
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (actual rows=7196 loops=1)
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (actual rows=8 loops=1)
Index Cond: (device_id = ANY ('{1,2}'::integer[]))
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=8 loops=1)
Filter: (device_id = ANY ('{1,2}'::integer[]))
Rows Removed by Filter: 12
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (actual rows=10076 loops=1)
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (actual rows=12 loops=1)
Index Cond: (device_id = ANY ('{1,2}'::integer[]))
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=12 loops=1)
Filter: (device_id = ANY ('{1,2}'::integer[]))
Rows Removed by Filter: 18
-> Custom Scan (DecompressChunk) on _hyper_X_X_chunk (actual rows=10076 loops=1)
-> Index Scan using compress_hyper_X_X_chunk_device_id__ts_meta_sequence_num_idx on compress_hyper_X_X_chunk (actual rows=12 loops=1)
Index Cond: (device_id = ANY ('{1,2}'::integer[]))
(15 rows)
-> Seq Scan on compress_hyper_X_X_chunk (actual rows=12 loops=1)
Filter: (device_id = ANY ('{1,2}'::integer[]))
Rows Removed by Filter: 18
(18 rows)

-- test query with now() should result in ordered ChunkAppend
:PREFIX
Expand Down