Change uid intersection policy in some cases #8919

harshil-goel · 2023-07-25T11:23:57Z

Title format:

Topic(Area): Feature
Topic must be one of build|ci|docs|feat|fix|perf|refactor|chore|test
Area must be one of acl|audit|backup|badger|cdc|dql|export|graphql|indexing|multi-tenancy|raft|restore|upgrade|zero
- Add [BREAKING] if it is a breaking change

Body Format:

Description:
Fixes:
Closes:
Docs:

LDBC Results. Upto 13.46% better

goos: linux
goarch: amd64
pkg: ldbc/standalone/dgraph
cpu: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
               │     in1      │                  in2                   │
               │    sec/op    │    sec/op      vs base                 │
Queries/IS01-8   659.9µ ± ∞ ¹    944.9µ ± ∞ ¹        ~ (p=1.000 n=1) ²
Queries/IS02-8   12.17m ± ∞ ¹    13.52m ± ∞ ¹        ~ (p=1.000 n=1) ²
Queries/IS03-8   833.3µ ± ∞ ¹   1219.4µ ± ∞ ¹        ~ (p=1.000 n=1) ²
Queries/IS04-8   575.7µ ± ∞ ¹   1009.3µ ± ∞ ¹        ~ (p=1.000 n=1) ²
Queries/IS05-8   770.8µ ± ∞ ¹   1079.8µ ± ∞ ¹        ~ (p=1.000 n=1) ²
Queries/IS06-8   985.6µ ± ∞ ¹   1193.3µ ± ∞ ¹        ~ (p=1.000 n=1) ²
Queries/IS07-8   4.792m ± ∞ ¹    4.810m ± ∞ ¹        ~ (p=1.000 n=1) ²
Queries/IC01-8    1.337 ± ∞ ¹     1.298 ± ∞ ¹        ~ (p=1.000 n=1) ²
Queries/IC02-8    1.308 ± ∞ ¹     1.392 ± ∞ ¹        ~ (p=1.000 n=1) ²
Queries/IC04-8    1.311 ± ∞ ¹     1.284 ± ∞ ¹        ~ (p=1.000 n=1) ²
Queries/IC06-8    2.853 ± ∞ ¹     3.035 ± ∞ ¹        ~ (p=1.000 n=1) ²
Queries/IC07-8   534.4m ± ∞ ¹    420.7m ± ∞ ¹        ~ (p=1.000 n=1) ²
Queries/IC08-8   33.02m ± ∞ ¹    32.92m ± ∞ ¹        ~ (p=1.000 n=1) ²
Queries/IC09-8    28.89 ± ∞ ¹     37.55 ± ∞ ¹        ~ (p=1.000 n=1) ²
Queries/IC10-8    35.33 ± ∞ ¹     36.77 ± ∞ ¹        ~ (p=1.000 n=1) ²
Queries/IC11-8   109.9m ± ∞ ¹    107.2m ± ∞ ¹        ~ (p=1.000 n=1) ²
geomean          67.41m          76.60m        +13.64%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05

Benchmark results:

BenchmarkListIntersectRatio/:IntersectWith:ratio=0.010000:size=100:overlap=0.01:-8                930638              1336 ns/op                                                                                                                                                                                                                                                                
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=0.010000:size=100:overlap=0.01:-8             1368980               797.5 ns/op                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                                                                                                
len: 10, compressed: 159, bytes/int: 15.900000                                                                                                                                                                                                                                                                                                                                                  
BenchmarkListIntersectRatio/:IntersectWith:ratio=0.100000:size=100:overlap=0.01:-8                        863913              1408 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=0.100000:size=100:overlap=0.01:-8             1411018               844.4 ns/op                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                                                                                                
len: 100, compressed: 156, bytes/int: 1.560000                                                                                                                                                                                                                                                                                                                                                  
BenchmarkListIntersectRatio/:IntersectWith:ratio=1.000000:size=100:overlap=0.01:-8                        679086              1526 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=1.000000:size=100:overlap=0.01:-8             1213830               982.4 ns/op                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                                                                                                                                
len: 1000, compressed: 158, bytes/int: 0.158000                                                                                                                                                                                                                                                                                                                                                 
BenchmarkListIntersectRatio/:IntersectWith:ratio=10.000000:size=100:overlap=0.01:-8                       610076              1950 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=10.000000:size=100:overlap=0.01:-8             677944              1728 ns/op                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                
len: 10000, compressed: 158, bytes/int: 0.015800                                                                                                                                                                                                                                                                                                                                                
BenchmarkListIntersectRatio/:IntersectWith:ratio=100.000000:size=100:overlap=0.01:-8                      404162              2945 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=100.000000:size=100:overlap=0.01:-8             52226             23293 ns/op                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                
len: 20000, compressed: 157, bytes/int: 0.007850                                                                                                                                                                                                                                                                                                                                                
BenchmarkListIntersectRatio/:IntersectWith:ratio=200.000000:size=100:overlap=0.01:-8                      360571              3185 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=200.000000:size=100:overlap=0.01:-8              4987            225165 ns/op                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                
len: 40000, compressed: 158, bytes/int: 0.003950                                                                                                                                                                                                                                                                                                                                                
BenchmarkListIntersectRatio/:IntersectWith:ratio=400.000000:size=100:overlap=0.01:-8                      302030              4073 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=400.000000:size=100:overlap=0.01:-8             14863             79218 ns/op                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                
len: 50000, compressed: 159, bytes/int: 0.003180                                                                                                                                                                                                                                                                                                                                                
BenchmarkListIntersectRatio/:IntersectWith:ratio=500.000000:size=100:overlap=0.01:-8                      243756              4642 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=500.000000:size=100:overlap=0.01:-8             24970             48431 ns/op                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                
len: 100000, compressed: 156, bytes/int: 0.001560                                                                                                                                                                                                                                                                                                                                               
BenchmarkListIntersectRatio/:IntersectWith:ratio=1000.000000:size=100:overlap=0.01:-8                     250286              4713 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=1000.000000:size=100:overlap=0.01:-8             2589            450080 ns/op                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                
len: 10, compressed: 1389, bytes/int: 138.900000                                                                                                                                                                                                                                                                                                                                                
BenchmarkListIntersectRatio/:IntersectWith:ratio=0.010000:size=1000:overlap=0.01:-8                       148689              8170 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=0.010000:size=1000:overlap=0.01:-8             296518              4018 ns/op                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                
len: 100, compressed: 1397, bytes/int: 13.970000                                                                                                                                                                                                                                                                                                                                                
BenchmarkListIntersectRatio/:IntersectWith:ratio=0.100000:size=1000:overlap=0.01:-8                       129889             12517 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=0.100000:size=1000:overlap=0.01:-8             186576              6241 ns/op                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                
len: 1000, compressed: 1388, bytes/int: 1.388000                                                                                                                                                                                                                                                                                                                                                
BenchmarkListIntersectRatio/:IntersectWith:ratio=1.000000:size=1000:overlap=0.01:-8                        76354             15379 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=1.000000:size=1000:overlap=0.01:-8              61672             19026 ns/op                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                
len: 10000, compressed: 1399, bytes/int: 0.139900                                                                                                                                                                                                                                                                                                                                               
BenchmarkListIntersectRatio/:IntersectWith:ratio=10.000000:size=1000:overlap=0.01:-8                       51190             22597 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=10.000000:size=1000:overlap=0.01:-8             34344             35790 ns/op                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                
len: 100000, compressed: 1386, bytes/int: 0.013860                                                                                                                                                                                                                                                                                                                                              
BenchmarkListIntersectRatio/:IntersectWith:ratio=100.000000:size=1000:overlap=0.01:-8                      24621             48535 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=100.000000:size=1000:overlap=0.01:-8             2541            457039 ns/op                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                
len: 200000, compressed: 1392, bytes/int: 0.006960                                                                                                                                                                                                                                                                                                                                              
BenchmarkListIntersectRatio/:IntersectWith:ratio=200.000000:size=1000:overlap=0.01:-8                      20144             58013 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=200.000000:size=1000:overlap=0.01:-8             1047           1090897 ns/op                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                
len: 400000, compressed: 1394, bytes/int: 0.003485                                                                                                                                                                                                                                                                                                                                              
BenchmarkListIntersectRatio/:IntersectWith:ratio=400.000000:size=1000:overlap=0.01:-8                      12552             94361 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=400.000000:size=1000:overlap=0.01:-8             2054            568504 ns/op                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                
len: 500000, compressed: 1389, bytes/int: 0.002778                                                                                                                                                                                                                                                                                                                                              
BenchmarkListIntersectRatio/:IntersectWith:ratio=500.000000:size=1000:overlap=0.01:-8                      10000            112407 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=500.000000:size=1000:overlap=0.01:-8              549           2062129 ns/op                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                
len: 1000000, compressed: 1392, bytes/int: 0.001392                                                                                                                                                                                                                                                                                                                                             
BenchmarkListIntersectRatio/:IntersectWith:ratio=1000.000000:size=1000:overlap=0.01:-8                      8742            134037 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=1000.000000:size=1000:overlap=0.01:-8            7544            148543 ns/op                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                
len: 100, compressed: 13807, bytes/int: 138.070000                                                                                                                                                                                                                                                                                                                                              
BenchmarkListIntersectRatio/:IntersectWith:ratio=0.010000:size=10000:overlap=0.01:-8                        9644            112963 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=0.010000:size=10000:overlap=0.01:-8             34372             35790 ns/op                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                
len: 1000, compressed: 13831, bytes/int: 13.831000                                                                                                                                                                                                                                                                                                                                              
BenchmarkListIntersectRatio/:IntersectWith:ratio=0.100000:size=10000:overlap=0.01:-8                        8937            128710 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=0.100000:size=10000:overlap=0.01:-8             25764             46519 ns/op                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                
len: 10000, compressed: 13814, bytes/int: 1.381400                                                                                                                                                                                                                                                                                                                                              
BenchmarkListIntersectRatio/:IntersectWith:ratio=1.000000:size=10000:overlap=0.01:-8                        5535            216706 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=1.000000:size=10000:overlap=0.01:-8              8152            143386 ns/op                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                
len: 100000, compressed: 13803, bytes/int: 0.138030                                                                                                                                                                                                                                                                                                                                             
BenchmarkListIntersectRatio/:IntersectWith:ratio=10.000000:size=10000:overlap=0.01:-8                       3475            312750 ns/op                                                                                                                                                                                                                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=10.000000:size=10000:overlap=0.01:-8             2124            492542 ns/op               

                                                                                                                                                                                                                                                                                                                                                                                                
len: 1000000, compressed: 13797, bytes/int: 0.013797                                                                                                                                                                                                                                                                                                                                            
BenchmarkListIntersectRatio/:IntersectWith:ratio=100.000000:size=10000:overlap=0.01:-8                      1648            635240 ns/op                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=100.000000:size=10000:overlap=0.01:-8             276           4166104 ns/op                                                        

len: 1000, compressed: 138236, bytes/int: 138.236000                                            
BenchmarkListIntersectRatio/:IntersectWith:ratio=0.010000:size=100000:overlap=0.01:-8                        882           1455047 ns/op                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=0.010000:size=100000:overlap=0.01:-8             3361            338829 ns/op                                                        

len: 10000, compressed: 138161, bytes/int: 13.816100                                            
BenchmarkListIntersectRatio/:IntersectWith:ratio=0.100000:size=100000:overlap=0.01:-8                        751           1597582 ns/op                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=0.100000:size=100000:overlap=0.01:-8             2313            530979 ns/op                                                        

len: 100000, compressed: 138193, bytes/int: 1.381930                                            
BenchmarkListIntersectRatio/:IntersectWith:ratio=1.000000:size=100000:overlap=0.01:-8                        489           2292397 ns/op                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=1.000000:size=100000:overlap=0.01:-8              672           1648515 ns/op                                                        

len: 1000000, compressed: 138170, bytes/int: 0.138170                                           
BenchmarkListIntersectRatio/:IntersectWith:ratio=10.000000:size=100000:overlap=0.01:-8                       387           3217208 ns/op                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=10.000000:size=100000:overlap=0.01:-8             213           5237041 ns/op                                                        

len: 10000, compressed: 1381938, bytes/int: 138.193800                                          
BenchmarkListIntersectRatio/:IntersectWith:ratio=0.010000:size=1000000:overlap=0.01:-8                        63          16173843 ns/op                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=0.010000:size=1000000:overlap=0.01:-8             344           3092056 ns/op                                                        

len: 100000, compressed: 1382024, bytes/int: 13.820240                                          
BenchmarkListIntersectRatio/:IntersectWith:ratio=0.100000:size=1000000:overlap=0.01:-8                        66          17806317 ns/op                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=0.100000:size=1000000:overlap=0.01:-8             224           4748480 ns/op                                                        

len: 1000000, compressed: 1382410, bytes/int: 1.382410                                          
BenchmarkListIntersectRatio/:IntersectWith:ratio=1.000000:size=1000000:overlap=0.01:-8                        54          22172310 ns/op                                                        
BenchmarkListIntersectRatio/compressed:IntersectWith:ratio=1.000000:size=1000000:overlap=0.01:-8              68          16036531 ns/op

damonfeldman

Minor note - possible dead code.

damonfeldman · 2023-07-27T14:35:26Z

posting/list.go

@@ -875,6 +878,21 @@ func (l *List) Rollup(alloc *z.Allocator, readTs uint64) ([]*bpb.KV, error) {
 	return kvs, nil
 }

+func (l *List) CreateUnpacked() {


Looks like all callers to this are commented out. Is it dead code?

sanjayk-github-dev · 2023-08-07T21:37:51Z

posting/list.go

@@ -71,6 +71,7 @@ const (
 // List stores the in-memory representation of a posting list.
 type List struct {
 	x.SafeMutex
+	immutable   *pb.List


Can you add a comment that this is unpacked posting list?

sanjayk-github-dev

Looks like this PR has more changes than just the uid intersection policy changes. Does it also include changes to read multiple keys using the new batch api? If yes, maybe you could split it into two separate PRs.

sanjayk-github-dev · 2023-08-07T21:48:03Z

worker/task.go

-			if err != nil {
-				return err
-			}
+			pickMultiplePostings := q.ExpandAll || (listType && len(q.Langs) == 0)


Didn't quite understand this block of changes.

Harshil Goel added 8 commits May 16, 2023 04:25

First commit

05c5f9d

fixed things

7fc02b1

Updated tracing percent

9d7a722

Added as a list

d0a5ef7

added some traces

cc92071

modified some benchmarks

0e493ae

added soem benchmarks

763821f

updated

76b4d08

harshil-goel requested review from mangalaman93, meghalims, sanjayk-github-dev, all-seeing-code, billprovince and joshua-goldstein as code owners July 25, 2023 11:23

dgraph-bot added area/testing Testing related issues area/core internal mechanisms go Pull requests that update Go code labels Jul 25, 2023

something something

b0073bb

damonfeldman reviewed Jul 27, 2023

View reviewed changes

Harshil Goel added 4 commits July 31, 2023 12:47

updated threshold

78dc8f1

updated handle value posting

4cdd30c

added more traces

15f821b

added multi api

13f7e22

sanjayk-github-dev reviewed Aug 7, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change uid intersection policy in some cases #8919

Change uid intersection policy in some cases #8919

harshil-goel commented Jul 25, 2023 •

edited

damonfeldman left a comment

damonfeldman Jul 27, 2023

sanjayk-github-dev Aug 7, 2023

sanjayk-github-dev left a comment •

edited

sanjayk-github-dev Aug 7, 2023

Change uid intersection policy in some cases #8919

Are you sure you want to change the base?

Change uid intersection policy in some cases #8919

Conversation

harshil-goel commented Jul 25, 2023 • edited

damonfeldman left a comment

Choose a reason for hiding this comment

damonfeldman Jul 27, 2023

Choose a reason for hiding this comment

sanjayk-github-dev Aug 7, 2023

Choose a reason for hiding this comment

sanjayk-github-dev left a comment • edited

Choose a reason for hiding this comment

sanjayk-github-dev Aug 7, 2023

Choose a reason for hiding this comment

harshil-goel commented Jul 25, 2023 •

edited

sanjayk-github-dev left a comment •

edited