Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Devops/benchmarks #2294

Merged
merged 69 commits into from Apr 18, 2023
Merged

Devops/benchmarks #2294

merged 69 commits into from Apr 18, 2023

Conversation

artgish
Copy link
Collaborator

@artgish artgish commented Apr 15, 2023

🚀 🚀 Pull Request

Checklist:

  • My code follows the style guidelines of this project and the Contributing document
  • I have commented my code, particularly in hard-to-understand areas
  • I have kept the coverage-rate up
  • I have performed a self-review of my own code and resolved any problems
  • I have checked to ensure there aren't any other open Pull Requests for the same change
  • I have described and made corresponding changes to the relevant documentation
  • New and existing unit tests pass locally with my changes

Changes

@artgish
update benchmarks timeout
c5b4e6b
@artgish
Merge branch 'main' into devops/benchmarks
7be8301
@artgish
update manual benchmarks config
253420e
@artgish
increase benchmark timeout
76ff955
@artgish
edit benchmarks
916c1fa
@artgish
edit benchmark workflows
e2dd706
@artgish
test
b38d21d
@artgish
test
5053bf6
@artgish
test
7ff061b
@artgish
test
0382c2d
@artgish
Merge branch 'main' into devops/benchmarks
27e124a
@artgish
edit benchmark workflows
f6e3460
@artgish
edit workflow names
8b67dcc
@artgish
Merge branch 'main' into devops/benchmarks
a71fb4e
@artgish
update workflow envs
504e3f0
@artgish
test
04a0e68
@artgish
update workflows
b9f0ffd
@artgish
update workflows
3e67a97
@artgish
test latest releae
d74d60b
@artgish
test
642c732
@artgish
add workflow env
eb4eebd
@artgish
edit workflows
1c7b5a2
@artgish
edit benchmark workflows
c9fdafd
@artgish
edit workflows
c94b808
@artgish
temp remove concurrency
0d09051
@artgish
revert latest changes
70a372d
@artgish
update benchmarks
143f3a1
@artgish
tst
6b51e03
@artgish
edit workflows
2160d20
@artgish
test
e0c1a0e
@artgish artgish added the run-benchmarks Run Deep Lake benchmark action label Apr 16, 2023
@github-actions github-actions bot removed the run-benchmarks Run Deep Lake benchmark action label Apr 16, 2023
@artgish artgish added the run-benchmarks Run Deep Lake benchmark action label Apr 16, 2023
@github-actions github-actions bot removed the run-benchmarks Run Deep Lake benchmark action label Apr 16, 2023
@artgish artgish added the run-benchmarks Run Deep Lake benchmark action label Apr 16, 2023
@github-actions github-actions bot removed the run-benchmarks Run Deep Lake benchmark action label Apr 16, 2023
@artgish artgish added the run-benchmarks Run Deep Lake benchmark action label Apr 17, 2023
@github-actions github-actions bot removed the run-benchmarks Run Deep Lake benchmark action label Apr 17, 2023
@artgish artgish added the run-benchmarks Run Deep Lake benchmark action label Apr 17, 2023
@github-actions github-actions bot removed the run-benchmarks Run Deep Lake benchmark action label Apr 17, 2023
@github-actions
Copy link

github-actions bot commented Apr 17, 2023 •
edited

Benchmark Report linux_cpu (Failed)

Benchmark Check Status Performance Diff Current Result Latest Result
image/cifar10
deeplake image/cifar10 local 50000 ingest_serial None None None None Passed 29.68171% 13.487797746999604 19.18107416699968
deeplake image/cifar10 local 50000 ingest_parallel 8 None None None Passed 43.66425% 20.039720467999814 35.571937448001336
deeplake image/cifar10 local 50000 ingest_parallel 16 None None None Passed 43.63196% 20.31386742099994 36.03792113199961
deeplake image/cifar10 local 50000 sequential_read None None None None Passed 1.32422% 4.815989710999929 4.880617341999823
deeplake image/cifar10 local 50000 shuffled_read None None None None Passed 2.54957% 5.495216943000287 5.6389908519995515
deeplake_cpp image/cifar10 local 50000 query None None None None Passed 17.16139% 0.38177358700022523 0.4608558889995038
deeplake image/cifar10 local 50000 query None None None None Passed 0.13317% 2.5346774489999007 2.5380602610002825
deeplake_python image/cifar10 local 50000 dataloader_iteration 4 4 True None Passed 0.98934% 14.648337828999956 14.794713019999108
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 4 4 True None Passed 1.18541% 37.82677452100006 38.28054784300002
deeplake_python image/cifar10 local 50000 dataloader_iteration 4 4 False None In Threshold -0.88766% 22.770939448999798 22.570587486001386
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 4 4 False None Passed 0.58337% 26.00256016200001 26.155143314999805
deeplake_python image/cifar10 local 50000 dataloader_iteration 8 4 True None In Threshold -5.41449% 24.178868580999733 22.936946240999532
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 8 4 True None In Threshold -4.0999% 39.24257420999993 37.697034479999274
deeplake_python image/cifar10 local 50000 dataloader_iteration 8 4 False None In Threshold -4.01901% 32.26030871100011 31.013864685000954
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 8 4 False None In Threshold -1.6546% 26.26083777599979 25.833399451999867
deeplake_python image/cifar10 local 50000 dataloader_iteration 16 4 True None In Threshold -5.76673% 42.98474141199995 40.64107725099893
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 16 4 True None In Threshold -4.12967% 39.99349787999972 38.407396555998275
deeplake_python image/cifar10 local 50000 dataloader_iteration 16 4 False None In Threshold -4.58979% 51.444616728000256 49.18703785100115
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 16 4 False None In Threshold -0.56772% 26.685082703999797 26.534437161000824
deeplake_python image/cifar10 local 50000 dataloader_iteration 4 8 True None Passed 0.36862% 13.678871178999998 13.729483551998783
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 4 8 True None Passed 2.62189% 27.802262821000113 28.550825795999117
deeplake_python image/cifar10 local 50000 dataloader_iteration 4 8 False None Passed 1.3245% 17.044864276000226 17.27365425300013
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 4 8 False None Passed 3.26437% 15.243007526999918 15.757385450999209
deeplake_python image/cifar10 local 50000 dataloader_iteration 8 8 True None In Threshold -3.11672% 22.862766789000034 22.17174381499899
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 8 8 True None In Threshold -0.31443% 28.10400456000025 28.015913945000648
deeplake_python image/cifar10 local 50000 dataloader_iteration 8 8 False None In Threshold -2.1986% 26.3147686100001 25.748655232999226
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 8 8 False None In Threshold -1.57915% 15.708174565999798 15.46396566399926
deeplake_python image/cifar10 local 50000 dataloader_iteration 16 8 True None In Threshold -4.89642% 41.76454842600015 39.81503772599899
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 16 8 True None In Threshold -2.95507% 29.555620356999952 28.707295349999185
deeplake_python image/cifar10 local 50000 dataloader_iteration 16 8 False None In Threshold -6.10893% 46.17990893399974 43.52122874499946
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 16 8 False None In Threshold -3.70927% 16.580795511999895 15.987769945000764
deeplake image/cifar10 s3 50000 ingest_serial None None None None Passed 14.19873% 20.588109175000227 23.99511197100037
deeplake image/cifar10 s3 50000 ingest_parallel 8 None None None Passed 38.25172% 28.222295812000084 45.70540029199947
deeplake image/cifar10 s3 50000 ingest_parallel 16 None None None Passed 19.42808% 41.22454992400003 51.164910073999636
deeplake image/cifar10 s3 50000 sequential_read None None None None Failed -75.27789% 8.713661107000007 4.971343782000986
deeplake image/cifar10 s3 50000 shuffled_read None None None None Passed 1.61102% 5.803096642000128 5.898123158998715
deeplake_cpp image/cifar10 s3 50000 query None None None None Passed 9.16338% 0.7612240500000098 0.8380077599995275
deeplake image/cifar10 s3 50000 query None None None None In Threshold -0.10603% 2.766397331999997 2.7634685280008853
deeplake_python image/cifar10 s3 50000 dataloader_iteration 4 4 True None Passed 5.52653% 18.768444492999834 19.86635564200151
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 4 4 True None Passed 2.34443% 41.351118899000085 42.34384368300016
deeplake_python image/cifar10 s3 50000 dataloader_iteration 4 4 False None In Threshold -1.86164% 27.290772915999923 26.791996346000815
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 4 4 False None In Threshold -1.68265% 26.524575484999787 26.085647760999564
deeplake_python image/cifar10 s3 50000 dataloader_iteration 8 4 True None Passed 4.66194% 28.733657317000052 30.138707628000702
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 8 4 True None Passed 4.94533% 39.953764386999865 42.03240446199925
deeplake_python image/cifar10 s3 50000 dataloader_iteration 8 4 False None In Threshold -0.64497% 38.744840121999914 38.4965508589994
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 8 4 False None In Threshold -1.47451% 26.73903275099974 26.350488662001226
deeplake_python image/cifar10 s3 50000 dataloader_iteration 16 4 True None In Threshold -4.50506% 50.66100032700024 48.4770793689986
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 16 4 True None Passed 4.81342% 40.98996586600015 43.06276261499988
deeplake_python image/cifar10 s3 50000 dataloader_iteration 16 4 False None In Threshold -3.44732% 61.47871507799982 59.42998385200008
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 16 4 False None In Threshold -1.63418% 27.641601991000243 27.197151719999965
deeplake_python image/cifar10 s3 50000 dataloader_iteration 4 8 True None In Threshold -3.36377% 17.24579586999971 16.684572132999165
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 4 8 True None Passed 0.43055% 28.199994156999765 28.321928990999368
deeplake_python image/cifar10 s3 50000 dataloader_iteration 4 8 False None Failed -4146.62982% 22.1266403730001 0.5210368309999467
deeplake_python image/cifar10 s3 50000 dataloader_iteration 8 8 True None Passed 0.42874% 27.030627327000275 27.147016205999535
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 8 8 True None In Threshold -0.36251% 28.579476612000235 28.476252443000703
deeplake_python image/cifar10 s3 50000 dataloader_iteration 8 8 False None Failed -10167.77771% 32.22233927199977 0.3138214990012784
deeplake_python image/cifar10 s3 50000 dataloader_iteration 16 8 True None In Threshold -0.10669% 49.08367067500012 49.03135667099923
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 16 8 True None In Threshold -1.19458% 29.776889655000105 29.425383113999487
deeplake_python image/cifar10 s3 50000 dataloader_iteration 16 8 False None In Threshold -3.01047% 54.59724531699976 53.00164906499958
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 16 8 False None In Threshold -2.84949% 17.270612675999928 16.792124517000047
image/mnist
deeplake image/mnist local 60000 ingest_serial None None None None Passed 23.3486% 16.40424703000008 21.40110922200074
deeplake image/mnist local 60000 ingest_parallel 8 None None None Passed 39.3096% 23.23259196900017 38.2804976339994
deeplake image/mnist local 60000 ingest_parallel 16 None None None Passed 38.68534% 23.73688365199996 38.713216134001414
deeplake image/mnist local 60000 sequential_read None None None None In Threshold -3.20214% 5.621075774000019 5.446666945999823
deeplake image/mnist local 60000 shuffled_read None None None None In Threshold -1.28499% 6.390859291000197 6.309780621000755
deeplake_cpp image/mnist local 60000 query None None None None Passed 5.66274% 0.3540099140000166 0.375264956999672
deeplake image/mnist local 60000 query None None None None Passed 2.1656% 2.977133430000322 3.0430263100006414
deeplake_python image/mnist local 60000 dataloader_iteration 4 4 True None In Threshold -0.79072% 15.227179948999947 15.107722798000395
deeplake_cpp image/mnist local 60000 dataloader_iteration 4 4 True None In Threshold -0.64579% 66.71134345600058 66.28329441400092
deeplake_python image/mnist local 60000 dataloader_iteration 4 4 False None In Threshold -2.99868% 27.445784515000014 26.64673062300062
deeplake_cpp image/mnist local 60000 dataloader_iteration 4 4 False None In Threshold -0.72059% 29.660169691000192 29.447973604999788
deeplake_python image/mnist local 60000 dataloader_iteration 8 4 True None In Threshold -5.04597% 25.26096084099936 24.047526665999612
deeplake_cpp image/mnist local 60000 dataloader_iteration 8 4 True None In Threshold -4.15026% 68.88442453700009 66.13946447200033
deeplake_python image/mnist local 60000 dataloader_iteration 8 4 False None In Threshold -1.91803% 36.345126461999826 35.66114377899976
deeplake_cpp image/mnist local 60000 dataloader_iteration 8 4 False None In Threshold -1.74321% 30.078567452999778 29.563216243999705
deeplake_python image/mnist local 60000 dataloader_iteration 16 4 True None In Threshold -6.44799% 43.961682781999116 41.29873505999967
deeplake_cpp image/mnist local 60000 dataloader_iteration 16 4 True None In Threshold -3.68981% 69.29808768900057 66.83211291899897
deeplake_python image/mnist local 60000 dataloader_iteration 16 4 False None In Threshold -4.5217% 56.01922157700028 53.59578287400109
deeplake_cpp image/mnist local 60000 dataloader_iteration 16 4 False None In Threshold -0.46147% 30.60628474999976 30.465686352001285
deeplake_python image/mnist local 60000 dataloader_iteration 4 8 True None In Threshold -4.97847% 14.218630141000176 13.544329350001135
deeplake_cpp image/mnist local 60000 dataloader_iteration 4 8 True None Passed 1.73541% 53.77961781299928 54.729399492000084
deeplake_python image/mnist local 60000 dataloader_iteration 4 8 False None In Threshold -2.91155% 20.377645665999808 19.801134620000084
deeplake_cpp image/mnist local 60000 dataloader_iteration 4 8 False None In Threshold -2.21096% 17.912877609000134 17.52540468100051
deeplake_python image/mnist local 60000 dataloader_iteration 8 8 True None In Threshold -6.33798% 23.924605377000262 22.498651889998655
deeplake_cpp image/mnist local 60000 dataloader_iteration 8 8 True None Passed 0.34638% 54.317549735000284 54.50635330300065
deeplake_python image/mnist local 60000 dataloader_iteration 8 8 False None In Threshold -4.82109% 29.949002686998938 28.571537768999406
deeplake_cpp image/mnist local 60000 dataloader_iteration 8 8 False None In Threshold -1.15111% 17.852139134000026 17.648980459000086
deeplake_python image/mnist local 60000 dataloader_iteration 16 8 True None In Threshold -5.60301% 42.732332141998995 40.46507379800096
deeplake_cpp image/mnist local 60000 dataloader_iteration 16 8 True None In Threshold -0.00034% 55.2399844520005 55.23979386700012
deeplake_python image/mnist local 60000 dataloader_iteration 16 8 False None In Threshold -3.84347% 48.70725879999918 46.90449547300159
deeplake_cpp image/mnist local 60000 dataloader_iteration 16 8 False None In Threshold -4.03144% 18.850832164998792 18.12031542600016
deeplake image/mnist s3 60000 ingest_serial None None None None Passed 9.98414% 21.896782429999803 24.325467934999324
deeplake image/mnist s3 60000 ingest_parallel 8 None None None Passed 26.98152% 35.137929767999594 48.12197381899932
deeplake image/mnist s3 60000 ingest_parallel 16 None None None Passed 21.01309% 42.29107016799935 53.54186603299968
deeplake image/mnist s3 60000 sequential_read None None None None In Threshold -3.19629% 5.793128413999511 5.6137041870006215
deeplake image/mnist s3 60000 shuffled_read None None None None Passed 0.01583% 6.693921571999454 6.694982147000701
deeplake_cpp image/mnist s3 60000 query None None None None In Threshold -2.68873% 0.7348185979990376 0.715580359001251
deeplake image/mnist s3 60000 query None None None None In Threshold -6.74803% 3.356027259000257 3.143883613000071
deeplake_python image/mnist s3 60000 dataloader_iteration 4 4 True None In Threshold -1.87758% 19.287826801999472 18.932363650999832
deeplake_cpp image/mnist s3 60000 dataloader_iteration 4 4 True None In Threshold -0.90148% 71.02421409599992 70.38965857200128
deeplake_python image/mnist s3 60000 dataloader_iteration 4 4 False None In Threshold -4.01931% 30.738102340001205 29.550384703999953
deeplake_cpp image/mnist s3 60000 dataloader_iteration 4 4 False None In Threshold -2.27724% 30.43873930799964 29.761008830999344
deeplake_python image/mnist s3 60000 dataloader_iteration 8 4 True None In Threshold -8.19446% 30.566199737000716 28.251174833998448
deeplake_cpp image/mnist s3 60000 dataloader_iteration 8 4 True None Passed 0.72277% 69.86945443199875 70.37812412200037
deeplake_python image/mnist s3 60000 dataloader_iteration 8 4 False None In Threshold -1.14483% 41.94278163099989 41.4680380789996
deeplake_cpp image/mnist s3 60000 dataloader_iteration 8 4 False None In Threshold -0.24832% 30.399334128998817 30.32402558099966
deeplake_python image/mnist s3 60000 dataloader_iteration 16 4 True None In Threshold -5.59433% 52.22966737299976 49.462569153000004
deeplake_cpp image/mnist s3 60000 dataloader_iteration 16 4 True None Passed 1.22879% 70.49293058199873 71.36992229099997
deeplake_python image/mnist s3 60000 dataloader_iteration 16 4 False None In Threshold -4.12192% 64.60813507199964 62.05046746900007
deeplake_cpp image/mnist s3 60000 dataloader_iteration 16 4 False None In Threshold -1.50486% 31.552315472999908 31.084536488000595
deeplake_python image/mnist s3 60000 dataloader_iteration 4 8 True None In Threshold -3.64922% 17.777207656001337 17.151316482000766
deeplake_cpp image/mnist s3 60000 dataloader_iteration 4 8 True None Passed 0.8112% 54.20769799599839 54.65102654399925
deeplake_python image/mnist s3 60000 dataloader_iteration 4 8 False None Failed -2739.6177% 24.18474491699999 0.8516905540000153
deeplake_python image/mnist s3 60000 dataloader_iteration 8 8 True None In Threshold -1.59901% 28.36870137400001 27.922217180999723
deeplake_cpp image/mnist s3 60000 dataloader_iteration 8 8 True None Passed 0.54759% 54.463236224000866 54.76312492600118
deeplake_python image/mnist s3 60000 dataloader_iteration 8 8 False None Failed -6201.39077% 34.88765413800138 0.5536464899996645
deeplake_python image/mnist s3 60000 dataloader_iteration 16 8 True None In Threshold -5.2467% 50.93452700399939 48.39537374500105
deeplake_cpp image/mnist s3 60000 dataloader_iteration 16 8 True None Passed 0.67838% 55.846754564998264 56.22819435400015
deeplake_python image/mnist s3 60000 dataloader_iteration 16 8 False None Failed -6700.01312% 57.02830828200058 0.8386543779997737
image/coco
deeplake image/coco local 2000 ingest_serial None None None None Passed 71.22837% 2.245807979001256 7.805639640000663
deeplake image/coco local 2000 ingest_parallel 8 None None None Passed 71.93424% 6.107497884000622 21.761387361000743
deeplake image/coco local 2000 ingest_parallel 16 None None None Passed 71.16659% 6.295777486000588 21.835015671998917
deeplake image/coco local 2000 sequential_read None None None None In Threshold -0.02733% 7.1012533780012745 7.09931239099933
deeplake image/coco local 2000 shuffled_read None None None None In Threshold -0.03983% 8.038155539001309 8.034958087000632
deeplake_cpp image/coco local 2000 query None None None None In Threshold -0.8034% 0.42409157899965066 0.4207058899992262
deeplake image/coco local 2000 query None None None None Passed 5.62362% 0.45294561499940755 0.47993801300071937
deeplake image/coco s3 2000 ingest_serial None None None None Passed 22.42764% 10.279467791999195 13.251455151999835
deeplake image/coco s3 2000 ingest_parallel 8 None None None Failed -918.65953% 20.85084158400059 2.0468870740005514
deeplake image/coco s3 2000 ingest_parallel 16 None None None Passed 34.61663% 28.5647572630005 43.688114950000454
deeplake image/coco s3 2000 sequential_read None None None None Passed 29.46964% 11.068472843000563 15.693202835000193
deeplake image/coco s3 2000 shuffled_read None None None None Passed 5.31101% 60.89397095300046 64.309448127
deeplake_cpp image/coco s3 2000 query None None None None Failed -26.40707% 0.8768084469993482 0.6936434459985321
deeplake image/coco s3 2000 query None None None None Passed 0.1939% 0.54046626499985 0.5415183829991292

Missing checks on benchmarks

Benchmark Check Exist Only Result
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 4 8 False None Current benchmark 15.77804
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 8 8 False None Current benchmark 16.22982
deeplake_cpp image/mnist s3 60000 dataloader_iteration 4 8 False None Current benchmark 17.72994
deeplake_cpp image/mnist s3 60000 dataloader_iteration 8 8 False None Current benchmark 18.14512
deeplake_cpp image/mnist s3 60000 dataloader_iteration 16 8 False None Current benchmark 19.26034

@artgish artgish added the run-benchmarks Run Deep Lake benchmark action label Apr 17, 2023
@github-actions github-actions bot removed the run-benchmarks Run Deep Lake benchmark action label Apr 17, 2023
@artgish artgish marked this pull request as ready for review April 17, 2023 13:42
@artgish artgish requested review from tatevikh and removed request for tatevikh April 17, 2023 13:48
@github-actions
Copy link

Benchmark Report linux_gpu (Failed)

Benchmark Check Status Performance Diff Current Result Latest Result
image/coco
deeplake image/coco local 2000 ingest_parallel 16 None None None Passed 63.43937% 9.276648525001292 25.373325124000075
deeplake_python image/coco local 2000 training 4 4 False 2 Passed 1.15452% 258.5323775290017 261.55203740899924
deeplake_python image/coco local 2000 inference 4 4 False None Passed 2.08809% 61.63579505399866 62.95026486499955
deeplake_cpp image/coco local 2000 training 4 4 False 2 Passed 1.02975% 257.7450696169999 260.42680907999966
deeplake_cpp image/coco local 2000 inference 4 4 False None Passed 1.92492% 69.07487589899938 70.43060893800066
deeplake_python image/coco local 2000 training 4 4 True 2 Passed 0.87478% 263.4063861529994 265.73094400399896
deeplake_python image/coco local 2000 inference 4 4 True None Passed 3.05898% 68.44949870900018 70.60943191200022
deeplake_cpp image/coco local 2000 training 4 4 True 2 Passed 1.17206% 263.1475690340012 266.2683999609999
deeplake_cpp image/coco local 2000 inference 4 4 True None Passed 1.913% 72.88013948200023 74.3015312030002
deeplake_python image/coco local 2000 training 8 4 False 2 Passed 1.68916% 260.0362848900004 264.5041892120007
deeplake_python image/coco local 2000 inference 8 4 False None Passed 2.92788% 63.731397719002416 65.65366061200075
deeplake_cpp image/coco local 2000 training 8 4 False 2 Passed 1.0497% 260.2063423639993 262.9667014750012
deeplake_cpp image/coco local 2000 inference 8 4 False None Passed 1.21254% 71.29999863500052 72.17514979599946
deeplake_python image/coco local 2000 training 8 4 True 2 Passed 1.52086% 268.8826836549997 273.0351481130001
deeplake_python image/coco local 2000 inference 8 4 True None Passed 2.55077% 71.49988517800011 73.37143217700032
deeplake_cpp image/coco local 2000 training 8 4 True 2 Passed 0.86081% 266.75578841699826 269.07199298900014
deeplake_cpp image/coco local 2000 inference 8 4 True None Passed 0.36551% 74.21003048000057 74.4822713350004
deeplake_python image/coco local 2000 training 16 4 False 2 Passed 2.58732% 271.7482936560009 278.9660370950005
deeplake_python image/coco local 2000 inference 16 4 False None Passed 3.95701% 70.2986636300011 73.19499419600106
deeplake_cpp image/coco local 2000 training 16 4 False 2 Passed 0.84189% 264.81051122900317 267.058845915999
deeplake_cpp image/coco local 2000 inference 16 4 False None Passed 1.60843% 73.56842425899958 74.77106000599997
deeplake_python image/coco local 2000 training 16 4 True 2 Passed 2.21537% 284.05929800300146 290.49484323400065
deeplake_python image/coco local 2000 inference 16 4 True None Passed 4.41437% 78.6127005620001 82.24321989300006
deeplake_cpp image/coco local 2000 training 16 4 True 2 Passed 0.43265% 271.2692576779991 272.4480041830011
deeplake_cpp image/coco local 2000 inference 16 4 True None Passed 1.4443% 75.70793430399863 76.81739635599843
deeplake_python image/coco local 2000 training 4 8 False 2 Passed 1.23887% 246.1823926219986 249.27053956499913
deeplake_python image/coco local 2000 inference 4 8 False None Passed 0.90595% 60.4976477550008 61.05073740999978
deeplake_cpp image/coco local 2000 training 4 8 False 2 Passed 1.51947% 248.00462647200038 251.83113475600112
deeplake_cpp image/coco local 2000 inference 4 8 False None Passed 1.86276% 68.27455414699944 69.57048379700063
deeplake_python image/coco local 2000 training 4 8 True 2 Passed 1.58367% 252.806325870999 256.8743714859993
deeplake_python image/coco local 2000 inference 4 8 True None Passed 3.26555% 66.89453742999831 69.15275890799967
deeplake_cpp image/coco local 2000 training 4 8 True 2 Passed 1.3636% 252.9348063590005 256.4315049779998
deeplake_cpp image/coco local 2000 inference 4 8 True None Passed 1.40055% 69.62868487100059 70.61771573899932
deeplake_python image/coco local 2000 training 8 8 False 2 Passed 1.94953% 249.81597657899692 254.78304863500125
deeplake_python image/coco local 2000 inference 8 8 False None Passed 3.08224% 61.69121236099818 63.65314746399963
deeplake_cpp image/coco local 2000 training 8 8 False 2 Passed 1.86493% 249.3961399530017 254.1355789070003
deeplake_cpp image/coco local 2000 inference 8 8 False None Passed 2.03882% 69.27176761200099 70.71348501600005
deeplake_python image/coco local 2000 training 8 8 True 2 Passed 1.6748% 258.593291186 262.99796909499855
deeplake_python image/coco local 2000 inference 8 8 True None Passed 2.12772% 69.34775315200022 70.85534542000096
deeplake_cpp image/coco local 2000 training 8 8 True 2 Passed 1.56377% 254.69561282700306 258.74173788999906
deeplake_cpp image/coco local 2000 inference 8 8 True None Passed 1.85848% 70.66010711900162 71.99818221699934
deeplake_python image/coco local 2000 training 16 8 False 2 Passed 3.17898% 260.6460473950028 269.20397839
deeplake_python image/coco local 2000 inference 16 8 False None Passed 3.3169% 68.57045797300088 70.9229019760005
deeplake_cpp image/coco local 2000 training 16 8 False 2 Passed 1.08107% 255.3177496870012 258.10808173400073
deeplake_cpp image/coco local 2000 inference 16 8 False None Passed 1.97418% 71.57382350699845 73.01527353599886
deeplake_python image/coco local 2000 training 16 8 True 2 Passed 2.05236% 273.601523526002 279.33446470499985
deeplake_python image/coco local 2000 inference 16 8 True None Passed 2.82788% 77.17492807000235 79.42085509000026
deeplake_cpp image/coco local 2000 training 16 8 True 2 Failed -22.22384% 260.32679180799823 212.99181626999962

Missing checks on benchmarks

Benchmark Check Exist Only Result
deeplake_cpp image/coco local 2000 inference 16 8 True None Current benchmark 72.97195
deeplake_python image/coco local 2000 training 4 16 False 2 Current benchmark 228.4599
deeplake_python image/coco local 2000 inference 4 16 False None Current benchmark 53.81412
deeplake_cpp image/coco local 2000 training 4 16 False 2 Current benchmark 233.84239
deeplake_cpp image/coco local 2000 inference 4 16 False None Current benchmark 65.4174
deeplake_python image/coco local 2000 training 4 16 True 2 Current benchmark 236.28473
deeplake_python image/coco local 2000 inference 4 16 True None Current benchmark 61.69483
deeplake_cpp image/coco local 2000 training 4 16 True 2 Current benchmark 236.65931
deeplake_cpp image/coco local 2000 inference 4 16 True None Current benchmark 64.34274
deeplake_python image/coco local 2000 training 8 16 False 2 Current benchmark 232.96074
deeplake_python image/coco local 2000 inference 8 16 False None Current benchmark 56.74272
deeplake_cpp image/coco local 2000 training 8 16 False 2 Current benchmark 236.29272
deeplake_cpp image/coco local 2000 inference 8 16 False None Current benchmark 66.18565
deeplake_python image/coco local 2000 training 8 16 True 2 Current benchmark 241.87449
deeplake_python image/coco local 2000 inference 8 16 True None Current benchmark 63.78979
deeplake_cpp image/coco local 2000 training 8 16 True 2 Current benchmark 240.1415
deeplake_cpp image/coco local 2000 inference 8 16 True None Current benchmark 65.61611
deeplake_python image/coco local 2000 training 16 16 False 2 Current benchmark 20.04982
deeplake image/cifar10 local 50000 ingest_parallel 16 None None None Latest benchmark 53.65010190999999
deeplake image/cifar10 s3 50000 ingest_parallel 16 None None None Latest benchmark 62.99502516900009
deeplake image/mnist local 60000 ingest_parallel 16 None None None Latest benchmark 60.34917944799997
deeplake_python image/mnist local 60000 training 4 4 False 2 Latest benchmark 135.73338498700002
deeplake_python image/mnist local 60000 inference 4 4 False None Latest benchmark 44.709398345999944
deeplake_cpp image/mnist local 60000 training 4 4 False 2 Latest benchmark 141.84922241999993
deeplake_cpp image/mnist local 60000 inference 4 4 False None Latest benchmark 53.44088587200008
deeplake_python image/mnist local 60000 training 4 4 True 2 Latest benchmark 141.40745484700005
deeplake_python image/mnist local 60000 inference 4 4 True None Latest benchmark 49.31092581600001
deeplake_cpp image/mnist local 60000 training 4 4 True 2 Latest benchmark 207.50157650699998
deeplake_cpp image/mnist local 60000 inference 4 4 True None Latest benchmark 57.1719844480001
deeplake_python image/mnist local 60000 training 8 4 False 2 Latest benchmark 167.17846722500008
deeplake_python image/mnist local 60000 inference 8 4 False None Latest benchmark 60.510181186999944
deeplake_cpp image/mnist local 60000 training 8 4 False 2 Latest benchmark 145.24066736600003
deeplake_cpp image/mnist local 60000 inference 8 4 False None Latest benchmark 54.045874284999854
deeplake_python image/mnist local 60000 training 8 4 True 2 Latest benchmark 172.37033348600016
deeplake_python image/mnist local 60000 inference 8 4 True None Latest benchmark 64.55066901300006
deeplake_cpp image/mnist local 60000 training 8 4 True 2 Latest benchmark 217.33133449499974
deeplake_cpp image/mnist local 60000 inference 8 4 True None Latest benchmark 58.287158420000196
deeplake_python image/mnist local 60000 training 16 4 False 2 Latest benchmark 231.3719305579998
deeplake_python image/mnist local 60000 inference 16 4 False None Latest benchmark 91.86259168300012
deeplake_cpp image/mnist local 60000 training 16 4 False 2 Latest benchmark 149.21740289800027
deeplake_cpp image/mnist local 60000 inference 16 4 False None Latest benchmark 56.38090685299994
deeplake_python image/mnist local 60000 training 16 4 True 2 Latest benchmark 233.82486923700026
deeplake_python image/mnist local 60000 inference 16 4 True None Latest benchmark 95.5339563120001
deeplake_cpp image/mnist local 60000 training 16 4 True 2 Latest benchmark 220.58889900700024
deeplake_cpp image/mnist local 60000 inference 16 4 True None Latest benchmark 60.5188536669998
deeplake_python image/mnist local 60000 training 4 8 False 2 Latest benchmark 86.01368278200016
deeplake_python image/mnist local 60000 inference 4 8 False None Latest benchmark 31.216364513999906
deeplake_cpp image/mnist local 60000 training 4 8 False 2 Latest benchmark 85.40276477999987
deeplake_cpp image/mnist local 60000 inference 4 8 False None Latest benchmark 30.77606492299992
deeplake_python image/mnist local 60000 training 4 8 True 2 Latest benchmark 104.84151024599987
deeplake_python image/mnist local 60000 inference 4 8 True None Latest benchmark 40.26880459399945
deeplake_cpp image/mnist local 60000 training 4 8 True 2 Latest benchmark 147.46138482199967
deeplake_cpp image/mnist local 60000 inference 4 8 True None Latest benchmark 32.533572156000446
deeplake_python image/mnist local 60000 training 8 8 False 2 Latest benchmark 117.09869198599972
deeplake_python image/mnist local 60000 inference 8 8 False None Latest benchmark 46.77325756500068
deeplake_cpp image/mnist local 60000 training 8 8 False 2 Latest benchmark 87.782322002
deeplake_cpp image/mnist local 60000 inference 8 8 False None Latest benchmark 31.446419957000217
deeplake_python image/mnist local 60000 training 8 8 True 2 Latest benchmark 135.90162996900017
deeplake_python image/mnist local 60000 inference 8 8 True None Latest benchmark 56.58728998999959
deeplake_cpp image/mnist local 60000 training 8 8 True 2 Latest benchmark 148.78492256700065
deeplake_cpp image/mnist local 60000 inference 8 8 True None Latest benchmark 33.50284768199981
deeplake_python image/mnist local 60000 training 16 8 False 2 Latest benchmark 179.54521041200042
deeplake_python image/mnist local 60000 inference 16 8 False None Latest benchmark 78.2278923240001
deeplake_cpp image/mnist local 60000 training 16 8 False 2 Latest benchmark 91.86556474100053
deeplake_cpp image/mnist local 60000 inference 16 8 False None Latest benchmark 33.793420661999335
deeplake_python image/mnist local 60000 training 16 8 True 2 Latest benchmark 196.888232755
deeplake_python image/mnist local 60000 inference 16 8 True None Latest benchmark 86.6432283969998
deeplake_cpp image/mnist local 60000 training 16 8 True 2 Latest benchmark 152.85101111199947
deeplake_cpp image/mnist local 60000 inference 16 8 True None Latest benchmark 36.00275086500005
deeplake_python image/mnist local 60000 training 4 16 False 2 Latest benchmark 60.31432908500028
deeplake_python image/mnist local 60000 inference 4 16 False None Latest benchmark 25.01991402800013
deeplake_cpp image/mnist local 60000 training 4 16 False 2 Latest benchmark 52.74809954200009
deeplake_cpp image/mnist local 60000 inference 4 16 False None Latest benchmark 20.51788860799934
deeplake_python image/mnist local 60000 training 4 16 True 2 Latest benchmark 85.28131844899963
deeplake_python image/mnist local 60000 inference 4 16 True None Latest benchmark 37.209573673000705
deeplake_cpp image/mnist local 60000 training 4 16 True 2 Latest benchmark 109.70756277300006
deeplake_cpp image/mnist local 60000 inference 4 16 True None Latest benchmark 23.834775706000073
deeplake_python image/mnist local 60000 training 8 16 False 2 Latest benchmark 90.97765988599986
deeplake_python image/mnist local 60000 inference 8 16 False None Latest benchmark 39.816607435999686
deeplake_cpp image/mnist local 60000 training 8 16 False 2 Latest benchmark 54.71438044500064
deeplake_cpp image/mnist local 60000 inference 8 16 False None Latest benchmark 21.381966859000386
deeplake_python image/mnist local 60000 training 8 16 True 2 Latest benchmark 115.25972695699966
deeplake_python image/mnist local 60000 inference 8 16 True None Latest benchmark 52.5977779320001
deeplake_cpp image/mnist local 60000 training 8 16 True 2 Latest benchmark 111.36036478999995
deeplake_cpp image/mnist local 60000 inference 8 16 True None Latest benchmark 24.611332791000677
deeplake_python image/mnist local 60000 training 16 16 False 2 Latest benchmark 153.6414474510002
deeplake_python image/mnist local 60000 inference 16 16 False None Latest benchmark 70.66440681400036
deeplake_cpp image/mnist local 60000 training 16 16 False 2 Latest benchmark 58.806783181000355
deeplake_cpp image/mnist local 60000 inference 16 16 False None Latest benchmark 23.497605657999884
deeplake_python image/mnist local 60000 training 16 16 True 2 Latest benchmark 177.29381167500014
deeplake_python image/mnist local 60000 inference 16 16 True None Latest benchmark 82.97184543800086
deeplake_cpp image/mnist local 60000 training 16 16 True 2 Latest benchmark 114.18728806699983
deeplake_cpp image/mnist local 60000 inference 16 16 True None Latest benchmark 26.859680356000354
deeplake image/mnist s3 60000 ingest_parallel 16 None None None Latest benchmark 68.71801992699966
deeplake_python image/mnist s3 60000 training 4 4 False 2 Latest benchmark 20.270067161999577

@tatevikh tatevikh merged commit 00abcf9 into main Apr 18, 2023
10 of 16 checks passed
@tatevikh tatevikh deleted the devops/benchmarks branch April 18, 2023 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@artgish @tatevikh