Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor: restructure to tests and CI workflow to reduce CI time #5951

Merged
merged 28 commits into from
Sep 22, 2023
Merged

refactor: restructure to tests and CI workflow to reduce CI time #5951

merged 28 commits into from
Sep 22, 2023

Conversation

nazarhussain
Copy link
Contributor

@nazarhussain nazarhussain commented Sep 12, 2023 •
edited

Motivation

Reduce the CI time

Description

  • Reduce the CI time by cutting every bit from here and there.
  • Restructure the Test CI pipeline to avoid duplicate work.
  • Run all tests with transpileOnly option to save type checking time as we have check-types task.
  • Restructure the spec tests to run in parallel mode
image

Steps to test or reproduce

Run all tests.

===== NOTE =====

For future reference because of the transpileOnly option we need to enable isolateModule tsconfig. Which requires you to take care of two things in the code for future:

  1. If you re-export then export the values and types independently with export type keyword.
  2. Don't use const enum, rather just enum.

@nazarhussain nazarhussain self-assigned this Sep 12, 2023
@nazarhussain nazarhussain changed the title test: E2E debugging test: e2e debugging Sep 12, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Sep 12, 2023 •
edited

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 678bcdf Previous: 9dae25f Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 530.42 us/op 534.96 us/op 0.99
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 90.925 us/op 51.773 us/op 1.76
BLS verify - blst-native 1.3573 ms/op 1.1391 ms/op 1.19
BLS verifyMultipleSignatures 3 - blst-native 3.0434 ms/op 2.4188 ms/op 1.26
BLS verifyMultipleSignatures 8 - blst-native 6.3302 ms/op 5.3656 ms/op 1.18
BLS verifyMultipleSignatures 32 - blst-native 22.555 ms/op 19.735 ms/op 1.14
BLS verifyMultipleSignatures 64 - blst-native 45.008 ms/op 39.296 ms/op 1.15
BLS verifyMultipleSignatures 128 - blst-native 91.118 ms/op 78.399 ms/op 1.16
BLS deserializing 10000 signatures 927.85 ms/op 812.86 ms/op 1.14
BLS deserializing 100000 signatures 9.5072 s/op 8.0060 s/op 1.19
BLS verifyMultipleSignatures - same message - 3 - blst-native 1.3942 ms/op 1.1195 ms/op 1.25
BLS verifyMultipleSignatures - same message - 8 - blst-native 1.5663 ms/op 1.2894 ms/op 1.21
BLS verifyMultipleSignatures - same message - 32 - blst-native 2.4467 ms/op 2.0611 ms/op 1.19
BLS verifyMultipleSignatures - same message - 64 - blst-native 3.5916 ms/op 3.0422 ms/op 1.18
BLS verifyMultipleSignatures - same message - 128 - blst-native 5.8285 ms/op 5.0162 ms/op 1.16
BLS aggregatePubkeys 32 - blst-native 27.003 us/op 21.830 us/op 1.24
BLS aggregatePubkeys 128 - blst-native 104.31 us/op 85.934 us/op 1.21
getAttestationsForBlock 44.349 ms/op 26.795 ms/op 1.66
isKnown best case - 1 super set check 516.00 ns/op 223.00 ns/op 2.31
isKnown normal case - 2 super set checks 407.00 ns/op 213.00 ns/op 1.91
isKnown worse case - 16 super set checks 453.00 ns/op 212.00 ns/op 2.14
CheckpointStateCache - add get delete 6.1040 us/op 3.5090 us/op 1.74
validate api signedAggregateAndProof - struct 2.8999 ms/op 2.4769 ms/op 1.17
validate gossip signedAggregateAndProof - struct 2.9840 ms/op 2.4685 ms/op 1.21
validate gossip attestation - vc 640000 1.4432 ms/op 1.1715 ms/op 1.23
batch validate gossip attestation - vc 640000 - chunk 32 176.36 us/op 138.08 us/op 1.28
batch validate gossip attestation - vc 640000 - chunk 64 155.28 us/op 125.52 us/op 1.24
batch validate gossip attestation - vc 640000 - chunk 128 144.71 us/op 120.14 us/op 1.20
batch validate gossip attestation - vc 640000 - chunk 256 142.45 us/op 133.34 us/op 1.07
pickEth1Vote - no votes 1.3214 ms/op 1.1863 ms/op 1.11
pickEth1Vote - max votes 14.399 ms/op 14.983 ms/op 0.96
pickEth1Vote - Eth1Data hashTreeRoot value x2048 19.218 ms/op 17.003 ms/op 1.13
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 34.436 ms/op 17.271 ms/op 1.99
pickEth1Vote - Eth1Data fastSerialize value x2048 677.68 us/op 425.09 us/op 1.59
pickEth1Vote - Eth1Data fastSerialize tree x2048 6.1821 ms/op 8.8790 ms/op 0.70
bytes32 toHexString 534.00 ns/op 401.00 ns/op 1.33
bytes32 Buffer.toString(hex) 308.00 ns/op 210.00 ns/op 1.47
bytes32 Buffer.toString(hex) from Uint8Array 484.00 ns/op 321.00 ns/op 1.51
bytes32 Buffer.toString(hex) + 0x 320.00 ns/op 235.00 ns/op 1.36
Object access 1 prop 0.18000 ns/op 0.13800 ns/op 1.30
Map access 1 prop 0.16400 ns/op 0.10700 ns/op 1.53
Object get x1000 8.3140 ns/op 4.9170 ns/op 1.69
Map get x1000 0.69400 ns/op 0.48600 ns/op 1.43
Object set x1000 58.107 ns/op 28.080 ns/op 2.07
Map set x1000 45.904 ns/op 18.359 ns/op 2.50
Return object 10000 times 0.27520 ns/op 0.21760 ns/op 1.26
Throw Error 10000 times 4.3181 us/op 2.7695 us/op 1.56
fastMsgIdFn sha256 / 200 bytes 3.6360 us/op 1.9210 us/op 1.89
fastMsgIdFn h32 xxhash / 200 bytes 314.00 ns/op 221.00 ns/op 1.42
fastMsgIdFn h64 xxhash / 200 bytes 390.00 ns/op 243.00 ns/op 1.60
fastMsgIdFn sha256 / 1000 bytes 12.355 us/op 6.1270 us/op 2.02
fastMsgIdFn h32 xxhash / 1000 bytes 490.00 ns/op 370.00 ns/op 1.32
fastMsgIdFn h64 xxhash / 1000 bytes 447.00 ns/op 316.00 ns/op 1.41
fastMsgIdFn sha256 / 10000 bytes 110.77 us/op 54.339 us/op 2.04
fastMsgIdFn h32 xxhash / 10000 bytes 2.0870 us/op 1.7000 us/op 1.23
fastMsgIdFn h64 xxhash / 10000 bytes 1.4260 us/op 1.1200 us/op 1.27
send data - 1000 256B messages 21.386 ms/op 11.816 ms/op 1.81
send data - 1000 512B messages 28.461 ms/op 15.644 ms/op 1.82
send data - 1000 1024B messages 42.173 ms/op 24.787 ms/op 1.70
send data - 1000 1200B messages 30.483 ms/op 18.404 ms/op 1.66
send data - 1000 2048B messages 33.563 ms/op 23.405 ms/op 1.43
send data - 1000 4096B messages 35.029 ms/op 23.880 ms/op 1.47
send data - 1000 16384B messages 84.003 ms/op 54.052 ms/op 1.55
send data - 1000 65536B messages 319.18 ms/op 236.15 ms/op 1.35
enrSubnets - fastDeserialize 64 bits 1.5450 us/op 892.00 ns/op 1.73
enrSubnets - ssz BitVector 64 bits 507.00 ns/op 352.00 ns/op 1.44
enrSubnets - fastDeserialize 4 bits 210.00 ns/op 140.00 ns/op 1.50
enrSubnets - ssz BitVector 4 bits 519.00 ns/op 355.00 ns/op 1.46
prioritizePeers score -10:0 att 32-0.1 sync 2-0 108.76 us/op 70.382 us/op 1.55
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 145.06 us/op 90.738 us/op 1.60
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 185.23 us/op 122.93 us/op 1.51
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 353.25 us/op 217.89 us/op 1.62
prioritizePeers score 0:0 att 64-1 sync 4-1 405.60 us/op 233.96 us/op 1.73
array of 16000 items push then shift 1.8290 us/op 1.2766 us/op 1.43
LinkedList of 16000 items push then shift 10.423 ns/op 9.4180 ns/op 1.11
array of 16000 items push then pop 99.800 ns/op 94.076 ns/op 1.06
LinkedList of 16000 items push then pop 9.3730 ns/op 6.2070 ns/op 1.51
array of 24000 items push then shift 2.7264 us/op 2.1061 us/op 1.29
LinkedList of 24000 items push then shift 10.529 ns/op 6.9430 ns/op 1.52
array of 24000 items push then pop 131.97 ns/op 122.54 ns/op 1.08
LinkedList of 24000 items push then pop 9.0340 ns/op 6.4850 ns/op 1.39
intersect bitArray bitLen 8 7.0030 ns/op 6.6400 ns/op 1.05
intersect array and set length 8 58.745 ns/op 41.778 ns/op 1.41
intersect bitArray bitLen 128 33.343 ns/op 26.299 ns/op 1.27
intersect array and set length 128 829.94 ns/op 609.03 ns/op 1.36
bitArray.getTrueBitIndexes() bitLen 128 1.5520 us/op 1.2620 us/op 1.23
bitArray.getTrueBitIndexes() bitLen 248 2.6060 us/op 2.1990 us/op 1.19
bitArray.getTrueBitIndexes() bitLen 512 5.2770 us/op 3.8630 us/op 1.37
Buffer.concat 32 items 964.00 ns/op 761.00 ns/op 1.27
Uint8Array.set 32 items 1.7960 us/op 1.7670 us/op 1.02
Set add up to 64 items then delete first 4.5941 us/op 1.7279 us/op 2.66
OrderedSet add up to 64 items then delete first 5.4988 us/op 2.6265 us/op 2.09
Set add up to 64 items then delete last 4.8933 us/op 1.9922 us/op 2.46
OrderedSet add up to 64 items then delete last 6.6515 us/op 3.2016 us/op 2.08
Set add up to 64 items then delete middle 4.7954 us/op 2.2906 us/op 2.09
OrderedSet add up to 64 items then delete middle 7.5791 us/op 4.7460 us/op 1.60
Set add up to 128 items then delete first 9.9191 us/op 4.6894 us/op 2.12
OrderedSet add up to 128 items then delete first 12.283 us/op 7.8633 us/op 1.56
Set add up to 128 items then delete last 9.6257 us/op 4.4110 us/op 2.18
OrderedSet add up to 128 items then delete last 12.792 us/op 7.0295 us/op 1.82
Set add up to 128 items then delete middle 9.9750 us/op 4.9871 us/op 2.00
OrderedSet add up to 128 items then delete middle 18.843 us/op 11.896 us/op 1.58
Set add up to 256 items then delete first 20.685 us/op 9.2175 us/op 2.24
OrderedSet add up to 256 items then delete first 27.263 us/op 13.094 us/op 2.08
Set add up to 256 items then delete last 19.825 us/op 7.6799 us/op 2.58
OrderedSet add up to 256 items then delete last 27.513 us/op 11.295 us/op 2.44
Set add up to 256 items then delete middle 19.721 us/op 7.3137 us/op 2.70
OrderedSet add up to 256 items then delete middle 49.724 us/op 31.463 us/op 1.58
transfer serialized Status (84 B) 1.8410 us/op 1.4330 us/op 1.28
copy serialized Status (84 B) 1.5420 us/op 1.2900 us/op 1.20
transfer serialized SignedVoluntaryExit (112 B) 2.1070 us/op 1.5950 us/op 1.32
copy serialized SignedVoluntaryExit (112 B) 1.6940 us/op 1.3980 us/op 1.21
transfer serialized ProposerSlashing (416 B) 2.2630 us/op 1.9310 us/op 1.17
copy serialized ProposerSlashing (416 B) 2.2410 us/op 1.8170 us/op 1.23
transfer serialized Attestation (485 B) 2.2340 us/op 1.8900 us/op 1.18
copy serialized Attestation (485 B) 2.2120 us/op 2.0260 us/op 1.09
transfer serialized AttesterSlashing (33232 B) 2.3610 us/op 2.2270 us/op 1.06
copy serialized AttesterSlashing (33232 B) 6.5480 us/op 3.9530 us/op 1.66
transfer serialized Small SignedBeaconBlock (128000 B) 2.7710 us/op 1.9170 us/op 1.45
copy serialized Small SignedBeaconBlock (128000 B) 14.730 us/op 7.7500 us/op 1.90
transfer serialized Avg SignedBeaconBlock (200000 B) 2.9870 us/op 1.5800 us/op 1.89
copy serialized Avg SignedBeaconBlock (200000 B) 19.276 us/op 10.228 us/op 1.88
transfer serialized BlobsSidecar (524380 B) 3.0330 us/op 2.1090 us/op 1.44
copy serialized BlobsSidecar (524380 B) 92.927 us/op 75.880 us/op 1.22
transfer serialized Big SignedBeaconBlock (1000000 B) 3.1850 us/op 2.6020 us/op 1.22
copy serialized Big SignedBeaconBlock (1000000 B) 203.32 us/op 126.71 us/op 1.60
pass gossip attestations to forkchoice per slot 3.9808 ms/op 2.6534 ms/op 1.50
forkChoice updateHead vc 100000 bc 64 eq 0 710.83 us/op 610.78 us/op 1.16
forkChoice updateHead vc 600000 bc 64 eq 0 4.9802 ms/op 2.9835 ms/op 1.67
forkChoice updateHead vc 1000000 bc 64 eq 0 7.4142 ms/op 4.4434 ms/op 1.67
forkChoice updateHead vc 600000 bc 320 eq 0 4.4074 ms/op 2.7525 ms/op 1.60
forkChoice updateHead vc 600000 bc 1200 eq 0 4.9609 ms/op 2.8091 ms/op 1.77
forkChoice updateHead vc 600000 bc 7200 eq 0 6.3473 ms/op 3.5721 ms/op 1.78
forkChoice updateHead vc 600000 bc 64 eq 1000 12.167 ms/op 10.264 ms/op 1.19
forkChoice updateHead vc 600000 bc 64 eq 10000 12.495 ms/op 10.059 ms/op 1.24
forkChoice updateHead vc 600000 bc 64 eq 300000 17.113 ms/op 13.184 ms/op 1.30
computeDeltas 500000 validators 300 proto nodes 6.6007 ms/op 3.1264 ms/op 2.11
computeDeltas 500000 validators 1200 proto nodes 6.6823 ms/op 3.2184 ms/op 2.08
computeDeltas 500000 validators 7200 proto nodes 7.0764 ms/op 3.2524 ms/op 2.18
computeDeltas 750000 validators 300 proto nodes 10.413 ms/op 4.8328 ms/op 2.15
computeDeltas 750000 validators 1200 proto nodes 10.310 ms/op 4.8300 ms/op 2.13
computeDeltas 750000 validators 7200 proto nodes 10.512 ms/op 4.6671 ms/op 2.25
computeDeltas 1400000 validators 300 proto nodes 20.756 ms/op 8.9300 ms/op 2.32
computeDeltas 1400000 validators 1200 proto nodes 21.101 ms/op 8.8267 ms/op 2.39
computeDeltas 1400000 validators 7200 proto nodes 19.617 ms/op 8.8295 ms/op 2.22
computeDeltas 2100000 validators 300 proto nodes 31.321 ms/op 15.088 ms/op 2.08
computeDeltas 2100000 validators 1200 proto nodes 33.199 ms/op 14.543 ms/op 2.28
computeDeltas 2100000 validators 7200 proto nodes 30.331 ms/op 14.432 ms/op 2.10
computeProposerBoostScoreFromBalances 500000 validators 3.7632 ms/op 2.9537 ms/op 1.27
computeProposerBoostScoreFromBalances 750000 validators 3.6257 ms/op 2.8644 ms/op 1.27
computeProposerBoostScoreFromBalances 1400000 validators 3.3938 ms/op 2.8558 ms/op 1.19
computeProposerBoostScoreFromBalances 2100000 validators 3.3048 ms/op 2.8087 ms/op 1.18
altair processAttestation - 250000 vs - 7PWei normalcase 2.7082 ms/op 2.4225 ms/op 1.12
altair processAttestation - 250000 vs - 7PWei worstcase 3.1219 ms/op 2.2033 ms/op 1.42
altair processAttestation - setStatus - 1/6 committees join 169.40 us/op 105.11 us/op 1.61
altair processAttestation - setStatus - 1/3 committees join 329.99 us/op 200.50 us/op 1.65
altair processAttestation - setStatus - 1/2 committees join 445.14 us/op 301.65 us/op 1.48
altair processAttestation - setStatus - 2/3 committees join 558.98 us/op 381.03 us/op 1.47
altair processAttestation - setStatus - 4/5 committees join 780.43 us/op 498.07 us/op 1.57
altair processAttestation - setStatus - 100% committees join 903.93 us/op 593.97 us/op 1.52
altair processBlock - 250000 vs - 7PWei normalcase 6.9400 ms/op 5.1506 ms/op 1.35
altair processBlock - 250000 vs - 7PWei normalcase hashState 30.463 ms/op 22.078 ms/op 1.38
altair processBlock - 250000 vs - 7PWei worstcase 38.434 ms/op 31.736 ms/op 1.21
altair processBlock - 250000 vs - 7PWei worstcase hashState 91.090 ms/op 74.262 ms/op 1.23
phase0 processBlock - 250000 vs - 7PWei normalcase 2.4426 ms/op 2.2119 ms/op 1.10
phase0 processBlock - 250000 vs - 7PWei worstcase 34.150 ms/op 27.314 ms/op 1.25
altair processEth1Data - 250000 vs - 7PWei normalcase 517.59 us/op 417.90 us/op 1.24
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 15.374 us/op 7.3090 us/op 2.10
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 69.090 us/op 49.157 us/op 1.41
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 18.659 us/op 13.435 us/op 1.39
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 13.949 us/op 6.8790 us/op 2.03
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 205.07 us/op 108.55 us/op 1.89
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 1.3498 ms/op 839.04 us/op 1.61
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 1.9221 ms/op 1.1253 ms/op 1.71
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 1.8110 ms/op 1.0111 ms/op 1.79
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 3.7348 ms/op 2.8004 ms/op 1.33
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 2.3176 ms/op 1.4960 ms/op 1.55
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 5.1208 ms/op 4.3036 ms/op 1.19
Tree 40 250000 create 323.26 ms/op 232.40 ms/op 1.39
Tree 40 250000 get(125000) 194.71 ns/op 118.35 ns/op 1.65
Tree 40 250000 set(125000) 943.81 ns/op 666.98 ns/op 1.42
Tree 40 250000 toArray() 19.105 ms/op 21.109 ms/op 0.91
Tree 40 250000 iterate all - toArray() + loop 17.455 ms/op 24.004 ms/op 0.73
Tree 40 250000 iterate all - get(i) 64.222 ms/op 57.856 ms/op 1.11
MutableVector 250000 create 11.221 ms/op 9.1075 ms/op 1.23
MutableVector 250000 get(125000) 6.4820 ns/op 6.0170 ns/op 1.08
MutableVector 250000 set(125000) 240.10 ns/op 235.81 ns/op 1.02
MutableVector 250000 toArray() 3.0036 ms/op 2.5841 ms/op 1.16
MutableVector 250000 iterate all - toArray() + loop 3.0519 ms/op 2.3417 ms/op 1.30
MutableVector 250000 iterate all - get(i) 1.5450 ms/op 1.6387 ms/op 0.94
Array 250000 create 2.6476 ms/op 2.4975 ms/op 1.06
Array 250000 clone - spread 1.0511 ms/op 857.09 us/op 1.23
Array 250000 get(125000) 0.52900 ns/op 0.43500 ns/op 1.22
Array 250000 set(125000) 0.60400 ns/op 0.50200 ns/op 1.20
Array 250000 iterate all - loop 82.107 us/op 79.829 us/op 1.03
effectiveBalanceIncrements clone Uint8Array 300000 26.205 us/op 23.739 us/op 1.10
effectiveBalanceIncrements clone MutableVector 300000 300.00 ns/op 203.00 ns/op 1.48
effectiveBalanceIncrements rw all Uint8Array 300000 180.04 us/op 170.45 us/op 1.06
effectiveBalanceIncrements rw all MutableVector 300000 85.068 ms/op 59.620 ms/op 1.43
phase0 afterProcessEpoch - 250000 vs - 7PWei 114.85 ms/op 76.675 ms/op 1.50
phase0 beforeProcessEpoch - 250000 vs - 7PWei 37.693 ms/op 35.438 ms/op 1.06
altair processEpoch - mainnet_e81889 629.34 ms/op 363.98 ms/op 1.73
mainnet_e81889 - altair beforeProcessEpoch 102.88 ms/op 54.401 ms/op 1.89
mainnet_e81889 - altair processJustificationAndFinalization 35.193 us/op 6.9710 us/op 5.05
mainnet_e81889 - altair processInactivityUpdates 13.336 ms/op 6.6337 ms/op 2.01
mainnet_e81889 - altair processRewardsAndPenalties 68.135 ms/op 56.060 ms/op 1.22
mainnet_e81889 - altair processRegistryUpdates 3.0730 us/op 2.0280 us/op 1.52
mainnet_e81889 - altair processSlashings 577.00 ns/op 359.00 ns/op 1.61
mainnet_e81889 - altair processEth1DataReset 1.6190 us/op 308.00 ns/op 5.26
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.6625 ms/op 1.0212 ms/op 2.61
mainnet_e81889 - altair processSlashingsReset 9.7380 us/op 3.5480 us/op 2.74
mainnet_e81889 - altair processRandaoMixesReset 6.9290 us/op 6.8720 us/op 1.01
mainnet_e81889 - altair processHistoricalRootsUpdate 994.00 ns/op 672.00 ns/op 1.48
mainnet_e81889 - altair processParticipationFlagUpdates 2.8210 us/op 2.3100 us/op 1.22
mainnet_e81889 - altair processSyncCommitteeUpdates 723.00 ns/op 828.00 ns/op 0.87
mainnet_e81889 - altair afterProcessEpoch 130.50 ms/op 95.210 ms/op 1.37
capella processEpoch - mainnet_e217614 1.5005 s/op 1.6677 s/op 0.90
mainnet_e217614 - capella beforeProcessEpoch 250.55 ms/op 287.44 ms/op 0.87
mainnet_e217614 - capella processJustificationAndFinalization 20.452 us/op 10.358 us/op 1.97
mainnet_e217614 - capella processInactivityUpdates 21.445 ms/op 18.232 ms/op 1.18
mainnet_e217614 - capella processRewardsAndPenalties 269.79 ms/op 243.69 ms/op 1.11
mainnet_e217614 - capella processRegistryUpdates 22.075 us/op 8.4180 us/op 2.62
mainnet_e217614 - capella processSlashings 696.00 ns/op 303.00 ns/op 2.30
mainnet_e217614 - capella processEth1DataReset 601.00 ns/op 321.00 ns/op 1.87
mainnet_e217614 - capella processEffectiveBalanceUpdates 4.4670 ms/op 3.2569 ms/op 1.37
mainnet_e217614 - capella processSlashingsReset 4.7800 us/op 3.9290 us/op 1.22
mainnet_e217614 - capella processRandaoMixesReset 5.8130 us/op 2.1750 us/op 2.67
mainnet_e217614 - capella processHistoricalRootsUpdate 1.0350 us/op 303.00 ns/op 3.42
mainnet_e217614 - capella processParticipationFlagUpdates 3.1430 us/op 697.00 ns/op 4.51
mainnet_e217614 - capella afterProcessEpoch 313.11 ms/op 216.27 ms/op 1.45
phase0 processEpoch - mainnet_e58758 418.21 ms/op 330.51 ms/op 1.27
mainnet_e58758 - phase0 beforeProcessEpoch 115.57 ms/op 100.16 ms/op 1.15
mainnet_e58758 - phase0 processJustificationAndFinalization 16.649 us/op 7.7160 us/op 2.16
mainnet_e58758 - phase0 processRewardsAndPenalties 49.521 ms/op 47.782 ms/op 1.04
mainnet_e58758 - phase0 processRegistryUpdates 11.360 us/op 5.7020 us/op 1.99
mainnet_e58758 - phase0 processSlashings 562.00 ns/op 321.00 ns/op 1.75
mainnet_e58758 - phase0 processEth1DataReset 416.00 ns/op 437.00 ns/op 0.95
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.0260 ms/op 836.89 us/op 1.23
mainnet_e58758 - phase0 processSlashingsReset 2.5330 us/op 2.3310 us/op 1.09
mainnet_e58758 - phase0 processRandaoMixesReset 4.5710 us/op 4.9350 us/op 0.93
mainnet_e58758 - phase0 processHistoricalRootsUpdate 680.00 ns/op 521.00 ns/op 1.31
mainnet_e58758 - phase0 processParticipationRecordUpdates 3.8670 us/op 4.0780 us/op 0.95
mainnet_e58758 - phase0 afterProcessEpoch 98.405 ms/op 86.327 ms/op 1.14
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.2969 ms/op 1.3190 ms/op 0.98
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.4458 ms/op 1.2567 ms/op 1.15
altair processInactivityUpdates - 250000 normalcase 21.886 ms/op 16.189 ms/op 1.35
altair processInactivityUpdates - 250000 worstcase 22.921 ms/op 15.210 ms/op 1.51
phase0 processRegistryUpdates - 250000 normalcase 8.7410 us/op 11.141 us/op 0.78
phase0 processRegistryUpdates - 250000 badcase_full_deposits 373.35 us/op 284.00 us/op 1.31
phase0 processRegistryUpdates - 250000 worstcase 0.5 139.89 ms/op 103.77 ms/op 1.35
altair processRewardsAndPenalties - 250000 normalcase 60.902 ms/op 53.722 ms/op 1.13
altair processRewardsAndPenalties - 250000 worstcase 60.032 ms/op 61.615 ms/op 0.97
phase0 getAttestationDeltas - 250000 normalcase 7.7340 ms/op 7.3499 ms/op 1.05
phase0 getAttestationDeltas - 250000 worstcase 7.6589 ms/op 7.2117 ms/op 1.06
phase0 processSlashings - 250000 worstcase 2.3433 ms/op 1.7090 ms/op 1.37
altair processSyncCommitteeUpdates - 250000 146.48 ms/op 118.74 ms/op 1.23
BeaconState.hashTreeRoot - No change 322.00 ns/op 238.00 ns/op 1.35
BeaconState.hashTreeRoot - 1 full validator 134.29 us/op 150.91 us/op 0.89
BeaconState.hashTreeRoot - 32 full validator 1.5609 ms/op 1.7209 ms/op 0.91
BeaconState.hashTreeRoot - 512 full validator 13.711 ms/op 19.687 ms/op 0.70
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 145.31 us/op 225.79 us/op 0.64
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 2.5081 ms/op 2.3327 ms/op 1.08
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 32.739 ms/op 25.672 ms/op 1.28
BeaconState.hashTreeRoot - 1 balances 120.82 us/op 136.49 us/op 0.89
BeaconState.hashTreeRoot - 32 balances 1.0091 ms/op 811.28 us/op 1.24
BeaconState.hashTreeRoot - 512 balances 9.4072 ms/op 7.4329 ms/op 1.27
BeaconState.hashTreeRoot - 250000 balances 165.86 ms/op 171.07 ms/op 0.97
aggregationBits - 2048 els - zipIndexesInBitList 14.352 us/op 10.288 us/op 1.40
regular array get 100000 times 32.917 us/op 32.500 us/op 1.01
wrappedArray get 100000 times 32.910 us/op 31.621 us/op 1.04
arrayWithProxy get 100000 times 14.018 ms/op 11.401 ms/op 1.23
ssz.Root.equals 207.00 ns/op 151.00 ns/op 1.37
byteArrayEquals 212.00 ns/op 152.00 ns/op 1.39
shuffle list - 16384 els 6.8354 ms/op 4.8566 ms/op 1.41
shuffle list - 250000 els 100.56 ms/op 71.687 ms/op 1.40
processSlot - 1 slots 16.766 us/op 13.698 us/op 1.22
processSlot - 32 slots 3.6639 ms/op 4.3534 ms/op 0.84
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 53.106 ms/op 43.003 ms/op 1.23
getCommitteeAssignments - req 1 vs - 250000 vc 2.6253 ms/op 2.3126 ms/op 1.14
getCommitteeAssignments - req 100 vs - 250000 vc 3.8780 ms/op 3.4210 ms/op 1.13
getCommitteeAssignments - req 1000 vs - 250000 vc 4.1793 ms/op 3.6462 ms/op 1.15
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 4.6400 ns/op 3.6800 ns/op 1.26
state getBlockRootAtSlot - 250000 vs - 7PWei 797.38 ns/op 359.01 ns/op 2.22
computeProposers - vc 250000 9.6295 ms/op 6.4438 ms/op 1.49
computeEpochShuffling - vc 250000 106.57 ms/op 68.306 ms/op 1.56
getNextSyncCommittee - vc 250000 156.90 ms/op 116.91 ms/op 1.34
computeSigningRoot for AttestationData 23.029 us/op 24.245 us/op 0.95
hash AttestationData serialized data then Buffer.toString(base64) 2.3405 us/op 1.2431 us/op 1.88
toHexString serialized data 1.0695 us/op 812.34 ns/op 1.32
Buffer.toString(base64) 239.01 ns/op 143.47 ns/op 1.67

by benchmarkbot/action

@nazarhussain nazarhussain marked this pull request as ready for review September 20, 2023 16:46
@nazarhussain nazarhussain requested a review from a team as a code owner September 20, 2023 16:46
@nazarhussain nazarhussain changed the title test: e2e debugging refac: restructure to tests and CI workflow to reduce CI time Sep 20, 2023
@nazarhussain nazarhussain changed the title refac: restructure to tests and CI workflow to reduce CI time refactor: restructure to tests and CI workflow to reduce CI time Sep 20, 2023
@wemeetagain
Copy link
Member

If you re-export then export the values and types independently with export type keyword.
Don't use const enum, rather just enum.

Will we get a tsc error if we do these things?

@nazarhussain
Copy link
Contributor Author

If you re-export then export the values and types independently with export type keyword.
Don't use const enum, rather just enum.

Will we get a tsc error if we do these things?

Yes the config option isolateModule will cause errors to trigger if we export mixed stuff.

@wemeetagain wemeetagain enabled auto-merge (squash) September 21, 2023 15:33
matthewkeil
matthewkeil reviewed Sep 22, 2023
View reviewed changes
packages/beacon-node/test/e2e/chain/lightclient.test.ts
@@ -53,7 +54,7 @@ describe("chain / lightclient", function () {

// delay a bit so regular sync sees it's up to date and sync is completed from the beginning
// also delay to allow bls workers to be transpiled/initialized
const genesisSlotsDelay = 16;
const genesisSlotsDelay = 7;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice optimization! 🚀

matthewkeil
matthewkeil reviewed Sep 22, 2023
View reviewed changes
Copy link
Member

@matthewkeil matthewkeil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tend to think that all of the type import/export changes should be in a separate PR. Im guessing the intention is to reduce transpilation time but still feels a bit funny to have all those changes, and the const addition to the couple enums in here.

@nazarhussain
Copy link
Contributor Author

@matthewkeil To improve the CI time we have make the transpileOnly enabled, and to that we have to make these code changes.

@matthewkeil
Copy link
Member

matthewkeil commented Sep 22, 2023 •
edited

@nazarhussain for my knowledge, what are the changes like this doing?

specTestIterator(path.join(ethereumConsensusSpecsTests.outputDir, "tests", ACTIVE_PRESET), {
  fork: {type: RunnerType.default, fn: fork},
});

Like, why not leave them in packages/beacon-node/test/spec/presets/index.test.ts?

@nazarhussain
Copy link
Contributor Author

@nazarhussain for my knowledge, what are the changes like this doing?

specTestIterator(path.join(ethereumConsensusSpecsTests.outputDir, "tests", ACTIVE_PRESET), {
  fork: {type: RunnerType.default, fn: fork},
});

Earlier the spec tests were structured in a way that can't be parallized in mocha. Mocha run the tests suits in parallel not the test cases. So have to switch every domain to it's own test suite with following changes.

matthewkeil
matthewkeil approved these changes Sep 22, 2023
View reviewed changes
Copy link
Member

@matthewkeil matthewkeil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!! 🚀

@wemeetagain wemeetagain merged commit 4fd3d4d into unstable Sep 22, 2023
15 checks passed
@wemeetagain wemeetagain deleted the nh/e2e-debugging branch September 22, 2023 10:05
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.12.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nflaig nflaig nflaig left review comments

@matthewkeil matthewkeil matthewkeil approved these changes

@wemeetagain wemeetagain Awaiting requested review from wemeetagain

Assignees

@nazarhussain nazarhussain

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@nazarhussain @wemeetagain @matthewkeil @nflaig