Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timing benchmark tweaks and fixes #15022

Merged
merged 6 commits into from Apr 15, 2024
Merged

Conversation

cameel
Copy link
Member

@cameel cameel commented Apr 12, 2024

This is a bunch of independent tweaks to the test/benchmarks/ runner script to make using it for evaluating optimizer sequences a little less tedious. Each could really be a separate small PR.

  • The script would not continue when legacy compilation failed with "Stack too deep".
  • Bytecode size is a useful additional data point that we can easily get here.
  • Putting the values manually in a table gets old quickly. Now the script prints a table I can just copy paste into a comment on Github.
  • The lighter tests now run first so you can start seeing results immediately. Also, now the script does not wait with printing legacy results until IR finishes.
  • solc binary is now a parameter so that I can easily use system-wide solc. For testing sequences I find it easier to use the release binary and just give it different --yul-optimizations.

@cameel cameel self-assigned this Apr 12, 2024
@cameel
Copy link
Member Author

cameel commented Apr 12, 2024

Before

=======================================================
            chains.sol
-------------------------------------------------------
legacy pipeline took 0.96 seconds to execute.
via-ir pipeline took 69.23 seconds to execute.
=======================================================
=======================================================
            OptimizorClub.sol
-------------------------------------------------------
legacy pipeline took Command exited with non-zero status 1
2.78 seconds to execute.
via-ir pipeline took 16.33 seconds to execute.
=======================================================
=======================================================
            verifier.sol
-------------------------------------------------------
legacy pipeline took 0.74 seconds to execute.
via-ir pipeline took 3.21 seconds to execute.
=======================================================

=======================================================
Warnings and errors generated during run:
=======================================================

After

| File                 | Pipeline | Bytecode size | Time     | Exit code |
|----------------------|----------|--------------:|---------:|----------:|
| `verifier.sol`       | legacy   |    4983 bytes |   0.79 s |         0 |
| `verifier.sol`       | via-ir   |    4408 bytes |   3.51 s |         0 |
| `OptimizorClub.sol`  | legacy   |       0 bytes |   2.97 s |         1 |
| `OptimizorClub.sol`  | via-ir   |   22427 bytes |  16.99 s |         0 |
| `chains.sol`         | legacy   |    5911 bytes |   0.88 s |         0 |
| `chains.sol`         | via-ir   |   21443 bytes |  66.46 s |         0 |

=======================================================
Warnings and errors generated during run:
=======================================================

Formatted

File Pipeline Bytecode size Time Exit code
verifier.sol legacy 4983 bytes 0.79 s 0
verifier.sol via-ir 4408 bytes 3.51 s 0
OptimizorClub.sol legacy 0 bytes 2.97 s 1
OptimizorClub.sol via-ir 22427 bytes 16.99 s 0
chains.sol legacy 5911 bytes 0.88 s 0
chains.sol via-ir 21443 bytes 66.46 s 0

Copy link
Member

@r0qs r0qs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! And much better than the previous output of the script :)
I just have a small nitpick to mention.

test/benchmarks/run.sh Show resolved Hide resolved
@cameel cameel force-pushed the timing-benchmark-tweaks-and-fixes branch from c211ab7 to e919327 Compare April 15, 2024 10:46
@cameel cameel force-pushed the timing-benchmark-tweaks-and-fixes branch from e919327 to 6b83a43 Compare April 15, 2024 15:00
@cameel cameel enabled auto-merge April 15, 2024 15:00
@cameel cameel merged commit b5eb800 into develop Apr 15, 2024
71 of 72 checks passed
@cameel cameel deleted the timing-benchmark-tweaks-and-fixes branch April 15, 2024 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants