Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(devops): Add example for spinning up performance testing VMs on Azure #4647

Merged
merged 9 commits into from May 17, 2024
Merged

feat(devops): Add example for spinning up performance testing VMs on Azure #4647

merged 9 commits into from May 17, 2024

Conversation

jamilbk
Copy link
Member

@jamilbk jamilbk commented Apr 17, 2024

Sets up boilerplate for Azure performance testing infra.

They have some really interesting high performance VMs that would be fitting for our use cases, including ones with RDMA-enabled Infiniband networking cards if we really want to go wild.

@vercel Vercel
Copy link

vercel bot commented Apr 17, 2024 •
edited

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
firezone ✅ Ready (Inspect) Visit Preview 💬 Add feedback May 16, 2024 5:01pm

@github-actions github-actions bot added the kind/chore Issues related to repository cleanup or maintenance label Apr 17, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 17, 2024 •
edited

Terraform Cloud Plan Output

Plan: 15 to add, 15 to change, 15 to destroy.

Terraform Cloud Plan

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 17, 2024 •
edited

Performance Test Results

TCP

Test Name Received/s Sent/s Retransmits
direct-tcp-client2server 240.2 MiB (+0%) 241.2 MiB (-0%) 528 (+84%)
direct-tcp-server2client 234.9 MiB (-4%) 236.1 MiB (-4%) 287 (-15%)
relayed-tcp-client2server 226.3 MiB (-0%) 227.2 MiB (-0%) 246 (+23%)
relayed-tcp-server2client 239.3 MiB (+1%) 240.0 MiB (+1%) 470 (+83%)

UDP

Test Name Total/s Jitter Lost
direct-udp-client2server 500.0 MiB (+0%) 0.01ms (-76%) 39.71% (-17%)
direct-udp-server2client 499.9 MiB (-0%) 0.03ms (+23%) 21.91% (+5%)
relayed-udp-client2server 500.0 MiB (-0%) 0.02ms (-34%) 52.22% (-7%)
relayed-udp-server2client 500.0 MiB (+0%) 0.05ms (+125%) 43.90% (+1%)

@jamilbk
Copy link
Member Author

jamilbk commented Apr 17, 2024

@conectado The NIC on this VM consistently gets over 1 Gbps:

adminuser@vm-westus2:~$ speedtest
Retrieving speedtest.net configuration...
Testing from Microsoft Azure (20.36.14.61)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Actual Broadband (The Dalles, OR) [239.46 km]: 21.291 ms
Testing download speed................................................................................
Download: 1471.64 Mbit/s
Testing upload speed......................................................................................................
Upload: 1279.35 Mbit/s

@conectado
Copy link
Collaborator

We should add some docs on how to use this.

I can do it once I figure it out.

thomaseizinger
thomaseizinger approved these changes Apr 25, 2024
View reviewed changes
Copy link
Member

@thomaseizinger thomaseizinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't tested this but LGTM

Would appreciate a series of commands for how to spin this up / down.

I am assuming the idea is to compile in the VM? Can we setup Rust and clone the repo as part of terraform?

terraform/environments/performance/main.tf
# Azure Credits
arm_subscription_id = "330ffa29-65cb-4bfd-9004-4a3b3c8647de"

# Generate these by following
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we have this in a README in this directory perhaps?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be in the variables.tf

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

AndrewDryga
AndrewDryga reviewed Apr 29, 2024
View reviewed changes
terraform/environments/performance/main.tf Outdated

admin_ssh_key {
username = "adminuser"
public_key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRCYOIJco1uI8CK1hsEvNL0SeQyL2wlnOBmuSPkIhffcyVcKIYWs8mJbtiUPK0GkOga9TV29KL8WkN2MD8G5zALjvW7XKXRYAj2mQ5MIO9n18yJD+1KNudJwsg0lVJgjhzwz7R7GGhN2FIivpwPboY4q4aJRT+fQuTXpwKjUWFixaqirEIsp6F5Ia6SKZPaV4AZ2MuSgawSxacO8GGibjFISBGpndnGOzKTgfmn5MZT1EcvQK3zD6dtjA2e0c/GjsYnZ4GgTTXa0BS8AMa4UhPFd3AZ+/xZRXe+2yJX1KWMvOkQeYS7zctu6xt7NMXZkV/aQIyG1AXEKpJB3imn3QEh22gYnKe0Qg81vkt2OIy/S+hUnJnPCfJviRtRjzdCpm7unJpNU0UqkW3L1wuOz3TUsL864efmvv9V+mn1d3HN1j8EJmz/FoIonSbT4op7hdiYvdbUof45rIXuyy22KAeY32fH6xZ41ViIaEsbqb4E4HpsH0YZErgxYKoOl9VPnU= jamil@firezone.dev"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we hardcode a public key? There is az ssh vm command that we can use instead of managing our pubkeys manually.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately we have to specify either a key or a password:

Screenshot 2024-04-29 at 10 52 22 AM

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can use az ssh vm in addition to the adminuser here

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, its weird they require it, using both sounds good then.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I was thinking this PR is more to just add an example that devs can use individually to spin up their own VMs for performance testing. I don't know if it's worth having a persistent environment for that -- we can tweak as necessary.

@jamilbk jamilbk changed the title chore(devops): Add example performance VM feat(devops): Add example for spinning up performance testing VMs on Azure Apr 29, 2024
@github-actions github-actions bot added the kind/feature New feature or request label Apr 29, 2024
@jamilbk
Copy link
Member Author

jamilbk commented Apr 29, 2024

I am assuming the idea is to compile in the VM? Can we setup Rust and clone the repo as part of terraform?

I think that's out of scope for this PR -- I was just adding this to demonstrate how to spin up some VMs on Azure using Terraform to test performance with.

Will have to think if it's worth adding the CI complexity to automatically performance-test in Azure in the pipeline... tbh I would like to avoid adding more external dependencies to CI if possible. Maybe it can be a manually-run workflow or something.

@jamilbk jamilbk added this pull request to the merge queue May 17, 2024
Merged via the queue into main with commit bf1abcf May 17, 2024
136 checks passed
@jamilbk jamilbk deleted the devops/perf-infra branch May 17, 2024 16:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@thomaseizinger thomaseizinger thomaseizinger approved these changes

@conectado conectado Awaiting requested review from conectado

@AndrewDryga AndrewDryga Awaiting requested review from AndrewDryga

Assignees
No one assigned
Labels
kind/chore Issues related to repository cleanup or maintenance kind/feature New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@jamilbk @conectado @AndrewDryga @thomaseizinger