Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Actual Report for Reference Setup, Performance, Scalability, and Sizing Guidelines #7905

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Actual Report for Reference Setup, Performance, Scalability, and Sizing Guidelines #7905

wants to merge 2 commits into from
+4,302 −0

Conversation

PhanLe1010
Copy link
Contributor

@PhanLe1010 PhanLe1010 force-pushed the 2598-report branch 2 times, most recently from 60e0b80 to 83a86aa Compare February 9, 2024 00:56
@PhanLe1010 PhanLe1010 mentioned this pull request Feb 9, 2024
Closed
@PhanLe1010
Copy link
Contributor Author

Hi team, I am starting pushing report continuously starting with Public Cloud - Medium Node Spec report. Hope that we finish this one soon.

@PhanLe1010 PhanLe1010 force-pushed the 2598-report branch 4 times, most recently from 1fe5d44 to d57e948 Compare February 9, 2024 05:29
ejweber
ejweber reviewed Feb 9, 2024
View reviewed changes
Copy link
Contributor

@ejweber ejweber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be good to add a section of "whys" at some point. For example, why 1TiB? Why those particular replica scheduling settings? This can help us reason about the testing in the future.

...ference-setup-performance-scalability-and-sizing-guidelines/public-cloud/medium-node-spec.md Outdated Show resolved Hide resolved
...ference-setup-performance-scalability-and-sizing-guidelines/public-cloud/medium-node-spec.md Outdated Show resolved Hide resolved
@PhanLe1010
Copy link
Contributor Author

PhanLe1010 commented Feb 9, 2024 •
edited

It may be good to add a section of "whys" at some point. For example, why 1TiB? Why those particular replica scheduling settings? This can help us reason about the testing in the future.

Good idea, I will add the explanation:

  • 1TB is purely due to cost issue. We gonna have many nodes and each nodes will have 1TB disk.
  • Storage Minimal Available Percentage setting 10%. As we are using dedicated disk, we don't need big reserve storage as mentioned in best practice https://longhorn.io/docs/1.6.0/best-practices/#minimal-available-storage-and-over-provisioning
  • Storage Over Provisioning Percentage setting: 110%:
    • We are planning to fill 15GB for each 20GB volume
    • If we schedule maximum amount, it would be 1200 Gib and actual usage will be (15/20)*1100 = 825. This leaves 100GiB as 10% Storage Minimal Available Percentage setting plus some filesystem space overhead

@PhanLe1010 PhanLe1010 force-pushed the 2598-report branch 19 times, most recently from dd2c03e to cf26a63 Compare February 13, 2024 18:53
@PhanLe1010 PhanLe1010 force-pushed the 2598-report branch 8 times, most recently from fe6d2c9 to 17f0897 Compare March 14, 2024 21:15
@PhanLe1010
Copy link
Contributor Author

PhanLe1010 commented May 7, 2024 •
edited

Update:

I am benchmarking Longhorn control plan. Looks like Longhorn cannot scale pass 310 pods (with 310 volumes) in the medium spec cluster (1 control + 3 worker nodes, EC2 instance type: ec2 m5zn.2xlarge - 8vCPUs - 32GB RAM)

The error event on pending pods:

  Warning  FailedMount         12s (x5 over 9m16s)   kubelet                  Unable to attach or mount volumes: unmounted volumes=[www], unattached volumes=[www], failed to process volumes=[]: timed out waiting for the condition

screencapture-143-198-70-16-k8s-clusters-c-m-6dhqt4x7-api-v1-namespaces-cattle-monitoring-system-services-http-rancher-monitoring-grafana-80-proxy-d-OUbS8NYIk-longhorn-control-plan-scale-test-2024-05-06-18_10_40

@PhanLe1010
Copy link
Contributor Author

The issue #7905 (comment) is a known issue #7919

@PhanLe1010 PhanLe1010 force-pushed the 2598-report branch 7 times, most recently from 022e585 to bc1580e Compare May 10, 2024 01:03
@PhanLe1010 PhanLe1010 marked this pull request as ready for review May 10, 2024 01:03
@PhanLe1010 PhanLe1010 requested a review from a team as a code owner May 10, 2024 01:03
@PhanLe1010
Copy link
Contributor Author

PhanLe1010 commented May 10, 2024 •
edited

This PR is ready for review.

I finished the reported for the cloud medium spec cluster (the last information about max volume size is coming soon). As discussed in the US team meeting, I decided to leave the report for cloud big spec and baremetal cluster for the next release and focus on 1.7.0 backlog. This is the first version of the report, I am doing it manually. For the next version, I will try to have automation to speed up the process.

Thank you for all the helpful feedbacks!

cc @ejweber @james-munson @shuo-wu @derekbit @innobead

ejweber
ejweber reviewed May 10, 2024
View reviewed changes
Copy link
Contributor

@ejweber ejweber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the nitpicky review! I don't think @jillian-maroket will be handling this one, so I paid a bit more attention to it grammatically than I normally would. From a technical perspective, things are looking quite good.

I only made it about halfway through so far. There aren't any dealbreakers for me yet, so please feel free to consider my comments/suggestions, either or adopt them or not, and directly resolve the conversation.

examples/reference-setup-performance-scalability-and-sizing-guidelines/README.md Outdated Show resolved Hide resolved
examples/reference-setup-performance-scalability-and-sizing-guidelines/README.md Outdated Show resolved Hide resolved
...ference-setup-performance-scalability-and-sizing-guidelines/public-cloud/medium-node-spec.md Outdated Show resolved Hide resolved
...ference-setup-performance-scalability-and-sizing-guidelines/public-cloud/medium-node-spec.md Outdated Show resolved Hide resolved
...ference-setup-performance-scalability-and-sizing-guidelines/public-cloud/medium-node-spec.md Outdated
**Comment:**
* We choose 10000 for EBS disk's IOPs simply because it is a middle value between minimum value 3000 and maximum value 16000 of the gp3 EBS disk
* We choose 360MiB/s for EBS disk's bandwidth because the m5zn.2xlarge EC2 instance has EBS bandwidth 396.25 MiB/s.
If we choose a bigger value than 396.25 MiB/s for EBS disk's bandwidth, the ec2 instance would not be able to push EBS disk to that value.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: Don't bother if it's too difficult or annoying, but links to where the reader can find this information might be useful.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I decided to keep the info here since there isn't a single page which explain this behavior. It would be multiple links if we decided to put the links here

...ference-setup-performance-scalability-and-sizing-guidelines/public-cloud/medium-node-spec.md Outdated Show resolved Hide resolved
...ference-setup-performance-scalability-and-sizing-guidelines/public-cloud/medium-node-spec.md Outdated

> Result:
> * Each Kbench pod is able to achieve 386 MiB/s random read bandwidth on its Longhorn volume
> * Total random read bandwidth can be achieved by all 3 Longhorn volumes is 1158
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
> * Total random read bandwidth can be achieved by all 3 Longhorn volumes is 1158
> * Total sequential read bandwidth can be achieved by all 3 Longhorn volumes is 1158

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should be random

...ference-setup-performance-scalability-and-sizing-guidelines/public-cloud/medium-node-spec.md Outdated
Scaling workload from 3 to 6, then 6 to 9, then 9 to 12, then 12 to 15

> Result:
> * At 6 pods, the average random read bandwidth per Longhorn volume is 196 MiB/s. Total random bandwidth is 1176 MiB/s
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
> * At 6 pods, the average random read bandwidth per Longhorn volume is 196 MiB/s. Total random bandwidth is 1176 MiB/s
> * At 6 pods, the average sequential read bandwidth per Longhorn volume is 196 MiB/s. Total random bandwidth is 1176 MiB/s

And below.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should be random

...ference-setup-performance-scalability-and-sizing-guidelines/public-cloud/medium-node-spec.md Outdated Show resolved Hide resolved
...ference-setup-performance-scalability-and-sizing-guidelines/public-cloud/medium-node-spec.md Outdated



### Random Read Bandwidth - Stress Tests
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stopped here for now. Will continue later.

@PhanLe1010
Report for Reference Setup, Performance, Scalability, and Sizing Guid…
a315e38 
…elines

longhorn-2598

Signed-off-by: Phan Le <phan.le@suse.com>
@PhanLe1010
Copy link
Contributor Author

Thanks @ejweber I resolved most of the comments. Looking forward to your next review

@PhanLe1010
Fixed Eric's review comments
322fcfa 
Signed-off-by: Phan Le <phan.le@suse.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ejweber ejweber ejweber left review comments

@shuo-wu shuo-wu Awaiting requested review from shuo-wu

@derekbit derekbit Awaiting requested review from derekbit

@innobead innobead Awaiting requested review from innobead

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@PhanLe1010 @yasker @ejweber