Skip to content

Latest commit

 

History

History
174 lines (135 loc) · 8.47 KB

README.md

File metadata and controls

174 lines (135 loc) · 8.47 KB
Raw

Artipie performance tests materials

Tests were performed in Linux, Ubuntu 22.04 LTS x86_64, with local docker & openJDK installed from Ubuntu repository. Technically, WSL2 with Windows 11 hypervisor was used for local runs.

JMeter notes

  • Note that JMeter in Ubuntu 22.04 LTS Linux repository is very old and some modules are missing
  • Note that JMeter may go crazy when recording requests from localhost to localhost. One may need to use external IP.
  • JMeter HTTP proxy mode recording could be done with GUI only.
  • At the moment we use the following version of JMeter: https://dlcdn.apache.org/jmeter/binaries/apache-jmeter-5.5.tgz

JFR and debugging support

JFR recording for running artipie docker container could be accomplished via remote JMX connections. Both, artipie-docker.sh and monitoring/client open port 9999 for JMX connections to artipie container. Java debugger could be attached via 8000 port. JMX connections for JFR recording were tested with JDK Mission Control v8.2.1 for Linux and Windows (native docker inside WSL2).

Datasets

Files adapter tested by files data generated by JMeter inside uploading test. Files downloading test uses list of files stored in the test data directory (test-data/files-dyn/files-list.csv).

Running tests

1. Setup Ubuntu 22.04 environment with dependencies:

sudo apt install curl openjdk-17-jdk docker.io maven

2. Ensure docker runs correctly: docker run hello-world. Also check java: java -version

3. Provide symlink to Apache JMeter

cd benchmarks/loadtests
ln -s /my/path/to/apache-jmeter-5.5 .

4. Prepare maven repository test data

./prep-maven-real.sh

5. Prepare maven dynamic test data. See script for the details:

time ./prep-maven-dyn.py --groups 3 --artifacts 4 --version 5

6. Start Artipie container

time ./artipie-docker.sh 8081

7. Run one of the tests. For example:

time ./jmx-maven-ul.sh localhost 8081 30 maven-dyn

Test data structure

test-data
├── files-dyn                    # artipie files adapter dynamic data test
│   ├── files-list.csv           # list of file names for download testing
│   ├── download-files-csv.jmx
│   └── upload-files.jmx
├── maven-dyn                    # artipie maven adapter dynamic data test
│   ├── files-list.csv
│   └── repository               # generated by prep-maven-dyn.py
└── maven-real                   # artipie maven adapter real-artifacts data test
    ├── files-list.csv
    └── repository               # generated by prep-maven-real.sh

Test scripts

Most scripts rely on JMeter 5.5 and were tested inside Linux with native Docker & java installed. Maven tests support testing with real artifacts data (maven-real) and generated dataset with several configurable parameters of the artifacts (maven-dyn). Maven data must be prepared locally by provided scripts.

  • jmeterReport.sh allows to generate report files for given JMeter jmx file.
  • Dockerfile.client allows to run for recording conan install command with given Conan server URL and JMeter proxy URL
  • artipie-docker.sh run test docker instance of artipie
  • jmx-run.sh binary data test with generated random data of small/medium/laarge files
  • jmx-local.sh run local artipie + binary test (above), then stop artipie
  • prep-maven-real.sh downloads required maven artifact files for tests. Artifacts put in test-data/maven-real/repository/
  • prep-maven-dyn.py generates required maven artifact files for tests. Artifacts put in test-data/maven-dyn/repository/
  • jmx-files-maven-ul.sh upload maven artifacts test set to the files repository adapter (bintest repo)
  • jmx-files-maven-dl.sh download maven artifacts test set to the files repository adapter (bintest repo)
  • jmx-maven-ul.sh upload maven artifacts test set to the maven repo (maventest repo)
  • jmx-maven-dl.sh download maven artifacts test set from the maven repo (maventest repo)
  • jmx-files-ul.sh upload random binary data files, like jmx-run.sh
  • jmx-files-ul.sh download files by list, generated for upload script above
  • maven-repo-reset.sh reset & wipe data for maven repository via Artipie REST API, for tests
  • bintest-repo-reset.sh reset & wipe data for binary files repository via Artipie REST API, for tests
  • sync_perftests.sh script to download all remote + upload new local test results of JMeter tests
  • monitoring/* monitoring support for artipie, which consist of two parts, see README.md there.
  • conan/ some basic scripts for future conan adapter support

Maven test data generator parameters

Maven test data generator prep-maven-dyn.py requires all generator parameters provided for the command line. It relies on Maven mvn command for artifact publishing. Here's the usage examples:

./prep-maven-dyn.py --total-artifacts 100
./prep-maven-dyn.py --groups 3 --artifacts 4 --versions=5

All available generator parameters (available by --help key):

  --total-artifacts TOTAL_ARTIFACTS  Specify tatal artifacts count. May override groups/artifacts/versions count. (default: None)
  --groups GROUPS           Count of maven groups to generate (default: 3)
  --artifacts ARTIFACTS     Count of maven artifacts to generate (default: 4)
  --versions VERSIONS       Count of maven artifact (sub)versions to generate (default: None)
  --big-size BIG_SIZE       Size of "big" artifact type in bytes (default: 9000000)
  --medium-size MEDIUM_SIZE Size of "medium" artifact type in bytes (default: 192000)
  --small-size SMALL_SIZE   Size of "small" artifact type in bytes (default: 16384)
  --big-p BIG_P             Percent of "big" artifacts (jars) to generate (default: 5)
  --medium-p MEDIUM_P       Percent of "medium" sized artifacts. The rest will be "small" sized. (default: 80)

Performance data

Performance data for tested tags is commited to perftests_repo/perftests/<tag>/<testname> directory on https://central.artipie.com/artipie/benchmarks server. Here tag is pushed git tag in artipie/artipie project repository, and testname is testing script benchmarks/loadtests/<testname>.sh in master git branch.

Graphs data is in JMeter statistics.json format. Performance graphs could be generated for the set of collected test results, using provided perfplot.py. For example:

./perfplot.py ../perftests ./graphs

$ ls graphs/
jmx-files-maven-dl.png  jmx-files-maven-ul.png

Performance results currently stored on the central.artipie.com server. Command below is used by CI for downloading and uploading performance data.

time ./sync_perftests.sh https://central.artipie.com/artipie/benchmarks Login password

Useful JMeter metrics:

  • throughput - number of requests per second.
  • meanResTime - average response time, msec.
  • medianResTime - the median value (50th percentile) of response time, msec.
  • pct1ResTime - 90th percentile, response time, msec.
  • pct2ResTime - 95th percentile, response time, msec.
  • pct3ResTime - 99th percentile, response time, msec.
  • sampleCount - number of requests during test.
  • errorCount - count of errors during the test.

Testing commands for curl

curl -v -X PUT -d 'test data' http://localhost:8081/bintest/test.txt
curl -v -X PUT -T ./jmeter.log http://localhost:8081/bintest/jmeter.log
curl -v -H 'Accept: text/plain' http://localhost:8081/bintest/
curl -v -H 'Accept: text/plain' https://central.artipie.com/artipie/benchmarks/perftests_repo

Last uploading performance test

JFR

Last JFR recordings for performance tests could be downloaded by:

https://central.artipie.com/artipie/benchmarks/perftests_repo/jfr/artipie.last.jfr.tar.xz

Downloading maven artifacts via JMeter proxy

  1. Run Apache JMeter in GUI mode
  2. Open project, and add new element: Edit->Add->Non-test elements->HTTPS test script recorder
  3. For this element, set Global settings->Port to 8888 and press Start.
  4. Use command below to download artifacts via this proxy:
time mvn clean install -Dhttp.proxyHost=localhost -Dhttp.proxyPort=8888 -Dhttps.proxyHost=localhost -Dhttps.proxyPort=8888 -Dmaven.wagon.http.ssl.insecure=true -Dmaven.test.skip -Ddockerfile.skip=true

Test delays

Delays for testing purposes could be added to a target branch by cherry-picking the following commit: https://github.com/artipie/artipie/commits/ech-delayed-storage-patch It is rebased on the artipie master and passes artipie tests & qulice checks.