Skip to content

DataVaccinator/dv-vault_tests

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

DataVaccinator test and performance measurement scripts

This repository contains several scripts to verify the complete DataVaccinaor Vault API functionality. It also offers single and multiple instances performance measurement.

Run a protocol test (vaccinator_test.php)

The protocol test script needs the url (and port) of the service to test. Call it like this:

php vaccinator_test.php http://127.0.0.1:8080

The above will connect to localhost on port 8080.

php vaccinator_test.php https://service.mydomain.com

The above will connect using https connection to service.mydomain.com (port 443 by default).

All tests have to return like this (positive example result):

Get version and check availability:
- pass
NOTE: Server supports 'search' module. We will test this, too.
NOTE: Server supports 'publish' module. We will test this, too.

Testing invalid requests (wrong data, missing data etc):
- pass
- pass
- pass
- pass
- pass

Testing to add data:
NOTE: New user VID: 8111823e306f95b93f67559aa205b874
- pass

Testing to publish data:
- pass
- pass
NOTE: New published user VID: 22eb8014df566c4ccceb2c0ec8109785
- pass

Tests updating data:
- pass
- pass
- pass
- pass

Tests retrieving data:
- pass
- pass

Testing 'search' plugin functions:
- pass
- pass

Testing 'publish' plugin functions:
- pass
- pass

Cleanup vid's created:
Removing vid [8111823e306f95b93f67559aa205b874]... OK
Removing vid [22eb8014df566c4ccceb2c0ec8109785]... OK

Done

Run a speed test (perf_test.php)

This is an example call to measure the localhost system performance:

php perf_test.php 1 vaccinator http://localhost:8080 30 4

The above call will do a test with:

  • provider id 1

  • provider password vaccinator

  • system at localhost port 8080

  • running for 30 seconds

  • utilizing 4 instances

Output:

Start client 0...
Start client 1...
Start client 2...
Start client 3...
Connection to http://localhost:8080 is working.
Duration: 31 sec
Transactions: 209 (each create, retrieve and delete)
Operations/sec: 20.225806451613

Connection to http://localhost:8080 is working.
Duration: 31 sec
Transactions: 211 (each create, retrieve and delete)
Operations/sec: 20.41935483871

Connection to http://localhost:8080 is working.
Duration: 31 sec
Transactions: 224 (each create, retrieve and delete)
Operations/sec: 21.677419354839

Connection to http://localhost:8080 is working.
Duration: 31 sec
Transactions: 223 (each create, retrieve and delete)
Operations/sec: 21.58064516129

Done

Interpretation of the results

All the instances return their own metrics. To get the final metrics, you can add the values. In the above example, each instance did about 20 operations/sec. So the overall number of operations was 4 x ~20 = ~80 operations/sec. Doing the exact calculation, it was 83.91 operations/sec or 302K operations/hour.

ℹ️

The above example was generated on a virtual machine (VBox):

Operating System: KDE neon 5.19
Kernel Version: 5.4.0-72-generic
OS Type: 64-bit
Processors: 2 × Intel® Core™ i5-7600K CPU @ 3.80GHz
Memory: 5,8 GiB of RAM

This virtual machine was also running the connected single instance of Cockroach Database and the test script itself. ~50% of the system performance was utilized by Cockroach DB, ~20% by dv-vault executable.

Please note that a production system with 4 CPU cores and 16GB of memory would do much more!

For example, on a 2vCPU/4GB RAM system at a German hoster, we got up to 330 operations/sec (using 20 threads while the CockroachDB had two nodes, one local and one in Helsinki with 25ms latency).

Placeholder

You can use a placeholder with the following scheme to generate random numbers in your URL:

[min-max]

This will get replaced by a number between given min and max.

Example:

php perf_test.php 1 vaccinator https://dc[1-3].sirius.datavaccinator.com 10 4

The above will generate URLs with a random number between 1 and 3 like for dc1.sirius.datavaccinator.com or dc2.sirius.datavaccinator.com.

This allows you to test access to multiple datavaccinator instances, all connected to the same database cluster.