OpenVidu Load Test

WARNING: The Openvidu Load Test project is having some big changes in functionality and architecture, so the latest versions in the repository are not stable.

What you can do is clone the repository on the commit of one of the previous stable releases (or use the latest tags in the Releases):

• KMS if you want to use emulation of browsers based on KMS.
• Node if you want to use emulation of browsers based on Node webrtc (both versions work for real browsers).

After that, you will have to download the latest openvidu-browser (or the version of openvidu-browser you are using in your OpenVidu deployment) and add it to the directory at browser-emulator/public, and modify line 6 of browser-emulator/public/index.html to use the latest openvidu-browser.

This repository contains a distributed tool that allows you to perform a load test against an OpenVidu CE or an OpenVidu PRO deployment.

Take into account that you must to deploy OpenVidu platform before using this tool.

Table of Contents

1. Project Architecture
2. Usage instructions
3. Sample test execution
4. Analyze test results
5. Browser Emulator documentation

---

Project Architecture

• Browser-emulator: Worker service implemented in NodeJS and controlled with a REST protocol that capable of connecting to an OpenVidu session and sending and receiving WebRTC media using openvidu-browser library. It is able to start and launch Chrome/Firefox browsers using Selenium, emulating a fully real user.

• Loadtest Controller: Controller service in charge of the coordination of the browser-emulator workers. It reads the load test scenario from a file and control the browser-emulator workers to connect participants loading OpenVidu platform.

Usage Instructions

1. Launch workers

For testing locally

This instructions assume you are using a linux system. Windows and Mac can require some adaptation.

To start with the load test you will have to start a worker first and the execute the controller.

Then follow these steps:

• Clone this repository

git clone https://github.com/OpenVidu/openvidu-loadtest.git
cd openvidu-loadtest/browser-emulator

Now you can use Vagrant to create a virtual machine running the browser-emulator. Ensure you have Vagrant and VirtualBox installed on your system.

You can either choose an already made box or make a personalized one yourself.

• Already made box

You can use the default Vagrantfile to create a preconfigured browser-emulator VM. You can start it with vagrant up to launch it with the default parameters.

• Customizable Parameters in Vagrantfile

  • BOX: Box to use as base. You can choose one of our already made boxes or choose the path to a box made by youself. Defaults to a box with the latest OpenVidu 2 CE and Firefox installed.
  • MEMORY: Amount of memory (in MB) to allocate for the virtual machine. Default: 8192. (Note: if START_MEDIASERVER is true, OpenVidu requires at least 8GB of memory).
  • CPUS: Number of CPUs to allocate for the virtual machine. Default: 4. (Note: if START_MEDIASERVER is true, OpenVidu requires at least 2 CPUs).
  • VAGRANT_PROVIDER: Virtualization provider to use (e.g., 'virtualbox', 'vmware'). Default: 'virtualbox'.
  • NODES: How many virtual machines will be created. Each machine will be created with the name node[i], where i is the node number. For each node, the ports open will be 5000 + (i * 10) for the BrowserEmulator server, 5001 + (i * 10) for the WebSocket connection and 5900 + (i * 10) for the VNC connection, where i is the node number. For example, the first node (node1) will have open ports 5000, 5001 and 5900, node2 will have 5010, 5011 and 5911 and so on. Defaults to 1.

• Available boxes

Here are our already made boxes that you can use:

- ivchicano/browseremulator-ov-ff: Default box. Comes ready to use against OpenVidu 2, using Firefox as the browser.
- ivchicano/browseremulator-lk-ff: Comes ready to use against LiveKit, using Firefox as the browser.
- ivchicano/browseremulator-ov-ff-dev: Comes with the latest OpenVidu 2 CE and Firefox installed. Starts an OpenVidu 2 CE server in the same machine.
- ivchicano/browseremulator-lk-ff-dev: Comes with the latest LiveKit and Firefox installed. Starts a LiveKit server in the same machine.

• Start vagrant

To customize these parameters, you can set environment variables before running vagrant up. For example:

export BOX=ivchicano/browseremulator-lk-ff
vagrant up

• Personalized box

You can also construct your personalized box with the parameters you choose. You can start it with VAGRANT_VAGRANTFILE=./Vagrantfile_create_box vagrant up to launch it with the default parameters. You can then use this box as is or package it with VAGRANT_VAGRANTFILE=./Vagrantfile_create_box vagrant package to create a box file.

• Customizable Parameters in Vagrantfile

  • MEMORY: Amount of memory (in MB) to allocate for the virtual machine. Default: 8192. (Note: if START_MEDIASERVER is true, OpenVidu requires at least 8GB of memory).
  • CPUS: Number of CPUs to allocate for the virtual machine. Default: 4. (Note: if START_MEDIASERVER is true, OpenVidu requires at least 2 CPUs).
  • VAGRANT_PROVIDER: Virtualization provider to use (e.g., 'virtualbox', 'vmware'). Default: 'virtualbox'.
  • FIREFOX: Set to 'true' to use Firefox instead of Chrome for testing. Default is 'false'.
  • START_MEDIASERVER: Set to 'true' to start the media server (either the latest Openvidu 2 CE or LiveKit) during provisioning. Default is 'true'. OpenVidu note: with this deployment, the OpenVidu URL is ^https://localhost and the OpenVidu secret is vagrant. LiveKit note: (Experimental) with this deployment, LiveKit is deployed in dev mode, so the API Key is devkey and the API Secret is secret.
  • QOE: Set to 'true' to install all necessary dependencies to run the quality of experience (QoE) analysis scripts in the browser-emulator. Slower installation. Default is 'false'.
  • LIVEKIT: (Experimental) Set to 'true' to use LiveKit instead of OpenVidu. Default: 'false'.
  • NODES: How many virtual machines will be created. Each machine will be created with the name node[i], where i is the node number. For each node, the ports open will be 5000 + (i * 10) for the BrowserEmulator server, 5001 + (i * 10) for the WebSocket connection and 5900 + (i * 10) for the VNC connection, where i is the node number. For example, the first node (node1) will have open ports 5000, 5001 and 5900, node2 will have 5010, 5011 and 5911 and so on. Defaults to 1.

• (Not recommended) Install and run without Vagrant

You can also install and run browser-emulator without using Vagrant.

You will need to choose if you want the worker to launch Chrome or Firefox browsers, as well as if you want to run QoE analysis on the worker's recorded videos.

If you are unsure, choose the prepare_scripts/openvidu/prepare.sh script to perform the installation (installs the browser-emulator with Chrome), as follows:

cd openvidu-loadtest/browser-emulator/
sudo su
./prepare_scripts/openvidu/prepare.sh # Install script, use the one you want
npm run start:prod-none # Start script, use the one you want depending on your installation

These are all the options available:

Browser to use	QoE Analysis Ready	Install Script to run	npm run start command to run
Chrome	YES	./prepare_scripts/openvidu/prepare.sh	npm run start:prod-none
Chrome	NO	./prepare_scripts/openvidu/prepare_no_qoe.sh	npm run start:prod-none
Firefox	YES	./prepare_scripts/openvidu/prepare_firefox.sh	npm run start:prod-none-firefox
Firefox	NO	./prepare_scripts/openvidu/prepare_no_qoe_firefox.sh	npm run start:prod-none-firefox

Running options

By default, this worker is listening on port 5000 that you have to specify later in the controller configuration. If you want to run it on another port just add SERVER_PORT=port_number before npm run start: command:

SERVER_PORT=6000 npm run start:prod-none-firefox

For testing on AWS

If you want to launch the load tests in a production environment, you can do that with AWS.

The browser-emulator services will be launched (by the loadtest-controller) in EC2 instances with the aim of emulate real users connected to OpenVidu.

Requirements

• Create browser-emulator AMI:

  Before run the create an AMI, ensure that your AWS credentials are properly configured. You can check it running aws configure.

  Once aws credentials have been configured in your pc, it's time to create the browser-emulator AMI. For creating it, you only have to run the createAMI.sh script.

   cd ./browser-emulator/aws/
   ./createAMI.sh

  Defaults to creating the AMI:

  • In the us-east-1 AWS region.
  • Using Chrome as the browser for tests.
  • Not installing QoE analysis software required if doing QoE analysis in situ (generally not necessary).

  After AMI was created, you must set the AMI ID to load-test/src/main/resources/application.properties, in particular WORKER_AMI_ID property.

  You can change the default values of the script in the createAMI.sh script like this:

   cd ./browser-emulator/aws/
   ./createAMI.sh --region eu-west-1 --version ./EC2-browser-emulator-firefox.yml

  The following versions are available:

  • Chrome with no qoe analysis software preinstalled (default): ./EC2-browser-emulator-no-qoe.yml
  • Chrome with qoe analysis software preinstalled: ./EC2-browser-emulator.yml
  • Firefox with no qoe analysis software preinstalled: ./EC2-browser-emulator-firefox-no-qoe.yml
  • Firefox with qoe analysis software preinstalled: ./EC2-browser-emulator-firefox.yml

• Create a security group:

  The instance which will contain the browser-emulator service must have the 5000 (REST API) and 5001 (WebSocket) ports opened. For doing this, you will have to create a AWS Security Group (SG) with these requeriments.

  After SG was created, you must set the security group ID to load-test/src/main/resources/application.properties, in particular WORKER_SECURITY_GROUP_ID property.

The loadtest-controller will use the BrowserEmulator AMI (previously created) for create instances on demand. All you have to do is fill the AWS configuration in loadtest-controller properties besides the required parameters

2. Execute controller

In the machine you want to execute the controller you need to have Java 11 platform installed.

Then follow these steps:

1. Clone this repository

git clone https://github.com/OpenVidu/openvidu-loadtest.git

2. Configure Loadtest Controller

Running the following command under project root directory you will be able to edit the load-test/src/main/resources/application.properties which contains a list of configuration parameters that you will can customize.

nano loadtest-controller/src/main/resources/application.properties

In this file you will see:

Required parameters

# Load Test Parameters (Required)
OPENVIDU_URL=https://openvidu_pro_url
OPENVIDU_SECRET=openvidu_pro_secret
SESSION_NAME_PREFIX=LoadTestSession
USER_NAME_PREFIX =User
SECONDS_TO_WAIT_BETWEEN_PARTICIPANTS=1
SECONDS_TO_WAIT_BETWEEN_SESSIONS=2
SECONDS_TO_WAIT_BEFORE_TEST_FINISHED=10
SECONDS_TO_WAIT_BETWEEN_TEST_CASES=10

Development mode parameters (only for testing locally)

# For testing locally, fill it with the worker ip address: 195.166.0.0
WORKER_URL_LIST=

AWS parameters (only for testing on AWS)

General AWS parameters

# For testing on AWS

# For launching EC2 instances and allowing to workers upload recordings to S3 bucket
AWS_ACCESS_KEY=your-aws-key
AWS_SECRET_ACCESS_KEY=your-secret-key
WORKER_AMI_ID=ami-xxxxxxxxxxx
# We recommend c5 type. https://aws.amazon.com/ec2/instance-types/
WORKER_INSTANCE_TYPE=c5.xlarge
# By default, the browser-emulator service is listening on:
# 5000 (REST API)
# 5001 (WebSocket)
# The SG will need these ports opened.
WORKER_SECURITY_GROUP_ID=sg-xxxxxxxxx
WORKER_INSTANCE_REGION=
# Numbers of workers to launch before the test starts
WORKERS_NUMBER_AT_THE_BEGINNING=1

For the AUTO distribution of participants to workers

# Overcoming this CPU percentage threshold a new worker will be launched
WORKER_MAX_LOAD=80
# Number of new workers incrementation
WORKERS_RAMP_UP=1

For the MANUAL distribution of participants to workers

# Boolean param for MANUAL participants allocation
MANUAL_PARTICIPANTS_ALLOCATION=false
# Number of participants per worker
USERS_PER_WORKER=

If you want to save the recordings in MinIO

################################################
#### MinIO credentials and information if using it to save recordings instead of S3
################################################
# MinIO Endpoint
MINIO_HOST=
# MinIO Port (default 443)
MINIO_PORT=443
# MinIO Access Key
MINIO_ACCESS_KEY=
# MinIO Secret Key
MINIO_SECRET_KEY=
# MinIO Bucket Name
MINIO_BUCKET=

For video quality control

These properties allow record a participant (launching an external chrome browser) for check the received video quality.

When test finishes, it will saved in the S3 Bucket.

# For video quality control (enabled for AWS testing only) 
# It'll start a new ec2 instance where a new participant will be connected using a real Chrome Browser
# it will start to record the session when media node has archieved this value
# Needs an ElasticSearch instance to work
MEDIANODE_LOAD_FOR_START_RECORDING=0
# it will start to record the sessions gruped by the specifyed value.
# 0: Recording disabled, 1 recording starts each session, 2 recording starts each two session ...
RECORDING_SESSION_GRUPED_BY=0
# Bucket name where the recordings will be saved
# Record each MediaStream in each worker for Quality of Experience analysis
QOE_ANALYSIS_RECORDINGS=true
# Perform qoe analysis in the same worker as they were recorded in, if false the recordings will only be uploaded to S3
QOE_ANALYSIS_IN_SITU=false
# Video information needed for in situ QoE analysis, read https://github.com/OpenVidu/openvidu-loadtest/tree/master/browser-emulator#running-qoe-analysis for more information
VIDEO_PADDING_DURATION=1
VIDEO_FRAGMENT_DURATION=5
S3_BUCKET_NAME=openvidu-loadtest-capacity

For retrying the participant creation

These properties allow retry the participant creation when it fails

RETRY_MODE=false
RETRY_TIMES=5

For selecting the video to use (only used in browserMode REAL)

These properties allow you to select the video to use when the browsers used are real browsers. There are 3 default options and you can also use a custom video, check the description of the browser-emulator Initialize instance API for more information about these options:

# Video type options: BUNNY, INTERVIEW, GAME, CUSTOM
VIDEO_TYPE=BUNNY
VIDEO_WIDTH=640
VIDEO_HEIGHT=480
VIDEO_FPS=30
# If VIDEO_TYPE is CUSTOM, you have to specify the video and audio URLs
VIDEO_URL=
AUDIO_URL=

Monitoring parameters

WARNING: The experiments we made have been tested with Elasticsearch 7.8.0 version. A deployment with a different version could cause errors.

# Monitoring Parameters (Optional)
ELASTICSEARCH_USERNAME=elasticusername
ELASTICSEARCH_PASSWORD=password
KIBANA_HOST=https://kibanahost

| Make sure your OpenVidu Configuration has the following properties for a good monitoring:

• OPENVIDU_PRO_STATS_MONITORING_INTERVAL=1
• OPENVIDU_PRO_STATS_WEBRTC_INTERVAL=1

Miscelaneous configuration

Recommended to leave default values unless you know what you are doing.

# Insert the users in batches or one by one
BATCHES=true
# Maximum number of concurrent requests. If left blank, defaults to number of available processors + 1.
BATCHES_MAX_REQUESTS=10
# Wait after user or batch insertion
WAIT_COMPLETE=true
# Install VNC server on workers for debug purposes
DEBUG_VNC=false

3. Configure session typology

Controller is being developed to allow the configuration of the load test cases: number of participants, typology, number of sessions, etc.

Currently, the only available option is the change of the number of participants of the session.

To configure the test cases the file loadtest-controller/src/main/resources/test_cases.json has to be edited:

{
	"testcases": [
		{
			"typology": "N:N",
			"participants": ["2", "3", "5"],
			"sessions": "20",
			"browserMode": "EMULATE",
			"frameRate": "30",
			"resolution": "640x480",
			"openviduRecordingMode": "",
			"desciption": "This test case will add infinite sessions (until it reaches its limit) of publishers that the array of participants indicates"
		},
		{
			"typology": "N:M",
			"participants": ["1:10", "1:100", "2:10", "2:30", "2:50", "3:10", "3:30", "3:50"],
			"sessions": "infinite",
			"desciption": "This test case will add infinite sessions (until it reaches its limit) with as many PUBLISHERS and SUBSCRIBERS as the participants array indicates."
		},
		{
			"typology": "TEACHING",
			"participants": ["2:10", "2:30", "2:50", "3:10", "3:30", "3:50"],
			"sessions": "infinite",
			"desciption": "This test case will add infinite sessions (until it reaches its limit) with as many PUBLISHERS (teachers:students) as the participants array indicates. The students (FAKE SUBSCRIBERS, they will be PUBLIHSERS with only audio) will only publish audio"
		},
		{
			"typology": "TERMINATE",
			"desciption": "This terminate all EC2 instances launched for loadtest purposes"
		}
	]
}

Test Case JSON properties

Properties	Type	Description
typology *	N:N, N:M, TEACHING or TERMINATE	N:N: All users will be PUBLISHERS and SUBSCRIBERS N:M: N number will be PUBLISHERS and M number will be SUBSCRIBERS TEACHING: It will emulate a teaching videoconference. The students (FAKE SUBSCRIBERS, they will be PUBLIHSERS with only audio) will only publish audio TERMINATE: It will terminate all EC2 instances launched for loadtest purposes
participants *	String []	Number of participants in each session. It will run the same test case as many time as the positions in the array. For example: For N:N typology: ["2","5"] where all of them will be publishers and test case will be run twice (sessions with 2 and 5 participants) For N:M typology: ["2:30"] where 2 will be publishers and 30 will be subscribers For TEACHING typology: ["2:30"] where 2 will be teachers and 30 will be students
sessions *	infinite or Number	infinite: It will create infinite sessions until the limit is reached. Number value: It will create only (number value) sessions
browserMode *	EMULATE or REAL	EMULATE: the service will emulate a browser. REAL: the service will launch a Chrome browser docker container. Choosing EMULATE, you must ensure that OpenVidu aren't forcing H264 coded
resolution	String	Desired resolution of the video. 640x480 or 1280x720 Default 640x480
frameRate	Number (0-30)	Desired framerate of the video in frames per second. Default 30
openviduRecordingMode	String	COMPOSED or INDIVIDUAL See OpenVidu recording.
browserRecording	Boolean	If browserMode is REAL and you want record the Chrome browser using ffmpeg. Otherwise, If browserMode is EMULATE and you have started browser-emulator with KMS user type (see worker running options) Default false.
showBrowserVideoElements	Boolean	If browserMode is REAL and you want show videos elements into the app running in Chrome. Default true
headlessBrowser	Boolean	If browserMode is REAL and you want launch a headless Chrome. Default false. See Headless Chromium

3. Run load test

When you execute the load test the controller will create sessions and will connect participants into them automatically.

The load test has several stop conditions. It will stop when any of the following conditions occur:

• When loadtest-controller receives an exception error from any of the workers.

• When any of workers can't create a new participant because of lack of resources or a long time trying creating it.

Take into account that errors can be produced by OpenVidu errors or if the worker itself is overloaded. Please control CPU usage in workers.

When an error in a worker is produced, the load test will stop and will order to destroy and close every participants and sessions to the workers.

For start with it, run the following command:

cd loadtest-controller
mvn spring-boot:run

Extra features

• For terminate AWS EC2 instances:

mvn spring-boot:run -Dspring-boot.run.jvmArguments="-DTERMINATE_WORKERS=true"

Sample test execution

For illustration proporses here is an example composed by 2 workers and the load test app all of them running locally.

As you can see in the load test and workers logs, the load test app will create sessions and will add participants into them following a round-robin scheduling.

Load test logs

Which it means that, for this sample with sessions of 2 participans, each participant will be create by one different worker. The User0 will be create by the worker 1 and the User1 will be created by the worker2.

Thus achieving more capacity in the load test and less resource consumption.

Worker 1 logs

Worker 2 logs

Test results

The loadtest-controller will create a report result on the root directory /openvidu-loadtest/loadtest-controller with the name result.txt. This file will contains the following information:

 ----- Test Case Report Fri Apr 30 12:39:46 CEST 2021 -----
Browser approach:
Browser with recording:
Session typology:
Participants per session:
Number of sessions created:
Number of sessions completed:
Number of participants created:
Number of workers used:
Stop reason:
Test duration:
Kibana url:

If you're testing OpenVidu PRO, the loadtest-controller will import a Kibana Dashboard automatically at the beginning of the test. This dashboard will include Kibana Visualizations with all metrics retrieved from OpenVidu.

To allow that the Load Test App import it, you must fill the monitoring parameters.

What happen if the dashboard have not been imported automatically?

An alternative if the dashboard is not imported automatically is import it manually. You just to follow these steps:

1. Go to your Kibana Home
2. Open the toggle menu and go to Management > Stack Management.
3. Once inside of Stack Management, you must click on Saved Objects option, under Kibana section.
4. Here, you can find an import button. You have to clik on it and import the loadtest.ndjson file.

Besides, if you have deployed OpenVidu PRO you can create your own visualizations in Kibana or you can export the raw data and use another tool. Remember that ELK stack has monitoring information about the platform (CPU, memory usage, bandwidth, etc) and also high level information (sessions, participants, etc.)

Browser Emulator documentation

This service provides a simple REST API that will be used by Load Test application and it allows:

• Ping to instance. Do ping to check if instance is ready.

• Initialize instance. Initialize monitoring stuffs like ElasticSearch env variables and Metricbeat container. This request also set the AWS public and secret keys for uploading the video recordings files to S3 (just in case the test case includes recording).

• Create a participant (PUBLISHER or SUBSCRIBER) using a custom token created by you or creating a new token.

• Delete a specific participant by its connectionId

• Delete all participant with a specific role (PUBLISHER or SUBSCRIBER).

• Delete all participant

This services also is listening for a WebSocket communication on ws:browser-emulator-addres:5001/events. It will send information from openvidu-browser to the loadtest-controller.

See browser-emulator docummentation for more info.

Acknowledgments

This work has been supported by the Regional Government of Madrid (Spain) (CM) through project EDGEDATA-CM (P2018/TCS-4499) cofunded by FSE & FEDER.