About elastic-hft

This repository contains the application used to perform the experimental evaluation of latency- and energy-aware scaling strategies for data stream processing partitioned-stateful operators proposed in the paper "Keep Calm and React with Foresight: Strategies for Low-Latency and Energy-Efficient Elastic Data Stream Processing" (PPoPP 2016).

The paper has passed the Artifact Evaluation process and the artifact and the related execution procedures are publicly released through this repository. Note that the evaluation was aimed at reproducing qualitatively the results reported in the papers, since the application performance varies depending on the used hardware. If you are willing to read the whole documentation please refer to the auxiliary material of the paper (publicly available, contained in the ae.pdf file).

The application represents the computational kernel of a High Frequency Trading application. A source thread generates a stream of financial quotes, while the application (named elastic-hft in the following) processes the incoming quotes grouped by stock symbols. A sliding window is maintained for each group (quotes belonging to the same stock symbol) and the computation returns a fitting polynomial and other statistics for the last received quotes.

Requisites

To run the application it is required:

• an Intel multicore machine with at least 8 (physical) cores (no virtual machines if you intend to reproduce the results of the PPoPP paper). Processor(s) has to be of the Sandy Bridge generation (or newer). The application does not have strict memory requirements, though having at least 16GB of RAM is a suggested configuration;
• a Linux-based Operating System (kernel 2.6.1 or newer) and root privileges are required. Possibily the application should be run without any concurrent application executed in the meanwhile. The CPU frequency driver used by the OS must be acpi-cpufreq
• gcc version 4.8 or newer. glibc version 2.17 or newer.

##Datasets The artifact takes in input a stream of financial quotes (i.e. bid and ask), which are generated by an external source (in the following generator) provided in the code repository. In the Experiment section of the paper, two different datasets have beenused to drive the generation of the quotes stream:

• a synthetic trace: everything you need to use it is already in the GIT repository;
• a real trace that contains the quotes of a trading day in the NASDAQ stock exchange (daily TaQ of 30 Oct. 2014). The original dataset (that contains other additional info) is available at http://www.nyxdata.com/Data-Products/Daily-TAQ. You can find a smaller binary version ready to be ingested by the generator at http://www.di.unipi.it/~dematteis/taq_nasdaq_2014_49544800. It contains all the quotes for the training day, for a total of 49,544,800 quotes related to 2,836 different stock symbols (approx. 3Gbytes).

##Installation

###Required Libraries The artifact uses external libraries. In particular:

• Fastflow: a C++ parallel programming framework targeting shared-memory architectures. Website: http://calvados.di.unipi.it/
• Lmfit: a C library for Levenberg-Marquardt least-squares minimization and curve fitting. It is used to produce a fitting polynomial of the aggregated quotes (per window). Website :http://apps.jcns.fz-juelich.de/doku/sc/lmfit
• Mammut: a C++ library providing mechanisms for collecting energy statistics and for changing the CPU frequency. Website: https://github.com/DanieleDeSensi/Mammut

In the sequel we will provide a brief description of the installation procedure.

####Fastflow It is an header-only library. Therefore, it is only required to download it from the website or the SVN. To download the latest version and save it into the fastflow directory, run the following command in the shell:

$ svn checkout svn://svn.code.sf.net/p/mc-fastflow/code/ fastflow

####Lmfit It can be downloaded from the SVN linked in the website. To download the version used in the artifact execute the following command:

$ wget http://sourceforge.net/projects/lmfit/files/lmfit/lmfit-6.1.tgz

Then, you have to extract and install it as usual. The local installation can be performed using the classical configure/make/make install chain.

####Mammut The library is released via a public GIT repository. The artifact uses the version 0.1. To download it, execute the following command:

$ git clone https://github.com/DanieleDeSensi/Mammut.git
$ cd Mammut
$ git checkout tags/v0.1.0

To perform a local installation, you have to manually edit the Makefile and specify at the line export MAMMUT_PATH your installation path. After that you can compile and install as usual:

$ make -j
$ make install

To properly use the library, in some kernels the msr module must be loaded by the operating system and the used CPU frequency driver must be acpi-cpufreq (default in many kernels up to 3.9).

###Artifact building

The source code of the artifact is available through this GIT repository:

 $ git clone https://github.com/tizianodem/elastic-hft.git
 $ cd elastic-hft
 $ git checkout tags/v1.0

For the compilation it is required to set proper environment variables to indicate the libraries installation path. The variables are FASTFLOW_DIR for the FastFlow library, LMFIT_DIR for the Lmfit library, and MAMMUT_DIR for the Mammut library. Furthermore, for the execution it could be required that the path to the Lmfit library is in your LD_LIBRARY_PATH environment variable. After that, the code can be compiled. The set of command is the following:

 $ export FASTFLOW_DIR=<...path to fastflow...>
 $ export LMFIT_DIR=<... path to lmfit...>
 $ export MAMMUT_DIR=<...path to mammut...>
 $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$LMFIT_DIR/lib
 $ make -j

This will produce a set of binaries:

• elastic-hft is the High Frequency Trading application used for the experimental evaluation;
• synthetic- and real-generator, respectively the data generator for the synthetic dataset and the real one;
• derive-voltage-table a utility program, whose scope is cleared in the following section.

##Experiment Workflow

In this section we will describe the principal steps required to evaluate the artifact and reproduce (qualitatively) the results obtained in Sect. Control Strategies Evaluation and Sect. Comparison with similar approach of the paper.

Please note that from now on the commands must be launched with root privileges (or sudoer user properly exporting the needed environment variables).

###Preliminaries

In order to estimate the energy consumption, it is required to have the information about the voltage levels of the CPU. To derive this values you can use the utility `derive-voltage-table present in the code repository. It must be run before starting the program for the first time:

 # ./derive-voltage-table

It produces a file voltages.txt that contains the approximate values of the voltages and it is used by the artifact (it must reside in the same folder of the binary). More accurate measurements of these values can be obtained using the proper demo in the Mammut Library (under demo/energy/voltageTable). However, the approximate version provided by the utility in this artifact is good enough for our scope and requires just a couple of minutes to be executed.

###Workflow for the control strategies evaluation The typical evaluation workflow will require to firstly start the elastic-hft application and then the generator:

1. start the elastic-hft program. It takes in input various parameters:

   1. the number of stock symbols (keys) that the stream will convey. This is used mainly for statistical purposes. For the datasets that are included in this artifact this value must be equal to 2,836;
   2. the starting number of replicas. This will be automatically adjusted during the execution according to the scaling strategy used. If this number exceeds the maximum allowed (see below how to derive it) for the used machine, an error is returned at program start up and the execution aborts;
   3. the TCP port (e.g, 8080) over which the program will wait for a connection from the generator;
   4. the window size expressed in number of quotes. In the experiments it was set to 1,000;
   5. the window slide expressed in number of quotes. In the experiments it was 25 (it can be changed provided that it is a divider of the window size);
   6. a configuration file that provides a set of configuration parameters of the used scaling strategy (see below). For example, the following command will launch the program with an initial number of replicas equal to 5: # ./elastic-hft 2836 5 8080 1000 25 path_name_of_the_configuration_file.cfg

2. start a generator: depending on the chosen dataset you can launch the two different generators provided with the artifact:

   • the synthetic-generator for the synthetic dataset. It requires the following command line parameters:

     1. the hostname (or IP address) of the machine in which elastic-hft is executed;
     2. the TCP port number over which elastic-hft is waiting for the connection;
     3. the number of stock symbols of the dataset: 2,836 in our case;
     4. a file containing the probability distribution of the various stock symbols;
     5. the initial rate, expressed in quotes (messages) per seconds;
     6. the total number of quotes to generate. This determines the execution length;
     7. optionally, a file that contains the different data generation rate during the execution. If not specified the generator will produce quotes with a fixed rate.

     The probability distribution file and the rates file used for the experiments with the synthetic dataset (random walk workload) are stored in the distr\_and\_rates folder of the code repository. For example, if you want to start the generator with the same characteristic of the one used for the experiments (assuming that elastic-hft is in execution on the same machine of the generator), type: #./synthetic-generator localhost 8080 2836 distr_and_rates/probability_distribution 300000 54000000 distr_and_rates/random_walk_rates This will produce a quotes stream that will last approximately 180 seconds.

   • the real-generator, if you are willing to use the real dataset. In this case you have to specify as command line arguments:

     1. the hostname (or IP address) of the machine in which elastic-hft is executed;
     2. the TCP port number over which the elastic-hft is waiting for the connection;
     3. the path to the provided dataset;
     4. the number of quotes to generate (up to 49,544,800 for the provided dataset);
     5. -s <number> optional parameter, it specifies how many times the original dataset must be accelerated. If not specified it will run at the original speed, resulting in 6 hours and an half of data generation. For this reason we recommend to accelerate it: in our experiments this throttling parameter was set to 100, resulting in a data generation time of 236 seconds.

     An example of execution is the following: # ./real-generator localhost 8080 ./dataset 49544800 -s 100

We recommend to execute the \texttt{generator} and the \texttt{elastic-hft} on the same machine mainly for two reasons:

• avoid problems due to network interconnection (e.g., slow data transfer);
• avoid problems due to clock drift in one (or both) the used machines, that may results in wrong latency measurements.

Processes (and relative threads) are automatically pinned on the available physical cores (one per core; Hyper-Threading, if present, is not used). For this reason the \textit{maximum number of replicas that can be used} is equal to the number of physical cores minus 4 (a core is used respectively by the generator, splitter, merger and controller). In any case, if you are willing to execute the generator on a different machine, please add the following macro definition: -DGENERATOR\_ON\_DIFFERENT\_MACHINEon the DEFINES line of the Makefile and recompile

###Configuration File description the application takes as input parameter a configuration file that details the type of scaling strategy to use. The configuration file is a plain text file and has to respect a proper syntax that is specified in the auxiliary material of the paper.

You can find all the configuration files used for the experiments reported in the paper in the config_file folder, organized according to the used dataset. You can modify them in order to create your own configurations.

###Evaluation and expected results The results must be validated qualitatively with respect to the ones in the paper. In fact, the exact numerical values depend on the hardware configuration

####Control strategies evaluation In the experiments section we compared our strategies in terms of the SASO properties (i.e. stability, accuracy, settling time, overshoot). For the two datasets we run the application using the dif- ferent strategies (Lat-Node and Lat-Power) and we evaluated the results according to various criteria. During the elastic-hft execution, various information are printed in the standard output typically at each second of the execution: numver of replica used, average latency, throughput.

At the end of the execution the so-called SASO summary is printed. These results are needed to evaluate and compare the various strategies. In particular it shows:

• the total number of reconfigurations performed. In the case of the Lat-Power strategy the SASO summary also differentiates the type of reconfigurations (changes in the number of replicas and in the CPU frequency). The total number of reconfigurations is meaningful to evaluate the stability property. Better stability requires fewer reconfigurations;
• the number of latency violations with respect to the specified threshold. This parameter is fundamental for evaluating the accuracy SASO property. Obtaining few violations is an evidence of a better accuracy of the strategy.
• the average reconfiguration amplitude. This measurement corresponds to the settling time SASO property. A better settling time is achieved with high reconfiguration amplitude values
• the average resource consumption of the strategy. This information corresponds to the overshoot SASO property. The Lat-Node strategy cannot modify the CPU frequency, so the meaningful parameter to evaluate the overshoot is the average number of replicas used. In constrast, the average watt consumption is the main overshoot measure for the Lat-Power strategy. For the sake of completeness, we also report the Watts consumed per second by Lat-Node strategy. In our artifact we measure the core energy counter, while the overall socket consumption is usually ~30 watt higher. Finally, it is worth remembering that the overshoot property is better with lower resource consumption values

All the measurements produced by elastic-hft (i.e. the number of produced results, average latency, number of replicas used and CPU frequency reported per seconds, and the SASO summary) are saved in a file named stats.dat. The generators produce a generator.dat file that contains information about the data rate generated at various time instants.

Remember that the goal of this artifcat is to reproduce the same qualitative bheavior of the results shown in the paper. It makes possible to reproduce the experiments in Figs. 9, 10, 12 and 13 of the paper, in which each strategy is analyzed by comparing different strategy configurations in terms of the SASO properties.

####Comparison with similar approaches The same qualitative comparison can be achieved by executing the two other strategies (spl and latency_rules). At the end of the execution the SASO summary is printed. It is possible to achieve results qualitatively similar to the one of Tab 3 of the paper

Tuning and scripts

The results of the artifact greatly depend on few parameters that are critical and require a certain tuning effort. In particular, for the Lat-Node and the Lat-Power strategies it is required to choose a proper threshold for the latency in the configuration file. You will find all the related information on the auxiliary material of the paper.

Moreover scripts are provided to run the same program configuration multiple times and extract average results (in the paper experiments were run 25 times). Similarly, you can find the information about how to use this scripts in the auxiliary material of the paper.