IReS-Platform

Intelligent, Multi-Engine Resource Scheduler for Big Data Analytics Workflows

Links

• IReS Paper
• ASAP Project Official Page

Using IReS-Platform

Before using IReS it should be configured properly. This setting consists of 3 main steps,

1. Cloning IReS-Platform to the server. For a quick reference of how to use git, click here Git - the simple guide
2. Running install.sh that
   • builds IReS-Platform using Maven. A tutorial about Maven can be found here Maven Getting Started Guide.
   • connects IReS to Hadoop YARN.
   • updates configuration files and folders appropriately.
3. Setting cluster resources and services monitoring

For demonstration reasons a Linux operating system like Ubuntu is assumed throughout this text. In Windows or other Linux distributions the equivalents should be done. The local home directory of the IReS-Platform project is depicted as

$IRES_HOME

following bash script variable notation. Similarly, for Hadoop YARN the local home directory is denoted as

$YARN_HOME.

Hands On

1. Clone

   Open a terminal( Linux) and navigate to a desired directory (create it if does not exist) where IReS-Platform files will be cloned. In IReS-Platform github page, under the green drop down list "Clone or download", the clone url can be found. Copy this url and execute in terminal the command,

   git clone clone_url

2. Run install.sh

   After successful cloning of IReS-Platform inside the $IRES_HOME various folders and files can be found. Among them there exists install.sh.

   install.sh is your friend!

   You can run install.sh from any directory you would like. Here for demnostration reasons is assumed that the current working directory is $IRES_HOME.

   Executing,

   ./install.sh

   will start building IReS-Platform. Upon successful building you will be prompted to provide the path where Hadoop YARN is located in your computer. By doing this, IReS gets connected with Hadoop YARN. You can skip this step and the installation will be finished.

   NOTE:

   1. if you do not provide an existing YARN installation, then IReS will not be able to execute any workflow. Also, resources and cluster services monitoring will not be functioning.
   2. you can provide YARN installation path afterwards as it will be shown straight ahead.

   Connecting IReS to Hadoop YARN

   Executing,

   ./install.sh -c $YARN_HOME,$IRES_HOME

   will make the connection of IReS and YARN, where $YARN_HOME and $IRES_HOME correspond to the absolute paths of YARN's and IReS's home folder.

3. Cluster Monitoring

   This step requires the connection of IReS with YARN. Assuming that this connection has been established, then the user should update the file

   $YARN_HOME/etc/hadoop/yarn-site.xml

   and more specifically the values of the following properties,
   • yarn.nodemanager.services-running.per-node
   • yarn.nodemanager.services-running.check-availability
   • yarn.nodemanager.services-running.check-status

   These properties and some others have been added during the connection of IReS and YARN to enable IReS run workflows over YARN and cluster resources and services monitoring. Although details about filling these values are provided into $YARN_HOME/etc/hadoop/yarn-site.xml, roughly speaking, yarn.nodemanager.services-running.per-node property describes the cluster services running per node. The property, yarn.nodemanager.services-running.check-availability provides the commands per service that "tell" if the relative service runs or not. Finally, the property yarn.nodemanager.services-running.check-status has the statuses per service that the corresponding service has when it runs.

Validate installation

Here are some tips to confirm IReS installation.

1. If anything goes wrong during the build process of IReS, error messages will be print out and a log file will be provided.
2. Run ASAP server by running the command

   ./install.sh -r start

   No exception should be raised. Also, the jps command should print a "Main" process running that corresponds to ASAP server.
3. Run ASAP server web user interface at http://your_hostname:1323/web/main. IReS home page should be displayed.
4. Run a workflow, for example run "hello_world" from "Abstrack Workflows" tab and see what happens not only in IReS web interface but also in YARN and HDFS web interfaces. Make sure that YARN has been started before running any workflow.
5. Click on "Cockpit" tab to verify that the expected services to run are really running.

Appendix

To see all available functionalities of install.sh run

./install.sh -h

However, in this section some details about install.sh script are given.

To begin with, install.sh supports can be run from user's working directory, it is not needed navigating to IRES_HOME folder. Moreover, user can access server's logs by running install.sh script using -l flag.

To see all available functionalities of install.sh run

./install.sh -h

Now assume that for some reason the IRES_HOME has been changed. For example you would like to rename IReS home folder or move it to another folder. In this case, all configuration files of IReS should be updated with the new value of $IRES_HOME. To do so, run the command

./install.sh -s NEW_IRES_HOME