Multi-Container Application for ELK Stack

This is a project for monitoring the performance of servers and network traffic using ELK (ElasticSearch, LogStash and Kibana) as well as grafana and a single file was created for creating multi-container application (docker-compose). Thus, using a single file makes possible to create a full-deployed stack required by OpenStack monitoring solution. It has two main part to be monitored in openstack such as the server performance where its samples were produced by ceilometer and network flow where its samples were produced by softflowd.

P.S. While exporting traffic via softflowd, we might need to use v1.0 rather than 0.9 because the tool does not capable of exporting vlan traffic to logstash properly. This is important especiallay for exporting the netflow obtained for internal traffic such a from br-eth interface.

Multi-container for an application - docker-compose.yaml

The volumes and network were seperated because make any change on the configuration files can be easy without modifying the compose files and adding/removing another container can be applied properly by adding just another ip address.

We have deployed 6 containers on a single computer which has 48 cores and 128 GB RAM. Furthermore, 2 additional containers were also deployed running a self-developed python code regulary. It is not possible to monitor a tenant network traffic since ceilometer does not provide the information. On the other hand, sotfflowd can capture the network traffic based on the ip adressed. However, we need to visualize the network traffic for each tenant separately. (floating ip and router gw interface belong to specific tenant). Mapping container was simply constructed (Dockerfile) because of this purpose.

The list of docker application and their properties were listed below.

The list of containers

• logstash

  • IP: 172.26.36.4
  • It has 3 main volumes for their own basic requirements such as storing the data and configuration files. (logstash-data, logstash-config, logstash-pipe) Furthermore, it has 2 additional volumes (mapping-res, mappingprod-res) for indexing the entries. Actually, these are the files which were generated by containers (mapping, mappingprod)
  • The java was set to 16G memory. Otherwise, it was crushed because of insufficient memory.

• elasticsearch

  • IP: 172.26.36.2, 172.26.36.3 and 172.26.36.7
  • The cluster has 3 containers. Even all of them are located on the same host, it would be separated over the multiple host in order to get high availability.
  • It has 2 main volumes for storing data and its configuration.
  • The logstash ip and memory size for java was set properly

• kibana

  • IP: 172.26.36.5
  • The elasticsearch ip was set in order to visualize their data.

• grafana

  • IP: 172.26.36.6
  • The elasticsearch ip was not initialized because the configuration is applied on the web application. However, the username and password were set here.

• mapping and mappingprod

  • IP: 172.26.36.10, 172.26.36.11
  • These are ubuntu containers which runs regularly self-developed python codes in order to index the entries in a proper format using the translate functionality of the logstash. There are number of ip addresses which belong to same tenant so that these ip addresses were accumulated under a single title and indexed properly. The xml files were generated and updated for each hour. The file were used by logstash because the volume were attached these both containers.
  • The name of the executable python as well as the output files were set here.

Network setup

The physical host has a management interface with 172.26.0.0/16 subnet. The interface were configured in order to communicate with all other servers in the same subnet. First, the interface was set to promisc mode and a new docker network was created as following. Than, the interface was added into the created docker bridge.

• The management interface (ip: 172.26.2.109) of the physical host was set to promisc mode

 auto eno1
 iface eno1 inet manual
      up ifconfig $IFACE 0.0.0.0 up
      up ip link set $IFACE promisc on
      down ip link set $IFACE promisc off
      down ifconfig $IFACE down 

• 172.26.36.0/24 subnet was properly created

 docker network create --driver=bridge --subnet=172.26.0.0/16 --gateway=172.26.2.109 --ip-range=172.16.36.0/24  br-elk 

• eno1 interface was added into the bridge

 brctl show
 bridge name    bridge id        STP enabled    interfaces
 br-9949d68762cf        8000.0242d463be6a    no        eno1
                            veth79b71fe
 docker0        8000.02429e179696    no

• Thus, br-elk network was created where a new container get IP address as following.

 docker run -it --network=br-elk  ubuntu /bin/bash 

• Finally, a new router was created on the physical host in oder to allow containers communicate the outside.

 route add default gw 172.26.0.1 

As a result, the management ip address of the host was set to gateway of the docker network and the created containers can have access to reach outside.