I use Cuba to create an application with an endpoint /health-check that returns an HTTP 200 in the 80% of cases and HTTP 500 in the other 20% of cases. Also there is a button in the index page that alter the /health-check endpoint returning an HTTP 500 in the 70% of cases, and slow down response time in the other 30% of requests, then after 10 requests, returns to his original behaviour.
With the ELK stack and Watcher I detect anomalies in the application (slow responses or HTTP 500 consecutive) and send an email if any of the anomalies happen.
Is installed like a service in Ubuntu with the .deb package.
I download the .tar.gz package because is easy to select what configuration file load when Logstash start.
To test the Logstash installation works execute in the extracted folder
bin/logstash -e 'input { stdin { } } output { stdout {} }'
The -e flag enables you to specify a configuration directly from the command line. This pipeline takes input from the standard input, stdin, and moves that input to the standard output, stdout, in a structured format. Type hello world at the command prompt to see Logstash response.
I chose the .tar.gz package, then in the extracted folder, run
bin/kibana
Before running Kibana service, you need to have the Elasticsearch service running.
Is a plugin for Elasticsearch, so you need to install in the path/to/elasticsearch/bin/plugin. In my case (using the .deb package) I need to execute:
sudo /usr/share/elasticsearch/bin/plugin install elasticsearch/license/latest
sudo /usr/share/elasticsearch/bin/plugin install elasticsearch/watcher/latest
To send an email first you need to modify the Elasticsearch configuration file, to do that you need to find elasticsearch.yml file (here you can see where is), here you can find details about email configuration.
If you use all the files in this repo, you'll have all the services running in your localhost.
rackup
sudo service elasticsearch start
In the extracted folder
bin/logstash -f path/to/config_file.conf
I created a gist with the configuration to run Logstash in localhost and in Bluemix.
In the extracted folder
bin/kibana
You need to make a request to the Elasticsearch API to create a new watch.
I created a gist with the configuration for watcher. To detect HTPP 500 and slow requests.
I use the curl command
curl -X PUT 'http://localhost:9200/_watcher/watch/500_detection' -d @500_watch
I create a new index in Elasticsearchm, with the name 500_detection, you can choose any one you want, with -d @ flag I’m telling to curl read the watch file to load the settings.
I use this example application to test all the stack in Bluemix. For this I modified a Docker image.