PHP Monitoring

Monitor your PHP application with logs, metrics, pings, and traces. Slides: https://speakerdeck.com/xeraa/monitor-your-php-application-with-the-elastic-stack

Features

1. Quick overview of what is running in Kibana's monitoring view.
2. Metricbeat System:
   1. Show the [Metricbeat System] Overview dashboard in Kibana.
   2. Then switch to [Metricbeat System] Host overview and see the spike.
   3. Build a visualization with Time Series Visual Builder to find out what is going on: system.memory.used.bytes per beat.name and system.process.memory.rss.bytes per system.process.name sorted by the Sum of system.process.memory.rss.bytes.
   4. Enable the disabled: yes in the deploy_bad.yml playbook to avoid any hiccups during the demo. Show the annotation of the event in the visualization above (potentially come back to it at the end of the talk).
3. Packetbeat: Let attendees hit the CMS with a few requests.
   1. Show the [Packetbeat] Overview and [Packetbeat] Flows.
   2. Explain why [Packetbeat] HTTP is empty.
   3. Show [Packetbeat] TLS Sessions and explain why this is one of the more important Packetbeat features.
4. Filebeat modules:
   1. Show the [Filebeat Nginx] Overview and [Filebeat Nginx] Access and error logs dashboards.
   2. Show the [Filebeat MySQL] Overview dashboard.
   3. Show the [Filebeat System] SSH login attempts, [Filebeat System] Sudo commands, and [Filebeat System] Syslog dashboard dashboards.
   4. Show the [Osquery Result] Compliance pack dashboard.
5. Run ./ab.sh on the backend instance to get a more interesting view of the [Filebeat Nginx] Overview and [Packetbeat] MySQL performance dashboards.
6. Metricbeat modules:
   1. Show the [Metricbeat Nginx] Overview dashboard based on https://xeraa.wtf/server-status.
   2. Show the [Metricbeat MySQL] Overview dashboard.
   3. Build a Time Series Visual Builder visualization for https://xeraa.wtf/status: Sum of php_fpm.pool.connections.accepted (optionally the derivative of this value), sum of php_fpm.pool.connections.queued, and sum of php_fpm_pool.process.active on a different axis and as a bar.
   4. Add annotations to the previous visualizations — they don't correlate in this example, but it is still handy to see.
7. Filebeat: Collecting both /var/www/html/silverstripe/logs/silverstripe.log and /var/www/html/silverstripe/logs/silverstripe.json. Hit https://xeraa.wtf/error/, https://xeraa.wtf/error/server/, https://xeraa.wtf/error/client/, and https://xeraa.wtf/error/exception/ for different errors and find them in the logs. Also point out the cloud meta.* and host.* information.
8. Heartbeat: Run Heartbeat and show the Heartbeat HTTP monitoring dashboard in Kibana, then stop either nginx or php-fpm (different response code).
9. Auditbeat: Show the dashboards for [Auditbeat Auditd] Overview and [Auditbeat File Integrity] Overview.
10. Kibana Dashboard Mode: Point attendees to the Kibana instance to let them play around on their own.

Setup

1. Make sure you have your AWS account set up, access key created, and added as environment variables in AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. Protip: Use https://github.com/sorah/envchain to keep your environment variables safe.
2. Create the Elastic Cloud instance with the same version as specified in variables.yml's elastic_version, enable Kibana as well as the GeoIP & user agent plugins, and set the environment variables with the values for ELASTICSEARCH_HOST, ELASTICSEARCH_USER, ELASTICSEARCH_PASSWORD, as well as KIBANA_HOST, KIBANA_ID.
3. Change into the lightsail/ directory.
4. Change the settings to a domain you have registered under Route53 in inventory, variables.tf, and variables.yml. Set the Hosted Zone for that domain and export the Zone ID under the environment variable TF_VAR_zone_id. If you haven't created the Hosted Zone yet, you should set it up in the AWS Console first and then set the environment variable.
5. If you haven't installed the AWS plugin for Terraform, get it with terraform init first. Then create the keypair, DNS settings, and instances with terraform apply.
6. Open HTTPS (443) on the network configuration on all instances as well as MySQL (3306) and APM server (8200) on the backend one (waiting for this Terraform issue to automate that step).
7. Apply the base configuration to all instances with ansible-playbook configure_all.yml.
8. Apply the instance specific configuration with ansible-playbook configure_frontend.yml and ansible-playbook configure_backend.yml.
9. Deploy the JAR with ansible-playbook deploy_bad.yml (Ansible is also building it) and ansible-playbook deploy_frontend.yml.

When you are done, remove the instances, DNS settings, and key with terraform destroy.

Todo

• Switch to: metricbeat keystore create && metricbeat keystore add output.elasticsearch.password
• Change: Alerting example
• APM: https://github.com/frankkoornstra/elastic-apm-agent/releases (needs PHP 7.2)