Septa Stats

Source code for http://www.SeptaStats.com/

This is a Splunk app which downloads Regional Rail train stats from SEPTA's public API once per minute and contains dashboards and reports for visualizing that data.

Installation Instructions

• Clone this repo
• Make sure Docker is installed
• Run docker-compose up -d to start all the Docker containers
• Splunk can be found at http://localhost:8001/, login is admin/password.
  • This instance should NOT be made available over the Internet. Seriously. Don't do it.
• The web app can be found at http://localhost:8002/
  • I recommend blocking this port off from the Internet and using your own instance of Nginx to proxy to it, with HTTPS.
• AWS credentials should be in aws-credentials.txt
  • Note that if you create a policy for your IAM credentials (as you should!), the ARN in the Resource array must end in /*. Exammple: arn:aws:s3:::septa-stats/*. There is a bug in the policy generator where this won't happen and your backups will fail. Be careful.

Architecture Overview

The Docker containers in this project are as follows:

• splunk: Runs a PHP script to fetch from SEPTA's API every 55 seconds, and saves the data to a Splunk Index.
  • Configuration of Splunk is saved in splunk-files/.
• web: Nginx webserver
• php: PHP running in FCGI mode to run PHP code
• redis: Used to cache the results of Splunk queries
• backup: Make regular backups of train data to AWS S3

Development

Working on PHP Code

• cp docker-compose.yml.example docker-compose.yml
• docker-compose up
• PHP code lives under htdocs/
• Go to https://localhost:8000/ in your web browser for Splunk. Password is in docker-compose.yml.
• Go to http://localhost:8001/ in your web browser to view the website

Working on Splunk

This will stand up Splunk and spawn a shell in it:

docker-compose kill splunk; \
	docker-compose rm -f splunk; \
	docker-compose build splunk; \
	docker-compose up -d splunk; \
	docker-compose exec splunk bash

Test the PHP script inside the container with:

• /opt/splunk/etc/apps/septa-stats/bin/get-regional-rail-trainview.php | jq .

How to run the backup script in the foreground for development:

docker-compose kill backup \
	&& docker-compose rm -f backup \
	&& docker-compose build backup \
	&& docker-compose up backup

Deployment

• SSH to the webserver
• cd /var/septa-stats/
• git pull
• If you're doing an HTML or PHP update, you're done. Otherwise, restart the affected containers.

Exporting Data

The Spunk Index is written to splunk-data/, so that when the container is restarted, no data is lost.

The Redis data is written to redis-data/, so that no data is lost when that container is restarted.

Importing Data

Assuming you exported the raw events from Splunk to a text file, you can import those events with these commands:

• docker-compose exec splunk bash
• /opt/splunk/bin/splunk add oneshot ./septa-stats.txt -sourcetype oneshot -index septa_analytics

FAQ: Why is Composer's vendor/ directory included?

I'm a huge fan of reproducable builds, and I'm a little concerned about the possibility of things like this happening, which can cause both large amounts of breakage and application build failures.

That said, I've been having some conversations with folks on r/PHP and other places, and while they acknowledged that my concern is legitimate, the risk is very low, and including vendor/ just adds so much more noise to code reviews. That's a valid point, and I am reconsidering my decision to include the vendor/ directory. Feel free to reach out to me if you feel strongly one way or the other on this issue.

Credits

This package uses the timewrap command, available at https://splunkbase.splunk.com/app/1645/.

Additional Questions?

More Questions? Please email me at dmuth@dmuth.org, doug.muth@gmail.com, or open an issue here.