River Charts is a Python
, Django
, Plotly
, and Pandas
web application that visualizes river data for a specific river/site/location.
- The line graph is driven by data pulled using an
API
from the United States Geological Survey (USGS).- The data is captured by the USGS using a gage height sensor every 15 minutes.
- The data is updated with the most recent river height data every time the application is loaded.
- Please be patient, the loading time of the application is around 60+ seconds (depending on your internet connection), due to the large
API
JSON
response and the amount of data being processed from the USGS.- As a future enhancement (or if you want to fork/develop it), I'd like to cache the data on the server, or store it in a database with a scheduled data refresh.
- I'd also like to investigate using Websockets for real time updates.
- This will allow the removal of the loading/skeleton screen for the initial load/wait time due to the API request that is sent out during every page visit/load.
- Please be patient, the loading time of the application is around 60+ seconds (depending on your internet connection), due to the large
- I recommend using the application on a desktop since the chart is interactive has a wider view, but it can be used on a mobile device as well.
- Visit the application here.
- Features
- Background Story
- Definitions
- Getting Started
- What's Inside?
- API Documentation
- Disclaimer
- What's Next?
- Project
- Contributing
- Resources
- License
- Credit
- Responsive: The application is responsive and can be used on a mobile device.
- Interactive: The application uses
Plotly
to create an interactive line graph. - Data Visualization: The application visualizes river data for a specific river/site/location.
- Data Source: The application sources data using an
API
that returnsJSON
output. - Data Processing: The application processes the
JSON
output and converts it to aPandas
DataFrame
. - Data Manipulation: The application manipulates the
DataFrame
to create aPlotly
line graph. - Data Update: The application updates the data with the most recent river height data every time the application is loaded.
- Data Capture: The data is captured by the USGS using a gage height sensor every 15 minutes.
Every year, my friends and I float 2 miles down the Susquehanna River in NEPA on river tubes (a 2 hour float). I wanted to create a web application that would allow us to visualize past river data in order to see the river height on the days we floated down the river. Some float dates, we still got together, but we didn't float due to the dangerous river levels. Green plots on the graph represent the dates we did float (with a "Floated" status of "Yes"), and Red plots represent the dates we did not float (with a "Floated" status of "No").
The closest river gauge to our float location is the Susquehanna River at Meshoppen, PA, which is the default site code in the application. You can change the site code in the application to visualize data for a different river.
Here are some definitions to help you understand the terminology used in this document:
- USGS: The United States Geological Survey. The USGS is a science organization that provides impartial information on the health of our ecosystems and environment, the natural hazards that threaten us, the natural resources we rely on, the impacts of climate and land-use change, and the core science systems that help us provide timely, relevant, and useable information.
- Gage Height: The height of the water surface above the gage datum (zero point). Gage height is often used interchangeably with the more general term, stage, although gage height is more appropriate when used with a gage reading. Stage is more appropriate when used with a recorded or calculated gage height.
This project makes use of several libraries and frameworks:
- Python: For the application logic.
- Django: For web application functionality.
- Plotly: For creating interactive visualizations.
- Pandas: For data manipulation and analysis.
- Requests: For making
API
calls. - Python-Decouple: For storing sensitive information in a
.env
file. - See requirements.txt for a full list of dependencies.
- Edit
config.py
to add your own USGSAPI
(and other) information. - Toggle
USE_DUMMY_DATA
toTrue
inconfig.py
to use dummy data instead of theAPI
.- This is useful for testing the application without making
API
calls.
- This is useful for testing the application without making
- The float data plots are driven from a
.csv
file located instatic/data/river_charts.csv
.- This file can be edited to add/remove float dates.
- The file is read in
views.py
and passed to the template as acontext
variable.
To install and run the project locally, follow the steps below:
-
Clone the repository:
git clone https://github.com/scottgriv/River-Charts
-
Navigate to the project directory:
cd [YOUR PROJECT DIRECTORY]
-
Create a virtual environment:
python -m venv venv
-
Activate the virtual environment:
source venv/bin/activate
-
Install the required packages using
requirements.txt
:pip install -r requirements.txt
Note: If you wany to generate a new
requirements.txt
file, run the following command:pip freeze > requirements.txt
-
Run the Django server:
python manage.py runserver
Now, you can visit http://127.0.0.1:8000/
in your browser to access the application.
- The application is hosted here on PythonAnywhere.
- The application is deployed using a
WSGI
configuration file. - First, make sure you adjust your
settings.py
fileALLOWED_HOSTS
to include your deployment host. - Second, make sure you adjust your
settings.py
fileDEBUG
toFalse
for production. - Finnaly, be sure to create a
.env
file where you host your application to store your sensitive information (excluded from this repository).
Below is a list of the main files and folders in this repository and their specific purposes:
River-Charts # Root folder
├─ config.py # A file that contains sensitive information (excluded from this repository).
├─ manage.py # A command-line utility that lets you interact with this Django project in various ways.
├─ requirements.txt # A list of Python packages required to run this project.
├─ River_Charts # A directory for the river_charts app.
│ ├─ __init__.py # An empty file that tells Python that this directory should be considered a Python package.
│ ├─ admin.py # A file that registers models to be displayed in the Django admin site.
│ ├─ apps.py # A file that contains the application configuration.
│ ├─ models.py # A file that contains the database models.
│ ├─ tests.py # A file that contains the tests for the application.
│ ├─ urls.py # A file that contains the URL declarations for the application.
│ └─ views.py # A file that contains the application logic.
├─ river_charts # The Django project directory.
│ ├─ __init__.py # An empty file that tells Python that this directory should be considered a Python package.
│ ├─ asgi.py # An entry-point for ASGI-compatible web servers to serve your project.
│ ├─ settings.py # Settings/configuration for this Django project.
│ ├─ urls.py # The URL declarations for this Django project.
│ └─ wsgi.py # An entry-point for WSGI-compatible web servers to serve your project.
├─ static # A directory for static files that are used in this Django project.
│ ├─ css # A directory for CSS files.
│ │ └─ styles.css # A CSS file that contains the styles for the application.
│ ├─ data # A directory for data files.
│ │ └─ river_charts.csv # A CSV file that contains the float dates for the application.
│ └─ images # A directory for image files.
├─ templates # A directory for HTML templates.
│ └─ river_charts # A directory for HTML templates specific to the river_charts app.
│ ├─ error.html # An HTML template that displays an error message.
│ └─ index.html # An HTML template that displays the application.
├─ views.py # A file that contains the application logic.
├─ .github # GitHub folder
├─ .gitignore # Git ignore file
├─ .gitattributes # Git attributes file
├─ PRG.md # PRG Connection File
├─ LICENSE # A file that contains the license for this project.
└─ README.md # This file.
Below is the documentation for the API
used in this application.
To call the API
and retrieve the data:
-
Make a GET request to:
http://nwis.waterservices.usgs.gov/nwis/...
(based on your requirements).- ex.
https://waterservices.usgs.gov/nwis/iv?format=json&sites=01533400&startDT=2015-07-01&endDT=2023-08-16¶meterCd=00065&siteStatus=active&siteType=ST
- ex.
-
Pass the necessary parameters in the request.
- ex.
params = { "format": "json", # Set your interchange format. "sites": "01533400", # Site Code: Susquehanna River at Meshoppen, PA. "startDT": "2015-07-01", # Set the date you want to start collecting data from. "endDT": "2023-09-14", # This is based on the current date in the application. "parameterCd": "00065", # Parameter Code: Gage height, ft. "siteStatus": "active", # Selects sites based on whether or not they are currently active. Each USGS Water Science Center determines whether a site is active or inactive. The default is all (show both active and inactive sites). "siteType": "ST", # ST = A body of running water moving under gravity flow in a defined channel. The channel may be entirely natural, or altered by engineering practices through straightening, dredging, and (or) lining. An entirely artificial channel should be qualified with the "canal" or "ditch" secondary site type. }
-
Process the JSON response as demonstrated in the example above.
Notes:
- The application uses the
API
to source data for the graph. If theAPI
is down, the graph will not render. - Errors are handled in the application by redirecting the user to an error page with the appropriate error message.
HTTP
errors are handled in the application by redirecting the user to an error page with the appropriate error message.- The application will also display an error message if the
API
returns an empty response or a timeout error (default is set to 90 seconds inconfig.py
). - The
API
is rate limited to 30 calls per minute. If you exceed this limit, you will receive a429
error.
The application sources data using an API
that returns JSON
output. Here's an example of what the API response looks like:
{
"name": "ns1:timeSeriesResponseType",
...
"timeSeries": [
{
"sourceInfo": {
"siteName": "Susquehanna River at Meshoppen, PA",
...
},
"variable": {
"variableName": "Gage height, ft",
...
},
"values": [
{
"value": [
{
"value": "15.13",
"dateTime": "2015-07-01T00:00:00.000-04:00"
},
...
]
}
]
}
]
}
(For the sake of brevity, the full output is abbreviated with ...
)
- The data provided by this application is sourced from the USGS.
- It's subject to revision, and for more information, please refer to their official disclaimer.
- Software is provided as-is and no warranty is given about its usability.
Thank you for taking the time to read through this document and I hope you find it useful! If you have any questions or suggestions, please feel free to reach out to me.
Please reference the SUPPORT file in this repository for more details.
I'm looking forward to seeing how this project evolves over time and how it can help others with their GitHub Portfolio.
Please reference the CHANGELOG file in this repository for more details.
Please reference the GitHub Project tab inside this repository to get a good understanding of where I'm currently at with the overall project.
- Issues and Enhancements will also be tracked there as well.
Feel free to submit a pull request if you find any issues or have any suggestions on how to improve this project. You can also open an issue with the tag "bug" or "enhancement".
- How to contribute:
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/River-Charts
) - Commit your Changes (
git commit -m 'Add new feature'
) - Push to the Branch (
git push origin feature/River-Charts
) - Open a Pull Request
Please reference the CONTRIBUTING file in this repository for more details.
Below are some external resources I found helpful when creating River Charts:
- Python - A programming language that lets you work quickly and integrate systems more effectively.
- Django - A high-level Python Web framework that encourages rapid development and clean, pragmatic design.
- Plotly - A Python graphing library that makes interactive, publication-quality graphs online.
- Pandas - A fast, powerful, flexible and easy to use open source data analysis and manipulation tool.
- Requests - A simple, yet elegant HTTP library.
- Python-Decouple - A Python library for separating the settings of your Django/Flask/FastAPI project from the source code.
- Deploying Django - A tutorial on how to deploy Django.
- Django Settings - A list of all settings available in Django.
- Running the Django server - A tutorial on how to run the Django server.
- Creating Virtual Environments - A tutorial on how to create virtual environments.
- Activating a virtual environment - A tutorial on how to activate a virtual environment.
- Installing Packages - A tutorial on how to install packages using
pip
. - Requirements Files - A file containing a list of items to be installed using pip install like so:
pip install -r requirements.txt
. - PythonAnywhere - A Python hosting provider with a free tier.
Note
To use external API's (like this application does), PythonAnywhere requires a basic paid tier.
- USGS - The United States Geological Survey.
- USGS API - The USGS Instantaneous Values Web Service.
- USGS API Documentation - The USGS Instantaneous Values Web Service Documentation.
- USGS Site Web Service - The USGS Site Web Service.
- Codes and Parameters - The USGS Codes and Parameters Documentation.
- How to Get Real-time Flood Data from the USGS API - A tutorial on how to get real-time flood data from the USGS API.
- USGS Client Library - Client library for interfacing with USGS datasets (GitHub repository).
- USGS Client Library Repository - Client library for interfacing with USGS datasets.
- USGS API Homepage - The USGS API Homepage.
- USGS Rest API web services - The USGS Rest API web services.
- USGS Instantaneous Water Service data URL generator - The USGS Instantaneous Water Service data URL generator.
- National Water Dashboard - The National Water Dashboard.
- Rivers vs. Streams vs. Creeks (USGS) - A USGS article on the difference between rivers, streams, and creeks.
- How to Make Colored Country Maps with GeoPandas and Python - A tutorial on how to make colored country maps with GeoPandas and Python.
- 5 Ways to Use Free Data on the Internet for Science - A tutorial on how to use free data on the internet for science.
- GitHub for Biologists - A tutorial on how to use GitHub for biologists.
This project is released under the terms of the GNU General Public License, version 3 (GPLv3).
- The GPLv3 is a "copyleft" license, ensuring that derivatives of the software remain open source and under the GPL.
- For more details and to understand all requirements and conditions, see the LICENSE file in this repository.
Author: Scott Grivner
Email: scott.grivner@gmail.com
Website: scottgrivner.dev
Reference: Main Branch