Skip to content

A list of publicly available datasets with real-time data maintained by the team at bytewax.io

License

Notifications You must be signed in to change notification settings

bytewax/awesome-public-real-time-datasets

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

28 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Awesome Public Real-Time Datasets and Sources

Awesome

This list is inspired by awesome public datasets, but for real-time datasets and sources. Normally accessed via HTTP or Websockets.

The list is separated into Free and Paid and broken into subsections based on loose categories.

Free

Finance/Crypto

  • Coinbase Market Data - Coinbase websocket to market data including level 2 orderbook data.
  • Blockchain transactions - Provides real-time notifications about new transactions and blocks.
  • Yahoo Financewss://streamer.finance.yahoo.com/ - This is not advertised in developer documentation, but discoverable as the websocket is used to update their website.
  • Finnhub - Limited free usage with a premium data sources also available.
  • CoinCheck - a cryptocurrency API that has a WebSocket interface (in beta)
  • Alpaca Markets Real-Time and historical market data via HTTP and Websocket.

Transportation

  • Open Rail Data - A collection of APIs that provide data relating to the UK rail network, including reference data, train timetables, and live service updates. The live data is streamed using the STOMP protocol.
  • GBFS New York - GBFS is the standard for bike share data with many locations around the world. Find more information here
  • Open Sky Flight - Data from Open Sky Api via HTTP endpoint. Supports real-time, but not streaming. Need to continually poll.
  • Open Glider Network - The OGN provides real-time traffic for gliders and other light aircraft. You can use an OGN client like python-ogn-client to connect to OGN servers, parse the APRS messages and push them to a broker like Kafka for streaming processing.
  • MTA GTFS Feed - Transit data in GTFS format for transit systems like NYC subway and Caltrain.
  • NY 511 live camera data - This live camera data requires some scraping to use. From this list of cameras you can then source the individual camera id and then request the timestamped image or most recent image by building the url like - https://511ny.org/map/Cctv/<image-id-goes-here>
  • Transport for London (TfL) - live data about the tube, buses, and more
  • Norwegian Coastal Administration - AIS data from vessels within the Norwegian economic zone and the protection zones off Svalbard and Jan Mayen.
  • German Traffic Data - German real-time traffic information
  • Swiss Traffic & Public Transport Data - Various real-time transport data from Switzerland such road traffic, status of EV charging stations, shared mobility services and live arrivals/departures of public transport

Information

IoT

  • ThingSpeak IoT Public Channels - Crowdsourced IoT channels of users publishing various IoT sensor data in real-time. Accessible via REST API or MQTT API.

Other

Paid

Finance/Crypto

  • IEX Trading - IEX was created in response to questionable trading practices that had become widely used across traditional exhcanges! Their API provides streaming Stock market data.
  • Twitter - Twitter provides a streaming interface for research or enterprise. It is unknown what support will be post Twitter purchase in 2022.
  • NYSE Cloud Streaming - Provides real-time access to high-quality NYSE exchange data feeds, streaming directly in the cloud using Kafka format.
  • Alpha Vantage Market News & Sentiment - Live market news & sentiment data from selected news outlets covering stocks, cryptocurrencies, forex, and a wide range of topics such as fiscal policy, mergers & acquisitions, IPOs, etc.

Transportation

  • AIS Data Maritime, Aviation and weather data available via Spire.

Information

  • PurpleAir Air Quality Data - Developer API for accessing purple air sensor data.
  • NewsAPI - NewsAPI tracks headlines in 7 categories across over 50 countries, and at over a hundred top publications and blogs, in near real time. Free developer version with 24 delays available.

Sports

  • Sports Livescores - Developer API of TheSportsDB that gives you access to livescores
  • Sportradar Sports Data - Global live data of 80 sports, 500 sport leagues and 750k events a year (free 30 day trial available)

About

A list of publicly available datasets with real-time data maintained by the team at bytewax.io

Topics

Resources

License

Code of conduct

Stars

Watchers

Forks

Contributors 4

  •  
  •  
  •  
  •