COVID-19 Scraper

Collection of jobs which scrape, parse and export coronavirus data automatically.
Live Demo

Report Bug · Request Feature

Table of Contents

• About the Project
  • Built With
• Getting Started
  • Prerequisites
  • Installation
• Usage
• Roadmap
• Contributing
• License
• Contact

About The Project

This project is a collection of jobs that scrape websites of official coronavirus data sources (e.g. Ministry of Health), parse the data and export it in a common format. Ultimately the exported data is automatically commited to the covid-database repository. Currently there are jobs for Bulgaria data only, but soon it will support other countries as well.

Built With

• Sidekiq
• sidekiq-cron
• etc.

Getting Started

To get a local copy up and running follow these simple example steps.

Prerequisites

• npm

gem install bundler

Installation

1. Clone the repo

git clone https://github.com/COVID-19-Bulgaria/covid-scraper.git

2. Install dependencies

bundle install

Usage

Run sidekiq:

bundle exec sidekiq -C ./config/sidekiq.yml -r ./crontab.rb

Run sidekiq-web:

bundle exec rackup config.ru

Or run both:

bundle exec foreman start

If you use foreman you can specify all environment variables within a .env file in the mail directory. Otherwise you will need to export them in the current terminal either by hand or with a script.

Then access http://localhost:9292, start developing and monitor the jobs execution!

Roadmap

See the open issues for a list of proposed features (and known issues).

Contributing

Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Project is distributed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Contact

Veselin Stoyanov: