Software as a service scraping the Hot Deals forums and delivering newsletter emails on trending deals from the Forum.
Red Flag Deals does aggregate deals on their front page, but the Hot Deals Forums are community driven and crowdsourced. This is where the purpose of my project comes into play, this project scrapes the Hot Deals Forums several times per day.
With this project, I saved myself the chore of checking the (messy) forum a few times a day while still being aware of the good deals posted by the community.
The project is split into several directories (services):
-
Scraper
Scrapes the forums periodically and depends on the database to persist the data.
-
Mailer
Sends the daily newsletter email.
-
Backend
REST API, takes care of sending data to the frontend and managing subscribing and unsubscribing.
-
Frontend
Displays the deals nicely, and provides the subscription form.
-
Proxy
Forwards requests to the right service.
-
HTML templates
Email templates built with mjml for the design of the newsletter emails and confirmation emails.
- Go programming language
- SendGrid
- Docker & Docker Swarm
- PostgreSQL
- Redis
- Drone CI
- Nginx
- Next.js
- React.js
- Material-UI
- Automated scraping of the Forum
- Insert and update a database
- Automated newsletter email
- Algorithm to filter the junk
- Website to browse the deals and subscribe to the emails
- Confirmation email when a user subscribes
- Go(lang) v1.14+
- Docker-compose v1.25.5+
- Docker v19.03+
- Node.js v14+
| Variable | Description |
|---|---|
| SENDGRID_API_KEY | SendGrid API key to use the service |
| SENDGRID_TEMPLATE | SendGrid requires you to upload an html template to use for dynamic emails, this is the ID of the template |
| SENDGRID_TEMPLATE_CONFIRM | SendGrid template ID for confirmation emails |
| POSTGRES_NONROOT_DB | Database for the appuser |
| POSTGRES_DB | Root database |
| POSTGRES_USER | Root username to initialize PostgreSQL |
| POSTGRES_PASSWORD | Root password to initialize PostgreSQL |
| POSTGRES_NONROOT_PASSWORD | The user password to use the database, we don't want to use the root user for these tasks |
| POSTGRES_NONROOT_USER | The user username to use the database |
| DEV | Set this variable to anything to run as dev (testing purposes) |
| SCRAPER_HC_UUID | UUID for Healthchecks.io |
| MAILER_HC_UUID | UUID for Healthchecks.io |
Build individual Go services into a binary
go build -o main .
Bring the services up using docker-compose.
go test
Check out the open issues for ideas and features I have planned!
You may open an issue for discussion.