Skip to content

abdollah-aghayan/crawler-worker-redis_queue

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Installation instructions

In order to run the app you need to have redis installed

You can change the server port in config

Then just need to run code by following command

go run main.go

There are 5 workers as default but you can increase the number of worker by max_workers flag

go run main.go -max_workers=1

Technologies

library I have used for this app are in the below

github.com/gin-contrib/cors v1.3.1
github.com/gin-gonic/gin v1.6.2
github.com/gomodule/redigo v1.8.0
golang.org/x/net v0.0.0-20200501053045-e0ff5e5a1de5

Requirement

I have implemented a API for user request to be able to send one or more url(s) stor them in a redis queue(FIFO) and fetch them with workers and get the title

Test

I haven't got time to write test yet but you can find a postman profile in extra folder which you can import and test the api

About

A small app to crawl sevral website with worker pattern and redis FIFO queue

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages