Web Crawler in Python

First attempt in web crawling, learned it from UDEMY. This web crawler, crawls through the website and fetches all the links, And even goes through all the fetched links and fetches more links. And this way, it keeps on crawling over all the links.

Its obvious but still, if you are using it don't forget to mention your url in main.py file. All the crawled links/urls will be stored in crawled.txt and those, remaining to be crawled, are in queue.txt file. Happy crawling :)