Popular repositories

1. ImmoScout24-scrapy-mongodb-docker ImmoScout24-scrapy-mongodb-docker Public

   Docker image for scrapy based spider to scrape https://www.immobilienscout24.de/ and store results in MongoDB

   Python 2

2. rabbitmq-minio_s3-elk-sitemap_crawler-cluster rabbitmq-minio_s3-elk-sitemap_crawler-cluster Public

   Scalable sitemap crawler cluster, using RabbitMQ, Elasticsearch, Logstash, Kibana and Minio S3.

   Python 1

3. Bias-to-Search-Engines-from-Robots.txt Bias-to-Search-Engines-from-Robots.txt Public

   Determining bias to search engines from Robots.txt

   Jupyter Notebook

4. In-memory-URL-Compression In-memory-URL-Compression Public

   AVL tree for In-memory URL Compression

   Jupyter Notebook

5. Lists Lists Public

   Forked from blocklistproject/Lists

   Primary Block Lists

   JavaScript

6. discriminative-feature-learning-for-url-based-topic-classification discriminative-feature-learning-for-url-based-topic-classification Public

   Discriminative feature learning for web page topic classification based on URL. Docker image and FastAPI are used to create to a classification endpoint.