Generates a robots.txt
-
Updated
Nov 1, 2019 - JavaScript
Generates a robots.txt
A tool for debugging robots.txt
The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).
🚫🤖 Override /robots.txt to disallow all web crawlers, regardless settings stored in the database. Compatible with Liferay 7.0, 7.1, 7.2, 7.3 and 7.4.
This is a python crawler that disregards robots.txt rules and downloads disallowed resources
🌐 Displays the contents of robots.txt and sitemap.xml files of a website google extension
Robots.txt parser / generator
Front-end workflow to start a new project with Eleventy and Webpack.
Robots.txt parser and generator - Work in progress
Robots Scanner
This is ready to use template to quickly start selling domain with minimum setup.
Optimizes your site's robots.txt to reduce server load and CO2 footprint by blocking unnecessary crawlers while allowing major search engines and specific tools.
Sitemaps and Robots.txt for websites around the world.
Fully native robots.txt parsing component without any dependencies.
A simple to use multi-threaded web-crawler written in C with libcURL and Lexbor.
💧 Test your robots.txt with this testing tool. Check if a URL is blocked, which statement is blocking it and for which user agent. You can also check if the resources for the page (CSS and JavaScript) are disallowed!. Robots.txt files help you guide how search engines crawl your site, and can be an integral part of your SEO strategy.
A package for generating a robots.txt programmatically.
robots.txt checker/monitoring - The python-script checks the robots.txt content and statuscode and sents an emails if the check fails
Add a description, image, and links to the robots-txt topic page so that developers can more easily learn about it.
To associate your repository with the robots-txt topic, visit your repo's landing page and select "manage topics."