With one line of code, Robotex (pronounced like “robotics”) will download and parse the robots.txt file and let you know if your program is allowed to visit a given link.
Usage:
robotex = Robotex.new "My User Agent" robotex.allowed?("http://www.example.com/foo") robotex.delay!("http://www.example.com/foo") # wait until any specified Crawl-Delay has passed robotex.sitemaps("http://www.example.com/") # return an array of sitemap urls
Robotex is a modified version of Kyle Maxwell’s excellent Robots library. Some folks were unable to use that gem due to packaging issues, so I used his code to create Robotex.