Mr-Roboto is aimed at handling and parsing a website's robots.txt file, providing a easy interface to validate if a specific path is allowed or not.
Mr-Roboto is NOT production ready, and as such should not be used in any mission critical codebase.