robots

A library that parses and validates rules from robots.txt.

Installation

This library is available on hex.pm. To install this library, simply add the following lines to your rebar.config:

{robots, "1.1.1"}

Usage

Content = <<"User-Agent: bot\nAllow: /fish">>,
%% This will return an opaque type that contains all the rules and their agents
{ok, RulesIndex} = robots:parse(Content, 200),
true = robots:is_allowed(<<"bot/1.0.0">>, <<"/fish/salmon.html">>, RulesIndex),
true = robots:is_allowed(<<"bot/1.0.0">>, <<"/Fish.asp">>, RulesIndex),

Development

Running all the tests and linters

You can run all the tests and linters with the rebar3 alias:

rebar3 check