Add a mode to handle "pretty URLs", i.e. URIs without .html extension

[dscho]

In many circumstances (GitHub Pages, Apache configured with MultiViews, etc), web servers process URIs by appending the .html file extension when no file is found at the path specified by the URI but a .html file corresponding to that path is found.

To allow Lychee to use the fast, offline method of checking such files locally via the file:// scheme, let's handle this scenario gracefully by adding the --auto-append-html-fileext option.

This is especially nice to have when migrating sites that have been run on, say, a web app, to a static site generated by Hugo, as I am trying to do with Git's home page.

[dscho]

Hmm. Most of the CI failures (all except one) appear outside the code touched by this PR. Help?

[mre]

In general, I like the idea. However, I'd love to make it more generic, i.e. cover more use-cases.
For instance, a user might want to try a different extension like .htm or .php instead.
Multiple extensions could be thinkable, too. And the order of extensions might also be nice to be configurable.
Here's an extended proposal.

--check-extensions <EXTS>
      Test the specified file extensions for URIs when checking files locally.
      Multiple extensions can be separated by commas. Extensions will be checked in order of appearance.
      
      Example: --check-extensions html,htm,php,asp,aspx,jsp,cgi

Any thoughts?

[dscho]

Here's an extended proposal.

--check-extensions <EXTS>
      Test the specified file extensions for URIs when checking files locally.
      Multiple extensions can be separated by commas. Extensions will be checked in order of appearance.
      
      Example: --check-extensions html,htm,php,asp,aspx,jsp,cgi

@mre done!

[thomas-zahner]

Sounds like a useful addition. I agree that we can add this more generic approach to lychee

[thomas-zahner]

Using the found flag and manually looping and breaking is not too nice. Instead try to use .iter and find_map.

Maybe similar to the following (didn't test it):

let check_extension_exists = |ext: &str| { // could also be a function
    let mut path_buf = path.clone();
    path_buf.set_extension(ext);
    if path_buf.exists() {
        Some(path_buf)
    } else {
        None
    }
};

self.check_extensions.iter()
    .find_map(check_extension_exists)
    .map(|path_buf| {
        if self.include_fragments {
            self.check_fragment(&path_buf, uri).await
        } else {
            Status::Ok(StatusCode::OK)
        }
    })
    .next()
    .unwrap_or_else(|| ErrorKind::InvalidFilePath(uri.clone()).into());

[dscho]

@thomas-zahner correct me if I am wrong, but don't we want to stop at the first found extension? The suggested map() would not stop there, and in the best case do unnecessary work, in the worst case do unwanted work.

[dscho]

Here is my proposed code to address your concerns.

[thomas-zahner]

Actually iterators are lazy and run only when it's really necessary. This is described in the rust book.

For example we can convert an infinite range of numbers into an iterator, map the numbers and take just a few of the numbers. Then the map only executes as many times as we take a result. In my code snippet above the next() consumes just one element so that the map only runs once.

    let iter = (0..).into_iter().map(|x| x * 2).take(5);
    println!("{:?}", iter.collect::<Vec<_>>());

[dscho]

Actually iterators are lazy and run only when it's really necessary. This is described in the rust book.

For example we can convert an infinite range of numbers into an iterator, map the numbers and take just a few of the numbers. Then the map only executes as many times as we take a result. In my code snippet above the next() consumes just one element so that the map only runs once.

    let iter = (0..).into_iter().map(|x| x * 2).take(5);
    println!("{:?}", iter.collect::<Vec<_>>());

Okay, I didn't know that :-)

So now I'm torn: the code you showed that uses a map() looks less obvious than I'd like it to be, certainly more complicated than equivalent constructs in languages that I'm more familiar with than Rust. It may very well be more idiomatic Rust code than the for loop code that is in the current version of this PR. However, the latter version is much easier to parse and grok for me. So I am torn.

[thomas-zahner]

Yeah, maybe my proposed iter solution is not perfect but normally iter is to be preferred over for loops. Maybe the iter solution could be simplified further. @mre what do you think?

[mre]

In general, I also prefer iterators, but in this case in particular, I'd lean towards the for-loop because of the early return. Now that the found flag got removed, it's gotten much more readable already. So I'm fine with the current version.