[Performance] Copying files with utils::fs::copy_files_except_ext is slow

[johamster]

Problem

Our book is growing and by now building it takes approx. 1m30s. Most of the time is spent in HtmlHandlebars::render when copying files from src to dest.

In a small proof-of-concept I replaced the call to copy_files_except_ext locally with rsync. Execution time dropped to 8s.

Proposed Solution

The copy_files_except_ext seems to be a bottleneck. Various performance improvements can be thought of:

• Parallelize, e.g., by starting threads for copying sub-directories (though not trivial to fine-tune for performance, e.g., limit to CPU cores or the like,...)
• Use platform-dependent tools like rsync if available. Very fast, allows for exclude patterns.

Notes

I've seen the comment on recursion while copying. I did not dig into that but is it really an issue? Only if the target directory is a sub-directory of source or if we encounter a (sym-)link into the target directory somewhere in source, I would assume. Both would be strange setups, wouldn't it?

[johamster]

I pushed a first very simple PR to reduce allocations while copying. No logical changes. The PR can be found here: #2355. I don't have permission to link it to this issue.