Separate and clearly document direct/transitive vendored dependencies with the help of pip-tools

[abravalheri]

The work in this PR was extracted from #4330.

Motivation

Right now it is a bit tricky to deal with the vendored packages:

• The developer either have to manually pin down all the direct and transitive dependencies in vendored.txt or
• There will be dependencies installed in the _vendor folder that are not listed in the vendored.txt file.
  • This makes it is hard to guess why they are there and to track the dependency chain...
  • It also make it possible for pkg_resources and setuptools to have different versions of the same vendored dependency. This is probably not nice for OS-level repackaging.

Summary of changes

The most relevant commit of this PR is b5e91c8.

It modifies tool/vendored so that:

1. Transitive dependencies are automatically documented
   (so we don't loose track of what needs what).
   • This is done via pip-compile and vendored.in (direct dependencies)
     ==> vendored.txt. See 1a31ed1 and 20df694.
2. Instead of a case-by-case import rewrite, all vendored imports are rewritten at once
   • Case-by-case workarounds were kept (e.g. intentionally delayed imports)
3. Sync versions of vendored packages in pkg_resources and setuptools
   so that it is easier for "de-vendoring" (this is done via --constraint in pip).

I also added a base_python to the tox environment intended to match the lowest version of Python supported by setuptools, so that dependency markers are solved for a more general case.

The remaining of the changes are collateral.

No attempt to remove extern was made.

If reviewing this PR I recommend checking commit-by-commit to avoid the big blob of automatic changes related to running tox -e vendor.

Closes

Pull Request Checklist

• Changes have tests
• News fragment added in newsfragments/.
  (See documentation for details)