Update package metadata

[ofek]

Background

Hello there! The Python packaging ecosystem has now standardized on the interface for build backends (PEP 517/PEP 660) and the format for metadata declaration (PEP 621/PEP 631).

As such, I'm spending my free time updating important projects so that they are modernized and set an example for others 😄

Summary of changes

This implements PEP 621, obviating the need for setup.py, setup.cfg, and MANIFEST.in. Support has not landed in setuptools, so builds will now use hatchling.

I've done this for such projects as pipx, all Datadog Agent integrations, etc.

Notes

• This also drops support for EOL Python 3.x versions since hatchling and other build backends like setuptools have already done so. This will not break anything because pip only looks for version candidates that satisfy the requires-python field. Here are stats for the curious: https://pypistats.org/packages/boltons
• This also resolves __version__ field #290 as I saw that had not yet been handled, @mahmoud let me know if that's okay!

[mahmoud]

Wow! Rocketing into the future here. boltons is a simple library, so this makes a lot of sense, and I appreciate you taking the time!

One small change, and a couple thoughts/concerns:

1. Hatchling seems very new (still 0ver ;) ). What's the plan for maintenance/stability?
2. Does Hatch/Hatchling offer anything like packaging tests we can add to the tox.ini?

Thanks in advance!

[ofek]

Hatchling seems very new (still 0ver ;) ). What's the plan for maintenance/stability?

It is quite stable and actively maintained, the only reason the version is 0.x is because 1.0.0 will drop support for Python 2.

Does Hatch/Hatchling offer anything like packaging tests we can add to the tox.ini?

If you mean like checking the contents of the sdist *.tar.gz file then no, I don't think any of the newer backends do that because there is no more MANIFEST.in which was the confusing and error prone part.

[ofek]

Oh btw it's also on conda-forge, Debian, Fedora, etc.

[ofek]

@mahmoud Would you mind triggering the CI with that button below?

[mahmoud]

Would` you mind triggering the CI with that button below?

Done :) All green.

Hatchling seems very new (still 0ver ;) ). What's the plan for maintenance/stability?

It is quite stable and actively maintained, the only reason the version is 0.x is because 1.0.0 will drop support for Python 2.

I guess my main concern is that Hatchling first came out (0.3) two months ago? And I love the velocity, but it also makes me nervous should something break. Presumably I'd own the fix at that point :)

Longer term, what's your recommendation when setuptools support lands?

If you mean like checking the contents of the sdist *.tar.gz file then no, I don't think any of the newer backends do that because there is no more MANIFEST.in which was the confusing and error prone part.

MANIFEST.in could certainly be tricky, but even in this modern approach, couldn't the includes here go out of sync if a new file is added?

Finally, maybe a very simple question, but if someone were to seek to install boltons from a source distribution built with this modern approach, how would it be recommended they do that without a setup.py? Does pip know what to do these days?

[ofek]

should something break. Presumably I'd own the fix at that point

No it would be on the maintainers, just as if you used setuptools or some other build backend. Since we use it at Datadog for shipping business critical things, our CI would be the first to yell at me 😄

---

Longer term, what's your recommendation when setuptools support lands?

I still would not use it. I can briefly tell you why we moved:

Our goal is to remove all setup.py files since their use is now heavily discouraged and build with a PEP 517 compatible build system while defining metadata with PEP 621.

setuptools

Disadvantages:

• Does not yet support pyproject.toml-based metadata declaration nor editable installation standards (PEP 660)
• Based on a deprecated standard library module that it now vendors
• Slow to get things merged since the code quality issue is exacerbated by the fact that it is mostly maintained by just one person

Flit

flit-core is the build backend behind Flit.

Disadvantages:

• No mechanism for extending behavior by design, so we could never do things like versioning with Git tags
• Very opinionated about project layouts since it targets beginners e.g. only supports implicit namespace packages, must define certain data like __version__ inside the root __init__.py file, etc.

Poetry

poetry-core is the build backend behind Poetry.

This was actually a no-go option for us due to its persistent unwillingness to adopt new standards if they are deemed suboptimal (see comments on PEP 621 and PEP 665)

Hatchling

• In comparison to setuptools, Hatchling is far smaller, making it less susceptible to security issues and more auditable. For example, the logic to build a wheel resides in this one file.
• In comparison to flit-core, Hatchling is extensible by design. For example, there is already a plugin for setuptools_scm (which despite the legacy naming is actually now decoupled from setuptools) and a plugin for compiling code with Mypyc.
• In comparison to poetry-core, Hatchling never strays from community standards and is in fact eager to adopt new ones (it's the only backend that currently supports PEP 639).

---

couldn't the includes here go out of sync if a new file is added

Well yes of course. If you want everything added to the source distribution that section can just be omitted, or define exclude.

---

if someone were to seek to install boltons from a source distribution built with this modern approach, how would it be recommended they do that without a setup.py? Does pip know what to do these days?

Yes, since version 19.0.

[ofek]

Just let me know if you want me to do anything else here!

[bwatson-dk]

You could also just use the setuptools build backend.

build-backend = "setuptools.build_meta"

It's pretty standardized and I don't think there are any concerns about maintenance. Also, since a build backend has a standard to follow, if Hatchling really takes off, it wouldn't be hard to switch to it.

The mentioned lack of support for pyproject.toml metadata in setuptools is also incorrect (it was at right at one point, but it's supported it for a while). It's actually possible to have a pyproject.toml as the only configuration file.

[build-system]
requires = [
    "setuptools ~= 61.0",
    "wheel",
    "build",
]
build-backend = "setuptools.build_meta"

[project]
name = "boltons"
version = "21.0.1dev"
description = "When they're not builtins, they're boltons."
urls = { "homepage" = "https://github.com/mahmoud/boltons" }
authors = [
    { "name" = "Mahmoud Hashemi", "email" = "mahmoud@hatnote.com" },
]
license = { "text" = "BSD"  }
classifiers = [
    "Topic :: Utilities",
    "Topic :: Software Development :: Libraries",
    "Intended Audience :: Developers",
    "Development Status :: 5 - Production/Stable",
    "License :: OSI Approved :: BSD License",
    "Operating System :: OS Independent",
    "Programming Language :: Python :: 2",
    "Programming Language :: Python :: 2.6",
    "Programming Language :: Python :: 2.7",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.4",
    "Programming Language :: Python :: 3.5",
    "Programming Language :: Python :: 3.6",
    "Programming Language :: Python :: 3.7",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: Implementation :: CPython",
    "Programming Language :: Python :: Implementation :: PyPy",
]

[project.optional-dependencies]
test = [
    "tox",
]

That's all you'd theoretically need; no need for setup.py or setup.cfg at all.

[ofek]

It's actually possible to have a pyproject.toml as the only configuration file.

Support landed a few days ago and is experimental. See notes and fixes: https://setuptools.pypa.io/en/latest/history.html

I don't think there are any concerns about maintenance

True but in comparison to setuptools, Hatchling has reproducible builds, supports the modern PEP 660 editable (pip install -e ...) installation standard, has a more intuitive plugin system, and the codebase is an order of magnitude smaller thus making it more auditable and less prone to bugs (setuptools is based on a deprecated standard library module that it now must vendor).

[induane]

I'm not sure having distutils in setuptools is a huge issue, it was actually removed from Python in part BECAUSE it was already in setuptools anyway and there wasn't a reason to maintain it in the standard library. Full support for pyproject.toml WAS only recently merged, but it's been possible to include metadata there for a while but it didn't validate it and defaulted to setup.cfg in any conflict; I've been doing so for a spell and with the new release the whole thing is working even better.

Really though @ofek I'm not disagreeing with your analysis; my swing through Hatchling actually impressed me and I'm going to take it for a spin on a personal project so I have a reminder to keep an eye on it. It may be (probably even IS?) the superior project. I hope it takes off!

Really though, technical analysis aside, my preference for widespread important projects like boltons tends to be conservative. For better or worse setuptools is the major player here. It's not the only one but it's the old hat in the ring and it's the conservative choice. Were Hatchling to really catch on and take off I'd say it would be a really cool thing to look at upgrading to. Swapping build backends isn't too hard anyway.

Of course it's not my call at all 😄 - I was just popping by to leave an unsolicited opinion 😉

[ofek]

Small update: Hatch will likely be adopted by PyPA in the next week

• https://mail.python.org/archives/list/pypa-committers@python.org/thread/J2BL5XTTOCXUCELUE7L5P5HHJ7NXZGZ5/
• https://mail.python.org/archives/list/pypa-committers@python.org/thread/ZWSSNKQJ3GRIAK73FJS6KHOJNUJZYOVX/

Nothing major will change, just the repo location

[ofek]

Transfer complete! https://github.com/pypa/hatch