Any best practice to manage lock file for dev-dependencies?

[huxuan]

Background

I am working on a template project, the prod dependencies is quite simple right now, but many dev-dependencies for lint, test and doc are already included. So the default group only takes a small proportion of the whole lock file (15/81 when starting this discussion). Moreover, there are some standalone tools I want to "lock", such as copier, pre-commit and absolutely pdm itself.

Thoughts / Questions

1. The chances of dependency conflicts will increase when the project becoming larger, so is it a good idea to separate the dev dependencies from prod dependencies?
2. Some dev-dependencies actually rely on the installation of the project, for example, mypy and pytest. It seems that we have to keep those dependent dev-dependencies alongside prod dependencies, right?
3. What if there are conflicts between dev dependencies and prod dependencies? A intuitive idea is to keep prod dependencies independent and override the dependencies only when necessary.
4. Currently, I use pipx to manage those standalone tools, but it lacks of lock strategy. Will pdm be helpful in some way? Although it may not be a good idea to manage pdm installation with pdm.

[frostming]

We need to classify development tools into several categories:

1. Can run independently without installing the project itself, or purely static analysis, such as ruff, pre-commit, flake8.
2. Semi-independent tools, may require installation of the project itself or some additional dependencies, such as mypy.
3. Need to run completely in the project environment, such as pytest.

From my personal habits, if you use pre-commit, you can install all type 1 with pre-commit, and don't specify them in dev-dependencies, except for pre-commit. But if you don't want to involve pre-commit, since type 1s run on their own and shouldn't have anything to do with other tools, there is no point locking them together in the lock file. Options I can see include pipx and nox/tox with pinned versions(Coincidentally, they all end with x.).

Type 2 can also be managed by pre-commit. For tools like mkdocs and sphinx that are only required for specific purposes, add them in its own dependency group and create separate lock files for them, if dependency conflicts become a concern, or just lock them with others.

Type 3 should always be locked with the project dependencies, so add them in dev-dependencies!

[huxuan]

Just want to double confirm, feel free to point out if there is any error.

From a practical perspective, an arbitrary conclusion would be:

1. Include pytest related packages in the test group of dev-dependencies and lock them together with the default group in the main pdm.lock.
2. Include sphinx/mkdocs related packages in the doc group of dev-dependencies and lock them in a separated lock file, for example, pdm.doc.lock.
3. Most lint tools can be managed by pipx/tox/nox (3x!), so no lock file or separated lock files should be OK.

To the best of my knowledge, seems the only exception of lint tool might be mypy as mypy needs the installation of all prod dependencies for type checking, maybe we should also treat it as Type 3, the test group?