Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nix shell updates for develop #18131

Merged
merged 3 commits into from Aug 23, 2022
Merged

Nix shell updates for develop #18131

merged 3 commits into from Aug 23, 2022

Conversation

sigprof
Copy link
Contributor

@sigprof sigprof commented Aug 21, 2022

Description

Update the nix-shell environment in the develop branch to use QMK CLI 1.1.1 and newest versions of other required Python modules (including pyserial, which is a recent addition to the Python dependencies).

The update is required to make the current state of the develop branch buildable using nix-shell (with the existing version qmk fails to do anything due to the missing pyserial module).

@AstridYu: This somewhat interferes with your change from #16602, because any externally supplied pkgs must also contain at least version 0.8.0 of python3.pkgs.tomlkit (but I did not specifically test that version, just noticed that python-poetry/tomlkit#148 should be fixed there). It could be possible to add a nixpkgsSource argument to override just sources.nixpkgs (then the shell.nix code could import that source with the overlay which upgrades tomlkit), but I'm not sure how this would align with your use case; and if you actually intend to supply a recent enough Nixpkgs in pkgs, things will probably just work without that overlay.

Some seemingly useless changes in util/nix/poetry.lock are caused by python-poetry/poetry#6153 (probably the order of some entries depended on the order of results from the PyPI API, which changed at some point); looks like the fix won't be backported to the 1.1 branch of Poetry, so we are stuck with that issue for some time (at least the order does not seem to change between further runs for now).

Tested on ubuntu-latest, macos-11, macos-12 (failures in that run are for the develop branch of qmk/qmk_firmware without this change).

Python module version updates:

  • atomicwrites: none → 1.4.1 (but this module is not actually used, because the corresponding dependency of pytest is win32-only)
  • attrs: 21.4.0 → 22.1.0
  • colorama: 0.4.4 → 0.4.5
  • coverage: 6.4 → none
  • dotty-dict: none → 1.3.1 (used instead of qmk-dotty-dict)
  • editables: none → 0.3
  • flake8: 4.0.1 → 5.0.4
  • flake8-polyfill: 1.0.2 → none
  • flit-core: none → 3.7.1
  • hatch-fancy-pypi-readme: none → 22.3.0
  • hatch-vcs: none → 0.2.0
  • hatchling: none → 1.8.0
  • hjson: 3.0.2 → 3.1.0
  • importlib-resources: 5.7.1 → 5.9.0
  • iniconfig: none → 1.1.1
  • jsonschema: 4.5.1 → 4.14.0
  • mccabe: 0.6.1 → 0.7.0
  • nose2: 0.11.0 → 0.12.0
  • packaging: none → 21.3
  • pathspec: none → 0.9.0
  • pep8-naming: 0.12.1 → 0.13.2
  • pillow: 9.1.1 → 9.2.0
  • pkgutil-resolve-name: none → 1.3.10
  • pluggy: none → 1.0.0
  • poetry-core: none → 1.0.8
  • py: none → 1.11.0
  • pycodestyle: 2.8.0 → 2.9.1
  • pyflakes: 2.4.0 → 2.5.0
  • pygments: 2.12.0 → 2.13.0
  • pyparsing: none → 3.0.9
  • pyserial: none → 3.5
  • pytest: none → 7.1.2
  • qmk: 1.1.0 → 1.1.1
  • qmk-dotty-dict: 1.3.0.post1 → none (replaced by dotty-dict)
  • setuptools-scm: none → 7.0.5
  • tomli: none → 2.0.1
  • tomlkit: 0.7.0 → 0.11.4 (used only during build, not included into the resulting environment)
  • typing-extensions: none → 4.3.0
  • zipp: 3.8.0 → 3.8.1

Cc: @purcell @AstridYu @andresilva @evils

Types of Changes

  • Core
  • Bugfix
  • New feature
  • Enhancement/optimization
  • Keyboard (addition or update)
  • Keymap/layout/userspace (addition or update)
  • Documentation

Checklist

  • My code follows the code style of this project: C, Python
  • I have read the PR Checklist document and have made the appropriate changes.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • I have tested the changes and verified that they work and don't break anything (as well as I can manage).

@sigprof
shell.nix: Update tomlkit to 0.11.4 using a Nixpkgs overlay
d717cc1 
The used Nixpkgs snapshot contains `tomlkit` version 0.7.0, which is
affected by https://www.github.com/sdispater/tomlkit/issues/148; that
bug is triggered by `pyproject.toml` from `jsonschema` >= 4.11.0,
preventing the build of that module.

Just adding `tomlkit = "*"` to the `[tool.poetry.dev-dependencies]`
section of `nix/pyproject.toml` does not fix the `jsonschema` build,
because `makeRemoveSpecialDependenciesHook` inside `poetry2nix` is not
affected by `nix/pyproject.toml`.  Add a Nixpkgs overlay which updates
the `tomlkit` Python module globally, so that `poetry2nix` would also
use the updated version internally.
@sigprof
shell.nix: Bump poetry2nix to the most recent version
b95723e 
The new `poetry2nix` version includes overrides which are required for
recent versions of some Python packages (in particular, `jsonschema` and
`dotty-dict`).
@sigprof
shell.nix: Bump QMK CLI to 1.1.1; update other Python deps
3a8a84a 
Update `pyproject.toml` to match `requirements*.txt`:

- add `pyserial = "*"`
- replace `qmk-dotty-dict = "*"` with `dotty-dict = "*"` (qmk#18117, also
  required for compatibility with `qmk` 1.1.1, where this replacement
  had already been performed)

Add build dependencies of various Python modules to `pyproject.toml`:

- `hatchling`, `hatch-vcs`, `hatch-fancy-pypi-readme` (required by
  `jsonschema` >= 4.11.0)
- `pytest` (a newer version is required to solve the dependency conflict
  with the `hatchling` module due to the upper bound on `pluggy`)
- `flit-core` (a more recent version is required to build `tomli`)
- `poetry-core` (required for `dotty-dict` >= 1.3.1, and the version
  from Nixpkgs does not build on Darwin due to NixOS/nix#4758)

Update `poetry.lock` to use the most recent versions of Python modules.

The complete list of Python module updates as listed in `poetry.lock`
(note that other modules might be present in the Python environment,
e.g., if they come from Nixpkgs):

- `atomicwrites`: none -> 1.4.1 (but this module is not actually used,
  because the corresponding dependency of `pytest` is win32-only)
- `attrs`: 21.4.0 -> 22.1.0
- `colorama`: 0.4.4 -> 0.4.5
- `coverage`: 6.4 -> none
- `dotty-dict`: none -> 1.3.1 (used instead of `qmk-dotty-dict`)
- `editables`: none -> 0.3
- `flake8`: 4.0.1 -> 5.0.4
- `flake8-polyfill`: 1.0.2 -> none
- `flit-core`: none -> 3.7.1
- `hatch-fancy-pypi-readme`: none -> 22.3.0
- `hatch-vcs`: none -> 0.2.0
- `hatchling`: none -> 1.8.0
- `hjson`: 3.0.2 -> 3.1.0
- `importlib-resources`: 5.7.1 -> 5.9.0
- `iniconfig`: none -> 1.1.1
- `jsonschema`: 4.5.1 -> 4.14.0
- `mccabe`: 0.6.1 -> 0.7.0
- `nose2`: 0.11.0 -> 0.12.0
- `packaging`: none -> 21.3
- `pathspec`: none -> 0.9.0
- `pep8-naming`: 0.12.1 -> 0.13.2
- `pillow`: 9.1.1 -> 9.2.0
- `pkgutil-resolve-name`: none -> 1.3.10
- `pluggy`: none -> 1.0.0
- `poetry-core`: none -> 1.0.8
- `py`: none -> 1.11.0
- `pycodestyle`: 2.8.0 -> 2.9.1
- `pyflakes`: 2.4.0 -> 2.5.0
- `pygments`: 2.12.0 -> 2.13.0
- `pyparsing`: none -> 3.0.9
- `pyserial`: none -> 3.5
- `pytest`: none -> 7.1.2
- `qmk`: 1.1.0 -> 1.1.1
- `qmk-dotty-dict`: 1.3.0.post1 -> none (replaced by `dotty-dict`)
- `setuptools-scm`: none -> 7.0.5
- `tomli`: none -> 2.0.1
- `typing-extensions`: none -> 4.3.0
- `zipp`: 3.8.0 -> 3.8.1
@github-actions github-actions bot added the core label Aug 21, 2022
@sigprof sigprof mentioned this pull request Aug 21, 2022
Closed
14 tasks
evils
evils approved these changes Aug 22, 2022
View reviewed changes
Copy link

@evils evils left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

works for me, thanks :)

@zvecr zvecr merged commit 7c0e5ce into qmk:develop Aug 23, 2022
nolanseaton pushed a commit to nolanseaton/qmk_firmware that referenced this pull request Jan 23, 2023
@sigprof @nolanseaton
Nix shell updates for develop (qmk#18131)
b1044f1 
* shell.nix: Update `tomlkit` to 0.11.4 using a Nixpkgs overlay

The used Nixpkgs snapshot contains `tomlkit` version 0.7.0, which is
affected by https://www.github.com/sdispater/tomlkit/issues/148; that
bug is triggered by `pyproject.toml` from `jsonschema` >= 4.11.0,
preventing the build of that module.

Just adding `tomlkit = "*"` to the `[tool.poetry.dev-dependencies]`
section of `nix/pyproject.toml` does not fix the `jsonschema` build,
because `makeRemoveSpecialDependenciesHook` inside `poetry2nix` is not
affected by `nix/pyproject.toml`.  Add a Nixpkgs overlay which updates
the `tomlkit` Python module globally, so that `poetry2nix` would also
use the updated version internally.

* shell.nix: Bump `poetry2nix` to the most recent version

The new `poetry2nix` version includes overrides which are required for
recent versions of some Python packages (in particular, `jsonschema` and
`dotty-dict`).

* shell.nix: Bump QMK CLI to 1.1.1; update other Python deps

Update `pyproject.toml` to match `requirements*.txt`:

- add `pyserial = "*"`
- replace `qmk-dotty-dict = "*"` with `dotty-dict = "*"` (qmk#18117, also
  required for compatibility with `qmk` 1.1.1, where this replacement
  had already been performed)

Add build dependencies of various Python modules to `pyproject.toml`:

- `hatchling`, `hatch-vcs`, `hatch-fancy-pypi-readme` (required by
  `jsonschema` >= 4.11.0)
- `pytest` (a newer version is required to solve the dependency conflict
  with the `hatchling` module due to the upper bound on `pluggy`)
- `flit-core` (a more recent version is required to build `tomli`)
- `poetry-core` (required for `dotty-dict` >= 1.3.1, and the version
  from Nixpkgs does not build on Darwin due to NixOS/nix#4758)

Update `poetry.lock` to use the most recent versions of Python modules.

The complete list of Python module updates as listed in `poetry.lock`
(note that other modules might be present in the Python environment,
e.g., if they come from Nixpkgs):

- `atomicwrites`: none -> 1.4.1 (but this module is not actually used,
  because the corresponding dependency of `pytest` is win32-only)
- `attrs`: 21.4.0 -> 22.1.0
- `colorama`: 0.4.4 -> 0.4.5
- `coverage`: 6.4 -> none
- `dotty-dict`: none -> 1.3.1 (used instead of `qmk-dotty-dict`)
- `editables`: none -> 0.3
- `flake8`: 4.0.1 -> 5.0.4
- `flake8-polyfill`: 1.0.2 -> none
- `flit-core`: none -> 3.7.1
- `hatch-fancy-pypi-readme`: none -> 22.3.0
- `hatch-vcs`: none -> 0.2.0
- `hatchling`: none -> 1.8.0
- `hjson`: 3.0.2 -> 3.1.0
- `importlib-resources`: 5.7.1 -> 5.9.0
- `iniconfig`: none -> 1.1.1
- `jsonschema`: 4.5.1 -> 4.14.0
- `mccabe`: 0.6.1 -> 0.7.0
- `nose2`: 0.11.0 -> 0.12.0
- `packaging`: none -> 21.3
- `pathspec`: none -> 0.9.0
- `pep8-naming`: 0.12.1 -> 0.13.2
- `pillow`: 9.1.1 -> 9.2.0
- `pkgutil-resolve-name`: none -> 1.3.10
- `pluggy`: none -> 1.0.0
- `poetry-core`: none -> 1.0.8
- `py`: none -> 1.11.0
- `pycodestyle`: 2.8.0 -> 2.9.1
- `pyflakes`: 2.4.0 -> 2.5.0
- `pygments`: 2.12.0 -> 2.13.0
- `pyparsing`: none -> 3.0.9
- `pyserial`: none -> 3.5
- `pytest`: none -> 7.1.2
- `qmk`: 1.1.0 -> 1.1.1
- `qmk-dotty-dict`: 1.3.0.post1 -> none (replaced by `dotty-dict`)
- `setuptools-scm`: none -> 7.0.5
- `tomli`: none -> 2.0.1
- `typing-extensions`: none -> 4.3.0
- `zipp`: 3.8.0 -> 3.8.1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@purcell purcell purcell approved these changes

@zvecr zvecr zvecr approved these changes

@evils evils evils approved these changes

Assignees
No one assigned
Labels
core
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@sigprof @purcell @zvecr @evils