builtins.match regex support "broken" on aarch64-darwin #4758

andir · 2021-05-01T11:59:49Z

Describe the bug

I just tried to use poetry2nix on aarch64-darwin and the eval failed invalid regular expression. Apparently this is an issue with the aarch64-darwin build of Nix. I'm using Nixpkgs from NixOS/nixpkgs#105026.

The problem appears to only exist on aarch64-darwin and not on any of the platforms. Also nixUnstable vs stable nix doesn't make a difference.

aarch64-darwin

[aarch64-darwin] $ nix repl
Welcome to Nix version 2.3.10. Type :? for help.

nix-repl> builtins.match "^.*?(egg|tar|tar.bz2|tar.gz|tar.lz|tar.lzma|tar.xz|tbz|tgz|tlz|txz|whl|zip)" "asd"
error: invalid regular expression '^.*?(egg|tar|tar.bz2|tar.gz|tar.lz|tar.lzma|tar.xz|tbz|tgz|tlz|txz|whl|zip)', at (string):1:1

[aarch64-darwin] $ nix repl
Welcome to Nix version 2.4pre20210326_dd77f71. Type :? for help.

nix-repl> builtins.match "^.*?(egg|tar|tar.bz2|tar.gz|tar.lz|tar.lzma|tar.xz|tbz|tgz|tlz|txz|whl|zip)" "asd"
error: invalid regular expression '^.*?(egg|tar|tar.bz2|tar.gz|tar.lz|tar.lzma|tar.xz|tbz|tgz|tlz|txz|whl|zip)'

       at «string»:1:1:

            1| builtins.match "^.*?(egg|tar|tar.bz2|tar.gz|tar.lz|tar.lzma|tar.xz|tbz|tgz|tlz|txz|whl|zip)" "asd"
             | ^
            2|

x86_64-linux

[x86_64-linux] $ nix repl
Welcome to Nix version 2.3.10. Type :? for help.

nix-repl> builtins.match "^.*?(egg|tar|tar.bz2|tar.gz|tar.lz|tar.lzma|tar.xz|tbz|tgz|tlz|txz|whl|zip)" "asd"
null

nix-repl> builtins.currentSystem
"x86_64-linux"


[x86_64-linux] $ nix repl
Welcome to Nix version 2.4pre20210326_dd77f71. Type :? for help.

nix-repl> builtins.match "^.*?(egg|tar|tar.bz2|tar.gz|tar.lz|tar.lzma|tar.xz|tbz|tgz|tlz|txz|whl|zip)" "asd"
null

aarch64-linux

[aarch64-linux] $ nix repl
Welcome to Nix version 2.4pre20210326_dd77f71. Type :? for help.

nix-repl>  builtins.match "^.*?(egg|tar|tar.bz2|tar.gz|tar.lz|tar.lzma|tar.xz|tbz|tgz|tlz|txz|whl|zip)" "asd"
null

x86_64-darwin


[x86_64-darwin]$ nix repl
Welcome to Nix version 2.3.10. Type :? for help.

nix-repl> builtins.match "^.*?(egg|tar|tar.bz2|tar.gz|tar.lz|tar.lzma|tar.xz|tbz|tgz|tlz|txz|whl|zip)" "asd"
[ ]

Steps To Reproduce

On a aarch64-darwin system open the nix repl and try to eval builtins.match "^.*?(egg|tar|tar.bz2|tar.gz|tar.lz|tar.lzma|tar.xz|tbz|tgz|tlz|txz|whl|zip)" "asd"
You should see an invalid regex error now.

Expected behavior

The behavior should be the same across all the platforms.

nix-env --version output

nix-env (Nix) 2.4pre20210326_dd77f71

The text was updated successfully, but these errors were encountered:

vcunat · 2021-05-01T16:46:41Z

I've already heard of some practical cases where x86_64-darwin was stricter than x86_64-linux. Perhaps this report belongs in here? #1537

I certainly agree that the differences are unfortunate. Maybe we could at least have ofborg check evaluation on multiple platforms or something.

andir · 2021-05-01T18:28:31Z

Good find! I haven't seen that issue when I searched. The interesting piece is that depending on architecture I am compiling for it works on darwin. Perhaps we are using different c++ library implementation.

domenkozar · 2021-06-11T13:05:57Z

Isn't this an upstream issue of the regex lib?

stale · 2022-01-03T21:44:24Z

I marked this as stale due to inactivity. → More info

kamadorueda · 2022-03-11T04:03:15Z

I also found this problem on x86_64-darwin, and the fix was removing the ? in the wildcard .*?: on-nix/python@4d3d14f

Seems that adding or not the ? yields the same results (at least in my case), with the difference that it does not break

aakropotkin · 2022-05-05T03:08:10Z

I got bitten by this same issue this week. It was driving me absolutely nuts.

I finally decided I wasn't crazy after running similar side by side comparisons.

At the very least a note should be added to the docs.

Related issue is x{0,} not working. Basically you can't do non-greedy matches on Darwin.

domenkozar · 2022-05-05T11:15:48Z

@toonn could you take a look at this one?

toonn · 2022-05-13T11:05:57Z

So I think this issue started happening on x86_64-darwin too as mentioned by @kamadorueda, because of the bump of the Darwin stdenv to LLVM 11, from 7. Aarch64-darwin always used LLVM 11 and the corresponding libc++ so that's why the problem surfaced there earlier.

As far as a fix goes, the issue has been pinpointed by others to come down to a difference between the libstdc++ (GCC) and libc++ (LLVM) C++ standard library implementations. I think it's pretty undesirable to have the Nix language regular expression behavior differ based on which stdlib Nix was compiled with.

I'm not sure what the best solution is, a compatibility shim that proxies for what we arbitrarily choose is the "correct" stdlib implementation is probably too fragile. The other alternative, already mentioned on Discourse, would be using a third-party library like PCRE to back Nix's regular expression needs. This would likely be a backwards-incompatible change across both platforms though.

sigprof · 2022-07-12T15:20:14Z

Today the change apparently propagated to the Nix 2.10.0 binary for x86_64-darwin which gets installed when using https://nixos.org/nix/install — in particular, users of cachix/install-nix-action on x86_64-darwin will now get a Nix with a significantly different behavior from the previous versions.

Update `pyproject.toml` to match `requirements*.txt`: - add `pyserial = "*"` - replace `qmk-dotty-dict = "*"` with `dotty-dict = "*"` (qmk#18117, also required for compatibility with `qmk` 1.1.1, where this replacement had already been performed) Add build dependencies of various Python modules to `pyproject.toml`: - `hatchling`, `hatch-vcs`, `hatch-fancy-pypi-readme` (required by `jsonschema` >= 4.11.0) - `pytest` (a newer version is required to solve the dependency conflict with the `hatchling` module due to the upper bound on `pluggy`) - `flit-core` (a more recent version is required to build `tomli`) - `poetry-core` (required for `dotty-dict` >= 1.3.1, and the version from Nixpkgs does not build on Darwin due to NixOS/nix#4758) Update `poetry.lock` to use the most recent versions of Python modules. The complete list of Python module updates as listed in `poetry.lock` (note that other modules might be present in the Python environment, e.g., if they come from Nixpkgs): - `atomicwrites`: none -> 1.4.1 (but this module is not actually used, because the corresponding dependency of `pytest` is win32-only) - `attrs`: 21.4.0 -> 22.1.0 - `colorama`: 0.4.4 -> 0.4.5 - `coverage`: 6.4 -> none - `dotty-dict`: none -> 1.3.1 (used instead of `qmk-dotty-dict`) - `editables`: none -> 0.3 - `flake8`: 4.0.1 -> 5.0.4 - `flake8-polyfill`: 1.0.2 -> none - `flit-core`: none -> 3.7.1 - `hatch-fancy-pypi-readme`: none -> 22.3.0 - `hatch-vcs`: none -> 0.2.0 - `hatchling`: none -> 1.8.0 - `hjson`: 3.0.2 -> 3.1.0 - `importlib-resources`: 5.7.1 -> 5.9.0 - `iniconfig`: none -> 1.1.1 - `jsonschema`: 4.5.1 -> 4.14.0 - `mccabe`: 0.6.1 -> 0.7.0 - `nose2`: 0.11.0 -> 0.12.0 - `packaging`: none -> 21.3 - `pathspec`: none -> 0.9.0 - `pep8-naming`: 0.12.1 -> 0.13.2 - `pillow`: 9.1.1 -> 9.2.0 - `pkgutil-resolve-name`: none -> 1.3.10 - `pluggy`: none -> 1.0.0 - `poetry-core`: none -> 1.0.8 - `py`: none -> 1.11.0 - `pycodestyle`: 2.8.0 -> 2.9.1 - `pyflakes`: 2.4.0 -> 2.5.0 - `pygments`: 2.12.0 -> 2.13.0 - `pyparsing`: none -> 3.0.9 - `pyserial`: none -> 3.5 - `pytest`: none -> 7.1.2 - `qmk`: 1.1.0 -> 1.1.1 - `qmk-dotty-dict`: 1.3.0.post1 -> none (replaced by `dotty-dict`) - `setuptools-scm`: none -> 7.0.5 - `tomli`: none -> 2.0.1 - `typing-extensions`: none -> 4.3.0 - `zipp`: 3.8.0 -> 3.8.1

* shell.nix: Update `tomlkit` to 0.11.4 using a Nixpkgs overlay The used Nixpkgs snapshot contains `tomlkit` version 0.7.0, which is affected by https://www.github.com/sdispater/tomlkit/issues/148; that bug is triggered by `pyproject.toml` from `jsonschema` >= 4.11.0, preventing the build of that module. Just adding `tomlkit = "*"` to the `[tool.poetry.dev-dependencies]` section of `nix/pyproject.toml` does not fix the `jsonschema` build, because `makeRemoveSpecialDependenciesHook` inside `poetry2nix` is not affected by `nix/pyproject.toml`. Add a Nixpkgs overlay which updates the `tomlkit` Python module globally, so that `poetry2nix` would also use the updated version internally. * shell.nix: Bump `poetry2nix` to the most recent version The new `poetry2nix` version includes overrides which are required for recent versions of some Python packages (in particular, `jsonschema` and `dotty-dict`). * shell.nix: Bump QMK CLI to 1.1.1; update other Python deps Update `pyproject.toml` to match `requirements*.txt`: - add `pyserial = "*"` - replace `qmk-dotty-dict = "*"` with `dotty-dict = "*"` (#18117, also required for compatibility with `qmk` 1.1.1, where this replacement had already been performed) Add build dependencies of various Python modules to `pyproject.toml`: - `hatchling`, `hatch-vcs`, `hatch-fancy-pypi-readme` (required by `jsonschema` >= 4.11.0) - `pytest` (a newer version is required to solve the dependency conflict with the `hatchling` module due to the upper bound on `pluggy`) - `flit-core` (a more recent version is required to build `tomli`) - `poetry-core` (required for `dotty-dict` >= 1.3.1, and the version from Nixpkgs does not build on Darwin due to NixOS/nix#4758) Update `poetry.lock` to use the most recent versions of Python modules. The complete list of Python module updates as listed in `poetry.lock` (note that other modules might be present in the Python environment, e.g., if they come from Nixpkgs): - `atomicwrites`: none -> 1.4.1 (but this module is not actually used, because the corresponding dependency of `pytest` is win32-only) - `attrs`: 21.4.0 -> 22.1.0 - `colorama`: 0.4.4 -> 0.4.5 - `coverage`: 6.4 -> none - `dotty-dict`: none -> 1.3.1 (used instead of `qmk-dotty-dict`) - `editables`: none -> 0.3 - `flake8`: 4.0.1 -> 5.0.4 - `flake8-polyfill`: 1.0.2 -> none - `flit-core`: none -> 3.7.1 - `hatch-fancy-pypi-readme`: none -> 22.3.0 - `hatch-vcs`: none -> 0.2.0 - `hatchling`: none -> 1.8.0 - `hjson`: 3.0.2 -> 3.1.0 - `importlib-resources`: 5.7.1 -> 5.9.0 - `iniconfig`: none -> 1.1.1 - `jsonschema`: 4.5.1 -> 4.14.0 - `mccabe`: 0.6.1 -> 0.7.0 - `nose2`: 0.11.0 -> 0.12.0 - `packaging`: none -> 21.3 - `pathspec`: none -> 0.9.0 - `pep8-naming`: 0.12.1 -> 0.13.2 - `pillow`: 9.1.1 -> 9.2.0 - `pkgutil-resolve-name`: none -> 1.3.10 - `pluggy`: none -> 1.0.0 - `poetry-core`: none -> 1.0.8 - `py`: none -> 1.11.0 - `pycodestyle`: 2.8.0 -> 2.9.1 - `pyflakes`: 2.4.0 -> 2.5.0 - `pygments`: 2.12.0 -> 2.13.0 - `pyparsing`: none -> 3.0.9 - `pyserial`: none -> 3.5 - `pytest`: none -> 7.1.2 - `qmk`: 1.1.0 -> 1.1.1 - `qmk-dotty-dict`: 1.3.0.post1 -> none (replaced by `dotty-dict`) - `setuptools-scm`: none -> 7.0.5 - `tomli`: none -> 2.0.1 - `typing-extensions`: none -> 4.3.0 - `zipp`: 3.8.0 -> 3.8.1

The *? doesn't work on aarch64-darwin NixOS/nix#4758 I also don't think it does anything on linux as far as I can tell, even though it is permitted, so it was an oversight to have it there in the first place.

nixos-discourse · 2022-11-25T09:50:48Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-40/23480/1

* shell.nix: Update `tomlkit` to 0.11.4 using a Nixpkgs overlay The used Nixpkgs snapshot contains `tomlkit` version 0.7.0, which is affected by https://www.github.com/sdispater/tomlkit/issues/148; that bug is triggered by `pyproject.toml` from `jsonschema` >= 4.11.0, preventing the build of that module. Just adding `tomlkit = "*"` to the `[tool.poetry.dev-dependencies]` section of `nix/pyproject.toml` does not fix the `jsonschema` build, because `makeRemoveSpecialDependenciesHook` inside `poetry2nix` is not affected by `nix/pyproject.toml`. Add a Nixpkgs overlay which updates the `tomlkit` Python module globally, so that `poetry2nix` would also use the updated version internally. * shell.nix: Bump `poetry2nix` to the most recent version The new `poetry2nix` version includes overrides which are required for recent versions of some Python packages (in particular, `jsonschema` and `dotty-dict`). * shell.nix: Bump QMK CLI to 1.1.1; update other Python deps Update `pyproject.toml` to match `requirements*.txt`: - add `pyserial = "*"` - replace `qmk-dotty-dict = "*"` with `dotty-dict = "*"` (qmk#18117, also required for compatibility with `qmk` 1.1.1, where this replacement had already been performed) Add build dependencies of various Python modules to `pyproject.toml`: - `hatchling`, `hatch-vcs`, `hatch-fancy-pypi-readme` (required by `jsonschema` >= 4.11.0) - `pytest` (a newer version is required to solve the dependency conflict with the `hatchling` module due to the upper bound on `pluggy`) - `flit-core` (a more recent version is required to build `tomli`) - `poetry-core` (required for `dotty-dict` >= 1.3.1, and the version from Nixpkgs does not build on Darwin due to NixOS/nix#4758) Update `poetry.lock` to use the most recent versions of Python modules. The complete list of Python module updates as listed in `poetry.lock` (note that other modules might be present in the Python environment, e.g., if they come from Nixpkgs): - `atomicwrites`: none -> 1.4.1 (but this module is not actually used, because the corresponding dependency of `pytest` is win32-only) - `attrs`: 21.4.0 -> 22.1.0 - `colorama`: 0.4.4 -> 0.4.5 - `coverage`: 6.4 -> none - `dotty-dict`: none -> 1.3.1 (used instead of `qmk-dotty-dict`) - `editables`: none -> 0.3 - `flake8`: 4.0.1 -> 5.0.4 - `flake8-polyfill`: 1.0.2 -> none - `flit-core`: none -> 3.7.1 - `hatch-fancy-pypi-readme`: none -> 22.3.0 - `hatch-vcs`: none -> 0.2.0 - `hatchling`: none -> 1.8.0 - `hjson`: 3.0.2 -> 3.1.0 - `importlib-resources`: 5.7.1 -> 5.9.0 - `iniconfig`: none -> 1.1.1 - `jsonschema`: 4.5.1 -> 4.14.0 - `mccabe`: 0.6.1 -> 0.7.0 - `nose2`: 0.11.0 -> 0.12.0 - `packaging`: none -> 21.3 - `pathspec`: none -> 0.9.0 - `pep8-naming`: 0.12.1 -> 0.13.2 - `pillow`: 9.1.1 -> 9.2.0 - `pkgutil-resolve-name`: none -> 1.3.10 - `pluggy`: none -> 1.0.0 - `poetry-core`: none -> 1.0.8 - `py`: none -> 1.11.0 - `pycodestyle`: 2.8.0 -> 2.9.1 - `pyflakes`: 2.4.0 -> 2.5.0 - `pygments`: 2.12.0 -> 2.13.0 - `pyparsing`: none -> 3.0.9 - `pyserial`: none -> 3.5 - `pytest`: none -> 7.1.2 - `qmk`: 1.1.0 -> 1.1.1 - `qmk-dotty-dict`: 1.3.0.post1 -> none (replaced by `dotty-dict`) - `setuptools-scm`: none -> 7.0.5 - `tomli`: none -> 2.0.1 - `typing-extensions`: none -> 4.3.0 - `zipp`: 3.8.0 -> 3.8.1

andir added the bug label May 1, 2021

andir changed the title ~~builtins.match regex support "broken" on aarch64-darwk~~ builtins.match regex support "broken" on aarch64-darwin May 1, 2021

andir mentioned this issue May 1, 2021

Native support for Apple Silicon NixOS/nixpkgs#105026

Merged

21 tasks

hlolli mentioned this issue May 31, 2021

fix regex for darwin-aarch64 compatability nix-community/poetry2nix#313

Merged

stale bot added the stale label Jan 3, 2022

puppe mentioned this issue Jan 26, 2022

Invalid regular expression breaks kresd NixOS module NixOS/nixpkgs#156861

Closed

stale bot removed the stale label Mar 11, 2022

domenkozar added the macos Nix on macOS, aka OS X, aka darwin label May 5, 2022

sigprof mentioned this issue Jul 13, 2022

shell.nix: Fix compatibility with Nix >= 2.10.0 on x86_64-darwin qmk/qmk_firmware#17673

Merged

14 tasks

yorickvP mentioned this issue Nov 23, 2022

Use PCRE for builtins.match and builtins.split #7336

Closed

yorickvP mentioned this issue Feb 6, 2023

Switch from std::regex to boost::regex #7762

Merged

7 tasks

roberth added the language The Nix expression language; parser, interpreter, primops, evaluation, etc label Oct 23, 2023

thufschmitt closed this as completed in #7762 Nov 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

builtins.match regex support "broken" on aarch64-darwin #4758

builtins.match regex support "broken" on aarch64-darwin #4758

andir commented May 1, 2021

vcunat commented May 1, 2021

andir commented May 1, 2021

domenkozar commented Jun 11, 2021

stale bot commented Jan 3, 2022

kamadorueda commented Mar 11, 2022

aakropotkin commented May 5, 2022 •

edited

domenkozar commented May 5, 2022

toonn commented May 13, 2022

sigprof commented Jul 12, 2022

nixos-discourse commented Nov 25, 2022

builtins.match regex support "broken" on aarch64-darwin #4758

builtins.match regex support "broken" on aarch64-darwin #4758

Comments

andir commented May 1, 2021

aarch64-darwin

x86_64-linux

aarch64-linux

x86_64-darwin

vcunat commented May 1, 2021

andir commented May 1, 2021

domenkozar commented Jun 11, 2021

stale bot commented Jan 3, 2022

kamadorueda commented Mar 11, 2022

aakropotkin commented May 5, 2022 • edited

domenkozar commented May 5, 2022

toonn commented May 13, 2022

sigprof commented Jul 12, 2022

nixos-discourse commented Nov 25, 2022

aakropotkin commented May 5, 2022 •

edited