Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

C API: Move limited C API tests from _testcapi to a new _testlimitedcapi extension #116417

Closed
vstinner opened this issue Mar 6, 2024 · 6 comments
Closed

C API: Move limited C API tests from _testcapi to a new _testlimitedcapi extension #116417

vstinner opened this issue Mar 6, 2024 · 6 comments

Comments

@vstinner
Copy link
Member

vstinner commented Mar 6, 2024 •
edited by bedevere-app bot

Currently, the _testcapi C extension is partially built with the limited C API and partially with the non-limited C API. It can lead to some confusion: which API is being tested?

I proposed to create a new _testlimitedcapi extension which is only built with the limited C API. It can only access to the limited C API and so there is a lower risk to test the non-limited C API by mistake.

Linked PRs
vstinner added a commit to vstinner/cpython that referenced this issue Mar 6, 2024
@vstinner
pythongh-116417: Add _testlimitedcapi C extension
af160ec 
Add a new C extension "_testlimitedcapi" which is only built with the
limited C API.

Move heaptype_relative.c and vectorcall_limited.c from
Modules/_testcapi/ to Modules/_testlimitedcapi/.
@bedevere-app bedevere-app bot mentioned this issue Mar 6, 2024
Merged
vstinner added a commit that referenced this issue Mar 7, 2024
@vstinner @erlend-aasland
gh-116417: Add _testlimitedcapi C extension (#116419)
d9bcdda 
Add a new C extension "_testlimitedcapi" which is only built with the
limited C API.

Move heaptype_relative.c and vectorcall_limited.c from
Modules/_testcapi/ to Modules/_testlimitedcapi/.

* configure: add _testlimitedcapi test extension.
* Update generate_stdlib_module_names.py.
* Update make check-c-globals.

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
vstinner added a commit to vstinner/cpython that referenced this issue Mar 10, 2024
@vstinner
pythongh-116417: Move limited C API tests to _testlimitedcapi
3b01457 
* Move bytes.c and sys.c from Modules/_testcapi/ to
  Modules/_testlimitedcapi/.
* Split some tests in two files, limited and non-limited:

  * abstract.c
  * dict.c
  * list.c
  * set.c

* Copy util.h to Modules/_testlimitedcapi/.
* Update test_capi.
@bedevere-app bedevere-app bot mentioned this issue Mar 10, 2024
Closed
vstinner added a commit to vstinner/cpython that referenced this issue Mar 10, 2024
@vstinner
pythongh-116417: Avoid PyFloat_AS_DOUBLE() in AC limited C API
39e6d67 
Argument Clinic no longer calls PyFloat_AS_DOUBLE() when the usage of
the limited C API is requested.
vstinner added a commit to vstinner/cpython that referenced this issue Mar 10, 2024
@vstinner
pythongh-116417: Avoid PyFloat_AS_DOUBLE() in AC limited C API
d3b6276 
Argument Clinic no longer calls PyFloat_AS_DOUBLE() when the usage of
the limited C API is requested.
@bedevere-app bedevere-app bot mentioned this issue Mar 10, 2024
Merged
vstinner added a commit that referenced this issue Mar 10, 2024
@vstinner
gh-116417: Avoid PyFloat_AS_DOUBLE() in AC limited C API (#116568)
729bfb3 
Argument Clinic no longer calls PyFloat_AS_DOUBLE() when the usage of
the limited C API is requested.
vstinner added a commit to vstinner/cpython that referenced this issue Mar 10, 2024
@vstinner
pythongh-116417: Fix make check-c-globals for _testlimitedcapi
256fdc6 
* Remove unused '_testcapimodule' global in Modules/_testcapi/unicode.c.
* Update c-analyzer to not use the internal C API in
  _testlimitedcapi.c.
@bedevere-app bedevere-app bot mentioned this issue Mar 10, 2024
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Mar 10, 2024
@vstinner
pythongh-116417: Move 4 limited C API test files to _testlimitedcapi
8569974 
Move the following files from Modules/_testcapi/ to
Modules/_testlimitedcapi/:

* bytearray.c
* bytes.c
* pyos.c
* sys.c

Update related test_capi tests.

Copy Modules/_testcapi/util.h to Modules/_testlimitedcapi/util.h.
@bedevere-app bedevere-app bot mentioned this issue Mar 10, 2024
Merged
vstinner added a commit that referenced this issue Mar 10, 2024
@vstinner
gh-116417: Fix make check-c-globals for _testlimitedcapi (#116570)
c5fa796 
* Remove unused '_testcapimodule' global in Modules/_testcapi/unicode.c.
* Update c-analyzer to not use the internal C API in
  _testlimitedcapi.c.
erlend-aasland added a commit to erlend-aasland/cpython that referenced this issue Mar 10, 2024
@erlend-aasland
pythongh-116417: Argument Clinic: test generated Limited C API code f…
e03947b 
…or float args
@bedevere-app bedevere-app bot mentioned this issue Mar 10, 2024
Merged
erlend-aasland added a commit that referenced this issue Mar 11, 2024
@erlend-aasland
gh-116417: Argument Clinic: test generated Limited C API code for flo…
8d7fde6 
…at args (#116573)
vstinner added a commit to vstinner/cpython that referenced this issue Mar 11, 2024
@vstinner
pythongh-116417: Move 4 limited C API test files to _testlimitedcapi
29341e6 
Move the following files from Modules/_testcapi/ to
Modules/_testlimitedcapi/:

* bytearray.c
* bytes.c
* pyos.c
* sys.c

Changes:

* Replace PyBytes_AS_STRING() with PyBytes_AsString().
* Replace PyBytes_GET_SIZE() with PyBytes_Size().
* Update related test_capi tests.
* Copy Modules/_testcapi/util.h to Modules/_testlimitedcapi/util.h.
vstinner added a commit that referenced this issue Mar 11, 2024
@vstinner
gh-116417: Move 4 limited C API test files to _testlimitedcapi (#116571)
1cc02ca 
Move the following files from Modules/_testcapi/ to
Modules/_testlimitedcapi/:

* bytearray.c
* bytes.c
* pyos.c
* sys.c

Changes:

* Replace PyBytes_AS_STRING() with PyBytes_AsString().
* Replace PyBytes_GET_SIZE() with PyBytes_Size().
* Update related test_capi tests.
* Copy Modules/_testcapi/util.h to Modules/_testlimitedcapi/util.h.
vstinner added a commit to vstinner/cpython that referenced this issue Mar 11, 2024
@vstinner
pythongh-116417: Build _testinternalcapi with limited C API version 3.5
580a4ba
@bedevere-app bedevere-app bot mentioned this issue Mar 11, 2024
Merged
vstinner added a commit that referenced this issue Mar 11, 2024
@vstinner
gh-116417: Build _testinternalcapi with limited C API version 3.5 (#1…
546eb7a 
…16598)
vstinner added a commit to vstinner/cpython that referenced this issue Mar 11, 2024
@vstinner
pythongh-116417: Move limited C API list.c tests to _testlimitedcapi
2f256df
vstinner added a commit to vstinner/cpython that referenced this issue Mar 11, 2024
@vstinner
pythongh-116417: Move limited C API list.c tests to _testlimitedcapi
d0386f1 
Split list.c and set.c tests of _testcapi into two parts: limited C
API tests in _testlimitedcapi and non-limited C API tests in
_testcapi.
@bedevere-app bedevere-app bot mentioned this issue Mar 11, 2024
Merged
@vstinner
Copy link
Member Author

Note to myself: pyos.c has no associated Python test!?

@serhiy-storchaka
Copy link
Member

What Limited C API? Shouldn't there be multiple separately compiled extensions, one for each version?

@vstinner
Copy link
Member Author

What Limited C API? Shouldn't there be multiple separately compiled extensions, one for each version?

I only plan to test a single limited C API version per C file. IMO it's already better than not testing the limited C API.

  • Currently, Modules/_testlimitedcapi/parts.h uses #define Py_LIMITED_API 0x03050000 by default, Python 3.5.
  • heaptype_relative.c needs the limited C API version 3.12 for PyType_FromMetaclass().
  • vectorcall_limited.c needs the limited C API version 3.12 for PyObject_Vectorcall().

@serhiy-storchaka
Copy link
Member

What is the purpose of testing the Limited C API separately? I thought that it is to check that these functions are available in the specified Limited C API version, but if we use higher version, this does not work.

There are many new functions in 3.13, so you would need to bump the version to 3.13.

PySlice_GetIndicesEx() is actually defined differently depending on Py_LIMITED_API.

@vstinner
Copy link
Member Author

What is the purpose of testing the Limited C API separately?

We made mistakes quite often in the past in the limited C API. The idea is to test our limited C API to make sure that it works as expected. For example, the "trashcan" API was exposed in the limited C API but it didn't work because it accessed a PyThreadState member which is not accessible in the limited C API. For me, it's awkward that we missed such obvious bug :-(

I thought that it is to check that these functions are available in the specified Limited C API version, but if we use higher version, this does not work. There are many new functions in 3.13, so you would need to bump the version to 3.13.

I don't worry much about the exact tested limited C API.

PySlice_GetIndicesEx() is actually defined differently depending on Py_LIMITED_API.

Right, more and more parts the limited C API have different implementations depending on the implementation. One example is Py_INCREF() / Py_DECREF() which is either implemented as static inline functions or implemented as function calls.

If we build C extensions with different limited C API versions, it's more likely to catch bugs.

Maybe later we can even test the same API on multiple limited C API versions. But I propose to move step by step. First, decide which tests are tested with the limited C API or not.

I propose to tests basically everything with the limited C API, unless the API is not available in the limited C API. That's why this PR proposes to split tests between limited and non-limited tests.

By the way, set.c is already split in two parts between public and internal tests!

adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@vstinner @erlend-aasland @adorilson
pythongh-116417: Add _testlimitedcapi C extension (python#116419)
79af108 
Add a new C extension "_testlimitedcapi" which is only built with the
limited C API.

Move heaptype_relative.c and vectorcall_limited.c from
Modules/_testcapi/ to Modules/_testlimitedcapi/.

* configure: add _testlimitedcapi test extension.
* Update generate_stdlib_module_names.py.
* Update make check-c-globals.

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@vstinner @adorilson
pythongh-116417: Avoid PyFloat_AS_DOUBLE() in AC limited C API (pytho…
b5092a2 
…n#116568)

Argument Clinic no longer calls PyFloat_AS_DOUBLE() when the usage of
the limited C API is requested.
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@vstinner @adorilson
pythongh-116417: Fix make check-c-globals for _testlimitedcapi (pytho…
ca8b8bb 
…n#116570)

* Remove unused '_testcapimodule' global in Modules/_testcapi/unicode.c.
* Update c-analyzer to not use the internal C API in
  _testlimitedcapi.c.
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@erlend-aasland @adorilson
pythongh-116417: Argument Clinic: test generated Limited C API code f…
493b403 
…or float args (python#116573)
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@vstinner @adorilson
pythongh-116417: Move 4 limited C API test files to _testlimitedcapi (p…
236233a 
…ython#116571)

Move the following files from Modules/_testcapi/ to
Modules/_testlimitedcapi/:

* bytearray.c
* bytes.c
* pyos.c
* sys.c

Changes:

* Replace PyBytes_AS_STRING() with PyBytes_AsString().
* Replace PyBytes_GET_SIZE() with PyBytes_Size().
* Update related test_capi tests.
* Copy Modules/_testcapi/util.h to Modules/_testlimitedcapi/util.h.
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@vstinner @adorilson
pythongh-116417: Build _testinternalcapi with limited C API version 3…
7a6d35c 
….5 (python#116598)
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@vstinner @adorilson
pythongh-116417: Move limited C API list.c tests to _testlimitedcapi (…
f7e5df1 
…python#116602)

Split list.c and set.c tests of _testcapi into two parts: limited C
API tests in _testlimitedcapi and non-limited C API tests in
_testcapi.
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@vstinner @adorilson
pythongh-116417: Fix WASI build of _testlimitedcapi (python#116974)
e56234d 
Use different function names between _testcapi and _testlimitedcapi
to not confuse the WASI linker.
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@vstinner @adorilson
pythongh-116417: Move limited C API abstract.c tests to _testlimitedc…
d924301 
…api (python#116986)

Split abstract.c and float.c tests of _testcapi into two parts:
limited C API tests in _testlimitedcapi and non-limited C API tests
in _testcapi.

Update test_bytes and test_class.
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@vstinner @adorilson
pythongh-116417: Move limited C API unicode.c tests to _testlimitedca…
be59d32 
…pi (python#116993)

Split unicode.c tests of _testcapi into two parts: limited C API
tests in _testlimitedcapi and non-limited C API tests in _testcapi.

Update test_codecs.
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@vstinner @adorilson
pythongh-116417: Move limited C API long.c tests to _testlimitedcapi (p…
89e2285 
…ython#117001)

* Split long.c tests of _testcapi into two parts: limited C API tests
  in _testlimitedcapi and non-limited C API tests in _testcapi.
* Move testcapi_long.h from Modules/_testcapi/ to
  Modules/_testlimitedcapi/.
* Add MODULE__TESTLIMITEDCAPI_DEPS to Makefile.pre.in.
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@vstinner @adorilson
pythongh-116417: Move limited C API dict.c tests to _testlimitedcapi (p…
a493183 
…ython#117006)

Split dict.c tests of _testcapi into two parts: limited C API tests
in _testlimitedcapi and non-limited C API tests in _testcapi.
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@vstinner @adorilson
pythongh-116417: Move limited C API complex.c tests to _testlimitedca…
2cdf499 
…pi (python#117014)

Split complex.c tests of _testcapi into two parts: limited C API
tests in _testlimitedcapi and non-limited C API tests in _testcapi.
This was referenced Apr 8, 2024
Closed
Merged
serhiy-storchaka added a commit that referenced this issue Apr 8, 2024
@serhiy-storchaka
gh-117642: Fix PEP 737 implementation (GH-117643)
24a2bd0 
* Fix implementation of %#T and %#N (they were implemented as %T# and
  %N#).
* Restore tests removed in gh-116417.
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@vstinner @erlend-aasland @diegorusso
pythongh-116417: Add _testlimitedcapi C extension (python#116419)
98642ca 
Add a new C extension "_testlimitedcapi" which is only built with the
limited C API.

Move heaptype_relative.c and vectorcall_limited.c from
Modules/_testcapi/ to Modules/_testlimitedcapi/.

* configure: add _testlimitedcapi test extension.
* Update generate_stdlib_module_names.py.
* Update make check-c-globals.

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@vstinner @diegorusso
pythongh-116417: Avoid PyFloat_AS_DOUBLE() in AC limited C API (pytho…
7a97ca7 
…n#116568)

Argument Clinic no longer calls PyFloat_AS_DOUBLE() when the usage of
the limited C API is requested.
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@vstinner @diegorusso
pythongh-116417: Fix make check-c-globals for _testlimitedcapi (pytho…
56b307f 
…n#116570)

* Remove unused '_testcapimodule' global in Modules/_testcapi/unicode.c.
* Update c-analyzer to not use the internal C API in
  _testlimitedcapi.c.
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@erlend-aasland @diegorusso
pythongh-116417: Argument Clinic: test generated Limited C API code f…
1c4c950 
…or float args (python#116573)
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@vstinner @diegorusso
pythongh-116417: Move 4 limited C API test files to _testlimitedcapi (p…
40ca5e7 
…ython#116571)

Move the following files from Modules/_testcapi/ to
Modules/_testlimitedcapi/:

* bytearray.c
* bytes.c
* pyos.c
* sys.c

Changes:

* Replace PyBytes_AS_STRING() with PyBytes_AsString().
* Replace PyBytes_GET_SIZE() with PyBytes_Size().
* Update related test_capi tests.
* Copy Modules/_testcapi/util.h to Modules/_testlimitedcapi/util.h.
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@vstinner @diegorusso
pythongh-116417: Build _testinternalcapi with limited C API version 3…
cf33c8f 
….5 (python#116598)
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@vstinner @diegorusso
pythongh-116417: Move limited C API list.c tests to _testlimitedcapi (…
22b2a83 
…python#116602)

Split list.c and set.c tests of _testcapi into two parts: limited C
API tests in _testlimitedcapi and non-limited C API tests in
_testcapi.
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@vstinner @diegorusso
pythongh-116417: Fix WASI build of _testlimitedcapi (python#116974)
ceabb26 
Use different function names between _testcapi and _testlimitedcapi
to not confuse the WASI linker.
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@vstinner @diegorusso
pythongh-116417: Move limited C API abstract.c tests to _testlimitedc…
9727d27 
…api (python#116986)

Split abstract.c and float.c tests of _testcapi into two parts:
limited C API tests in _testlimitedcapi and non-limited C API tests
in _testcapi.

Update test_bytes and test_class.
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@vstinner @diegorusso
pythongh-116417: Move limited C API unicode.c tests to _testlimitedca…
835d28e 
…pi (python#116993)

Split unicode.c tests of _testcapi into two parts: limited C API
tests in _testlimitedcapi and non-limited C API tests in _testcapi.

Update test_codecs.
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@vstinner @diegorusso
pythongh-116417: Move limited C API long.c tests to _testlimitedcapi (p…
ac4f5c4 
…ython#117001)

* Split long.c tests of _testcapi into two parts: limited C API tests
  in _testlimitedcapi and non-limited C API tests in _testcapi.
* Move testcapi_long.h from Modules/_testcapi/ to
  Modules/_testlimitedcapi/.
* Add MODULE__TESTLIMITEDCAPI_DEPS to Makefile.pre.in.
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@vstinner @diegorusso
pythongh-116417: Move limited C API dict.c tests to _testlimitedcapi (p…
a753d1c 
…ython#117006)

Split dict.c tests of _testcapi into two parts: limited C API tests
in _testlimitedcapi and non-limited C API tests in _testcapi.
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@vstinner @diegorusso
pythongh-116417: Move limited C API complex.c tests to _testlimitedca…
34fac11 
…pi (python#117014)

Split complex.c tests of _testcapi into two parts: limited C API
tests in _testlimitedcapi and non-limited C API tests in _testcapi.
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@serhiy-storchaka @diegorusso
pythongh-117642: Fix PEP 737 implementation (pythonGH-117643)
7bfeb8b 
* Fix implementation of %#T and %#N (they were implemented as %T# and
  %N#).
* Restore tests removed in pythongh-116417.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vstinner @serhiy-storchaka