compiled

Compiled variants of the Python standard library.

What is this project exactly?

Pure Python modules in the standard library can be a speed bottleneck sometimes, this package aims to provide "compiled" variants of the pure Python standard library modules, which are somewhere between 2-4x faster than the builtin ones.

Installation

pip install compiled

✨ This will install the pycompile CLI script as well.

Usage

Say your program asd.py looks like this:

import tomllib
from difflib import get_close_matches

# ... rest of the code

You can use the bundled pycompile script to turn those imports into the "compiled" variants:

$ pycompile asd.py
✨ Rewrote asd.py with compiled imports.

$ cat asd.py
import compiled.tomllib as tomllib
from compiled.difflib import get_close_matches

# ... rest of the code

With a real world program using tomllib and difflib, we get the following difference in speed:

Program	Pure Python time	compiled time	Speedup %
hashes.py	1.907 seconds	1.028 seconds	85.5% faster

How does it work?

It uses this awesome piece of software called mypyc which takes type-checked Python code and compiles it to C code, which is generally faster.

Simply making sure that the Python module is statically type checked is 95% of the work needed to be done to make the project work. The rest is just glue code to test the compiled code and package it.

Supported libraries

Library	Supported
tomllib	✅
difflib	✅
asyncio	⬜️
urllib	⬜️
zipfile	⬜️
argparse	⬜️
uuid	⬜️

PRs type checking other Python modules are welcome! Check the How to add a library section on the exact steps.

Local Development / Testing

• Create and activate a virtual environment.

• Run pip install mypy, as mypy[c] is the only dependency.

• Scripts to test, build and package standard libraries are present in build.py:

  $ ./build.py test tomllib
  .............
  ----------------------------------------------------------------------
  Ran 13 tests in 0.006s
  OK
  
  $ ./build.py mypy tomllib
  Success: no issues found in 4 source files
  
  $ ./build.py package
  [...]
  ✨Built ./build/dist/compiled-0.2.1-cp311-cp311-macosx_13_0_arm64.whl

• Run pytest compiled_tests to run tests.

How to add a library

Say you wanted to add the heapq library to the project. The steps will be as follows:

• verify ./build.py test heapq passes. This will run the CPython test suite on the existing code.

• Tweak the code and add annotations until ./build.py mypy heapq passes.

  Verify ./build.py test heapq still passes.

• Tweak the code until ./build.py mypyc heapq passes.

  Verify ./build.py test heapq still passes.

• Run ./build.py test_compiled heapq to ensure the compiled version still passes all tests.