v1.21.0: U-module renaming, deflate module, IDF 5, board variants and Pico-W BLE

[dpgeorge]

This release of MicroPython sees the renaming of built-in modules to remove the u-prefix, a new deflate module with optional compression support, the introduction of board variants, switching of the esp32 port to use IDF 5 together with improved heap management, support for BLE on RPi Pico W boards, and STM32H5xx support. The project is also now using codespell and ruff to improve code quality. New boards added in this release are: ARDUINO_NANO_ESP32 and UM_NANOS3 (esp32 port), ADAFRUIT_METRO_M7 (mimxrt port), ARDUINO_PORTENTA_C33 and VK_RA6M5 (renesas-ra port), ADAFRUIT_METRO_M4_EXPRESS (samd port), NUCLEO_L4A6ZG and STM32H573I_DK (stm32 port).

The renaming of built-in modules to remove the u-prefix -- for example utime becomes time, uasyncio becomes asyncio -- is done to improve compatibility with CPython and eliminate confusion about whether to import the u-version or the non-u-version of the name. Now, one should just always import the non-u-version and no longer think about u-naming. The only remaining module with a u-prefix is uctypes because it is not compatible with the CPython ctypes modules. The following modules are still available via their u-names for backwards compatibility: array, asyncio, binascii, bluetooth, collections, cryptolib, errno, hashlib, heapq, io, json, machine, os, platform, random, re, select, socket, ssl, struct, time, websocket. These modules (except for asyncio) are also extensible and can be overridden by a file with the same name, eg time.py. To force the import of a built-in, one must first clear (and subsequently restore) sys.path; the ability to write to the sys.path attribute has also been implemented in this release. For further information see the discussion at #9018.

Furthermore, importing itself has been tweaked and optimised, and importing of built-ins no longer touch the filesystem, which makes a typical built-in import take ~0.15ms rather than 3-5ms. For modules that fail to import, they are now removed from sys.modules, allowing the import to be attempted again. This required adding "NLR jump callbacks" to efficiently run some code if an exception is raised.

Additional improvements to the core runtime include support for conversion specifiers in f-strings (e.g. !r), speeding up of incremental GC cycles by tracking the last used block, addition of a new MICROPY_GC_SPLIT_HEAP_AUTO "auto grow heap" mode and support for pad bytes in struct format. The documentation, examples and test have also seen general improvements and additions.

For the extended modules, the zlib C module has been removed and replaced with a new MicroPython-specific deflate module and DeflateIO class that is optimised to provide efficient streaming compression and decompression. The zlib (and gzip) modules are now implemented in pure Python on top of the deflate module. The timeq module has been removed, it existed only for a previous version of (u)asyncio and is no longer used. In the ssl module, SSLContext has been added to be more compatible with CPython. The select module now supports using system/POSIX poll when possible and the unix port now uses this implementation instead of its own one. That means the unix port can now select/poll on custom Python objects that implement the appropriate ioctl. The socket module now supports SO_BROADCAST, and sys.std*.buffer objects now exist on unix. There is also a new esp-hosted network driver for external ESP32-based network coprocessors.

mpy-cross now allows reading source files from stdin and writing compiled code to stdout.

The esp8266 and esp32 ports add support for the Espressif ESP-NOW protocol, and the ability to set/get the power saving mode of the WLAN hardware. The esp8266 port adds board-variant support, combining all boards into a single ESP8266_GENERIC with FLASH_512K, FLASH_1M and OTA variants.

The esp32 port has now switched exclusively to ESP-IDF 5, and all existing components have been updated to work with this new IDF (except esp32.hall_sensor() which has been removed). The FSM ULP has been enabled for S2 and S3 chips, sockets now support SO_BINDTODEVICE, and board-named pins and the Pin.board dict have been implemented. The MicroPython heap has been reworked on this port to support the large variety of memory configurations: it now starts at 64kbytes and automatically grows as needed, with new segments allocated from the IDF heap. This means that boards with SPIRAM have much faster GC collection times if only a small amount of RAM is used, all available RAM can be used if needed, and the IDF has access to enough RAM for things like SSL sockets. For more information see #12141. The esp32 port has also had its generic boards consolidated and renamed to ESP32_GENERIC, ESP32_GENERIC_C3, ESP32_GENERIC_S2 and ESP32_GENERIC_S3, and some now have variants such as SPIRAM.

The mimxrt port has fixes and improvements to PWM and Pin.irq, as well as support for UART hardware flow control and UART.deinit. It has also seen integration of WiFi via the CYW43 driver, and Bluetooth via NimBLE. The mbedTLS bindings have enabled time validation of certificates.

The renesas-ra port has changed board names to match the product name, updated to use FSP v4.4.0, added support for: RA6M5 MCUs, machine PWM, DAC, RNG and SDCard classes, TinyUSB, Bluetooth via NimBLE, networking via lwIP, and mbedTLS.

Bluetooth support has also been added to the rp2 port for the Pico W board. And the RPi boards have been renamed from PICO to RPI_PICO, and PICO_W to RPI_PICO_W. Lightsleep has been fixed on this port so it works while WiFi is powered off, and time.time_ns() now has microsecond resolution.

The samd port sees the addition of SPI and QSPI flash drivers to support filesystems on external flash.

The stm32 port add support for STM32H5xx MCUs, basic support for the OCTOSPI peripheral, and USB support for STM32L1xx MCUs. New functions have been added to the stm module to support the sub-GHz radio on STM32WL55.

In micropython-lib, an extensive LoRa module has been added along with drivers for SX126x and SX127x chipsets, and the STM32WL55. This module supports both synchronous and asynchronous (asyncio) mode. Also, as part of the u-module renaming, urequests has been renamed to requests (but for backwards compatibility "import urequests" still works for now).

The change in code size since the previous release for various ports is (absolute and percentage change in the text section):

   bare-arm:    +192  +0.340%
minimal x86:    +310  +0.169%
   unix x64:   +4784  +0.610%
      stm32:    -524  -0.134%
     cc3200:    +280  +0.154%
    esp8266:   +8016  +1.151%
      esp32: +112133  +7.293%
     mimxrt:   +3624  +1.015%
 renesas-ra:   -2184  -0.348%
        nrf:    +616  +0.332%
        rp2:   +1920  +0.595%
       samd:   -7904  -2.953%

The changes that dominate these numbers are:

• bare-arm, minimal, cc3200, nrf: NLR jump callbacks and support for extensible modules
• unix: update of mbedTLS to v2.28.1, support for polling Python objects in select module
• stm32: removal of the timeq module
• esp8266: addition of the espnow module
• esp32: switching to ESP-IDF 5
• mimxrt: UART features, time validation of SSL certificates
• renesas-ra: move to FSP v4.4.0, remove timeq module
• rp2: machine.PWM enhancements
• samd: drop support for VfsLfs1

Performance is effectively unchanged since the previous release.

Note that this is the last release to use the current versioning scheme for nightly/unstable builds, whereby a build between releases is versioned as v1.20.0-<num>-g<hash> (following the release of v1.20.0). Moving forward, nightly builds will now be called preview builds and be versioned with the next release number. For example, if the last release was v1.21.0 then preview releases will be of the form v1.22.0-preview.<num>.g<hash>. For discussion see #12127.

Thanks to everyone who contributed to this release: Adam Green, Alexander Wilde, algonell, Andrew Leech, Andy Piper, Angus Gratton, Armin Brauns, brave ulysses, Brett Cannon, Brian 'redbeard' Harrington, Carlosgg, Chris Wilson, Christian Clauss, Damien George, Damien Tournoud, Daniël van de Giessen, David Grayson, David Lechner, David Yang, dotnfc, Duncan Lowther, Elecia White, elibdev, Elvis Pfutzenreuter, Felix Dörre, Francis Dela Cruz, Glenn Moloney, glenn20, iabdalkader, Ihor Nehrutsa, Jared Hancock, Jim Lipsey, Jim Mussared, Jon Nordby, Jonas Scharpf, Jos Verlinde, Kwabena W. Agyeman, Luca Burelli, marble, Mark Grosen, mbedNoobNinja, mcskatkat, Mingjie Shen, Mirko Vogt, Nicholas H. Tollervey, Oliver Joos, Ondrej Wisniewski, patrick, Peter Harper, Phil Howard, Philipp Ebensberger, Rene Straub, robert-hh, Sebastian Romero, Seon Rozenblum, stephanelsmith, stijn, Takeo Takahashi, Thomas, Tobias Thyrrestrup, UnexpectedMaker, Victor Rajewski, vsfos, Wang Xuancong, Wanlin Wang, Wilko Nienhaus, Wind-stormger, Yaroslav Halchenko, Yilin Sun, Yuuki NAGAO.

The work done in this release was funded in part through GitHub Sponsors, and in part by George Robotics, Planet Innovation, Espressif, Arduino, LEGO Education and OpenMV.

---

This discussion was created from the release U-module renaming, deflate module, IDF 5, board variants and Pico-W BLE.

[philevans]

I am using the latest 1.21 MicroPython for the FeatherS3 ESP32-S3 board, and I have found it does not have the compress ability for zlib. Is there any way I can get compress working with it?

[jimmo]

@philevans Please see http://docs.micropython.org/en/latest/library/deflate.html

Compression: Enabled via the MICROPY_PY_DEFLATE_COMPRESS build option, on by default on ports with the “full features” level or higher (generally this means you need to build your own firmware to enable this).

You will need to enable this at compile-time.

[n1kdo]

I'm wondering if the "full features" level builds could be made available next to the current builds. In other words, added to the nightlies and released binaries. This is completely because I don't want to build them myself, and I expect other people might also find the full-features builds useful. (FWIW, my use case is that I want to build .png files in my embedded application, and I don't have compress.)

Thanks for considering this.

Jeff

[jimmo]

@n1kdo The reason we choose a given feature level is typically because that's what fits on a given board. i.e. we use "full" on boards that have (relatively) large external spiflash. Many boards wouldn't compile if we enabled "full".

[karfas]

Well, according to https://esp32s3.com/ the FeatherS3 comes with 16MB flash - this I wouldn't consider (relatively) small.
Maybe it's an error not using "full" for the FeatherS3 ...

[jimmo]

This is true (and the ESP32S3 is able to map up to 32MiB into instruction memory, the previous ESP32s were limited, I forget exactly how much).

But one minor point is that our partition table currently only reserves 1984kiB for the app partition. So regardless of the flash chip size, this is the relevant limit for the firmware size without breaking backwards compatibility across updates. (Not sure how much people care about this, but MicroPython has supported in-place firmware updates without losing the filesystem for a long time).

The current generic S3 firmware is 1516kiB so moving to FULL isn't going to be a problem (and I think we should do this). There's still plenty of space left for user C modules and frozen code.

[Wind-stormger]

I'm looking forward to the new mpremote‘s features ,such as #11777 and #9573 .

[jimmo]

Yes, but just to be clear these aren't in the 1.21.0 release. Hopefully 1.22.0.

[ma261065]

Looks like ussl isn't aliased any more:

>>> import ussl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: no module named 'ussl'

[scruss]

Can confirm ussl is gone in v1.22.2 on RPI_PICO_W. Damian's note did say, though:

Now, one should just always import the non-u-version and no longer think about u-naming.

[ma261065]

True - but I had some legacy code break because of this, and that is exactly what the aliases were supposed to avoid. For new code, sure...