ESP32 flash speed, seems slow, but what should it be?

[xyzzy42]

I added support for external QSPI flash chips via ESP-IDF. And it seems slow. I can't find any sort of benchmarks for flash speed on ESP32s. Just nothing at all. So I benchmarked the built-in flash, and it seems slow too.

I wrote a filesystem benchmark that is meant to be somewhat like a data logging program. It appends chunks of data to a file until the file gets to be large, the opens another file, and so on. Then reads them back sequentially.

Here's the result for the onboard flash on an ESP32-S3-MINI-N8R2:

Write: 4.25 MB, 78.510 secs, 55.43243 kB/sec
Read: 4.25 MB,  4.294 sec, 1013.507 kB/sec

Only 55 kB/sec write speed and about 1 MB/sec read. It's running in quad mode and 80 MHz. I thought it would be a bit faster than that.

For comparison, here is the external flash, using basically the same code in ESP-IDF (but not really, since the onboard flash is made by gd and the external is mcix and each flash driver in esp-idf is separate code, even though they should be 99% identical, they aren't).

Write: 10.0 MB, 80.645 sec, 126.9763 kB/sec
Read: 10.0 MB, 3.084 sec, 3320.363 kB/sec

Double the write speed and triple the read speed.

And then for further comparison, I used https://github.com/peterhinch/micropython_eeprom (with a 4kB block size), on the external flash chip.

Write: 10.0 MB, 15.689 sec, 65.267 kB/sec
Read: 10.0 MB, 27.450 sec, 373.0419 kB/sec

Somewhat slower.

To get an idea of what the flash could do without a filesystem, I benchmarked it using the readblocks() and writeblocks() methods directly. I also tried directly reading the entire flash in blocks using the ESP-IDF C API and wrapping that in a single Python function. Here are the results in graphical form.

The internal (green) and external flash (purple) in QIO mode are similar speed. The drop-off at a 64kB block size or above is interesting, I suspect that this might be the result of the memory buffer being placed into PSRAM instead of the on chip memory.
Also shown, blue lines, is the external flash in SPI (1-bit) mode. This is slower, but it's interesting to see that the read speed of the python eeprom library is actually faster than the ESP-IDF code! One wonders why the eeprom lib filesystem benchmark is so much slower.

And finally, raw write speed. Since this overwrites flash, I didn't test the onboard flash as I needed that filesystem.

The jump in speed at 64kB block size is probably because the ESP-IDF driver will switch from 4kB sector erase to 64 kB block erase. The EEPROM library doesn't use that command. It's also slower, which I find odd, since it's mostly waiting for the flash to erase. As we can see from how QSPI and SPI modes are basically the same speed.

And finally, the actual code. One can get the flash filesystem partition on an ESP32 with vfs = esp32.Partition.find(type=1, label="vfs")[0], and then pass it to readbench(). Probably don't want to do that with writebench() unless you make another partition to overwrite, or it will clobber the filesystem. fstest() will default to writing into /, but it can be set with path argument to another mounted filesystem. It should only write and then delete the test files it makes and leave any others untouched.

I'd be interested if others see the same poor speeds as me.

def fstest(n=16, bs=16384, dsize=256*1024, path="/"):
    """  Write some files, read them back, delete them.

    n: Number of files
    dsize: Size of each file
    bs: write/read block size to use, i.e. size of memory buffer
    path: Directory to write files into
    """
    from time import ticks_ms, ticks_diff
    from os import unlink, listdir

    data = bytearray(bs)
    # Number of blocks
    nb = dsize // bs
    s = os.statvfs(path)
    free = s[0] * s[3]
    print(f"Total Size {nb * n * bs / 2**20} MB, Free Space {free/2**20} MB")
    if free < nb * n * bs:
        raise RuntimeError(f"Freespace {free} less than test size {nb * n * bs}")

    def pr(what, start, end):
        delta = ticks_diff(end, start)
        print(f"{what}: {n*nb*bs/2**20} MB, {delta/1000:.03f} sec, {n*nb*bs*1000/delta/2**10} kB/sec")

    start = ticks_ms()
    for i in range(n):
        if i % 10 == 0:
            print(f"{i}...", end="")
        for x in range(len(data)):
            data[x] = i & 0xff
        with open(f'{path}fstest{i:03d}.dat', 'wb') as f:
            for b in range(nb):
                f.write(data)
    end = ticks_ms()
    print()
    pr("write", start, end)

    # Quick check if data appears to have been written correctly
    for i in range(n):
        with open(f'{path}fstest{i:03d}.dat', 'rb') as f:
            if f.read(1)[0] != i & 0xff:
                raise RuntimeError(f"Data at byte 0 of file {i} doesn't match")
            at = f.seek(-1, 2)
            if at != nb*bs - 1:
                raise RuntimeError(f"Seek to end wrong size in file {i}, at {at}, wanted {n*bs-1}")
            if f.read(1)[0] != i & 0xff:
                raise RuntimeError("Data at end of file {i} doesn't match")

    start = ticks_ms()
    for i in range(n):
        with open(f'{path}fstest{i:03d}.dat', 'b') as f:
            for b in range(nb):
                f.readinto(data)
    end = ticks_ms()
    pr("read", start, end)

    start = ticks_ms()
    for name in listdir(path):
        if name.startswith("fstest") and name.endswith(".dat"):
          unlink(f"{path}{name}")
    end = ticks_ms()
    pr("unlink", start, end)

def readbench(flash):
    """ Flash read speeds, different block sizes

    flash: Device to test, such as esp32.Partition() or micropython_eeprom.FLASH()

    Reads entire flash in blocks of ascending power of two sizes and reports
    speed.  Starts at native flash block size and stops at 512 kB.
    """
    from time import ticks_ms, ticks_diff
    from gc import collect
    try:
        # ESP32 partitions return length this way
        fs = flash.info()[3]
    except AttributeError:
        # While Peter Hinch's EEPROM library does this
        fs = len(flash)

    bs = flash.ioctl(5, None)
    print(f"Flash Size: {fs//1024} kB   Block size: {bs} bytes")

    for p in range(7, 20):
        if bs > 1<<p:
            continue
        bc = (1<<p) // bs
        buf = bytearray(bc * bs)
        n = fs // len(buf)
        start = ticks_ms()
        for i in range(n):
            flash.readblocks(bc * i, buf)
        stop = ticks_ms()
        mbs = n * len(buf) * 1000 / ticks_diff(stop, start) / 2**20
        print(f"{len(buf)} {mbs}")
        del buf
        collect()

def writebench(flash):
    """ Flash write speeds, different block sizes

    WARNING: This is destructive to the contents of the flash device!

    flash: Device to test, such as esp32.Partition() or micropython_eeprom.FLASH()

    Reads entire flash in blocks of ascending power of two sizes and reports
    speed.  Starts at native flash block size and stops at 512 kB.
    """
    from time import ticks_ms, ticks_diff
    from gc import collect
    try:
        # ESP32 partitions return length this way
        fs = flash.info()[3]
    except AttributeError:
        # While Peter Hinch's flash library does this
        fs = len(flash)
    bs = flash.ioctl(5, None)
    print(f"Flash Size: {fs//1024} kB   Block size: {bs} bytes")

    for p in range(8, 20):
        if bs > 1<<p:
            continue
        bc = (1<<p) // bs
        buf = bytearray(bc * bs)
        n = fs // len(buf)
        start = ticks_ms()
        for i in range(n):
            flash.writeblocks(bc * i, buf)
        stop = ticks_ms()
        mbs = n * len(buf) * 1000 / ticks_diff(stop, start) / 2**20
        print(f"{len(buf)}: {mbs}")
        del buf
        collect()

[bixb922]

Interesting! When I run readblocks on my ESP32-N8R8, I get similar values, from 8.7 to 11, peaking at 32768 block size.

This is somewhat related to your post and may be of interest: I found that with the ESP32 and a large flash, the the Littlefs2 filesystem has a parameter that can speed up things a bit, see https://docs.micropython.org/en/latest/library/os.html os.VfsLFs2()

I do the following at the beginning of main.py:

os.umount("/")
os.mount(os.VfsLfs2(bdev,readsize=1024,progsize=32,lookahead=32),"/")

For me, this speeds up some file operations by a factor of 2 or 3 on the ESP32-N8R8. YMMV.

The readsize is the number of bytes that LittleFS2 reads in one operation. Micropython allocates a cache four times that size, i.e. if readsize=1024, Micropython assigns a cache of 4096 bytes. LittleFS2 allocates two caches for the system and one for each open file, so in this case that would be 8192 for the system and 4096 for each open file. These caches seem to speed up file operations, probably if the flash is noticeably slower than the RAM.

Also, the optimal value for lookahead is the number of flash blocks divided by 8, with a modest impact on write performance. The lookahead buffer is a bitmap of free blocks. After writing 32*8 blocks, LittleFS2 has to fill up the lookahead buffer again, since there is no free block bitmap on flash. So a large lookahead buffer avoids reconstructing this bitmap.

I didn't see any impact of the readsize parameter for a RP2040. The ``readblocks()```benchmark from the OP yields 8.0 to 8.6 with increasing read size, largest with 65536 reads.

This may also be of interest: writing small files (less than 1024 bytes on the ESP32) writes data to the directory structure instead of using dedicated data blocks and is quite fast.

[trentp-igor]

Default values of all those parameters appear to be 32. So setting progsize and lookahead to 32 should make no difference. I wonder if the rootfs is somehow mounted with non-default options?

Setting readsize to 2048 or above causes writes to the filesystem to fail with a mysterious OSError: 84. I can't find that code anywhere. Maybe the lfs source code is where I should look.

I tested values from 32 up to 1024. For my test, which is a sequential, not random, access pattern, it was able to almost double read speed. Write speed was basically unaffected. Contrary to what one might think, larges reads were more sped up by increase in readsize than small reads. I.e., with file.read(32) calls, the read speed ranged from ~2.56 to 3.29 MB/sec over readsize 32 - 1024. While 4096 byte read calls ranged from 4.64 - 7.41 MB/sec.

[xyzzy42]

One thing that makes writes slow on ESP32 is that it only writes in 64 byte chunks!

You can set CONFIG_SPI_FLASH_WRITE_CHUNK_SIZE in ESP-IDF to 8192 bytes, the default, but it doesn't matter. Everything is still copied into a fixed 64 byte bounce buffer and written 64 bytes per SPI transaction. Page size is configurable, usually 256 bytes, but all the code to keep track of it is pointless because no writes are larger than 64 bytes. The other chunks and bounce buffers (there are more!) before this final 64 byte limit are just a waste.

If sectors are erased immediately before being written, then sector erase time dominates and this doesn't matter so much that it goes through multiple buffers, because each layer doesn't know that the layer under also buffers, and so they get away with this inefficiency. A faster system would pre-erase blocks when idle to have a pool ready so when writes do occur they don't suffer from the erase time.

[xyzzy42]

Something that makes reads slow is that they are also limited to 64 byte chunks! There's code in ESP-IDF that forces all flash reads to be page aligned, not cross a page boundary, and be 64 bytes or less. And all reads are also first read into a temporary buffer, which is filled with 0xff before each read, and then are copied into another temporary buffer. None of this is necessary. It also reconfigures the flash controller before each read to the same settings it was already at.

Unfortunately, this 64 byte limit is inside the secret binary only part of the ESP-IDF code, so it can't be fixed.

But still, a bunch of bad code can be fixed, and after getting rid of some of this stuff, I was able to get the mpy filesystem read speed up to over 8 MB/sec. Quite a bit better than 1 MB/sec the onboard flash gets by default in Micropython and the 3.3 MB/sec from the initial external flash test. Still, it should be possible to do four times better than that.

Sustained write speed is limited by the 4kB flash sector erase time (about 20 ms on my flash chip) to ~200 kB/sec. While the observed speed is only 120 kB/sec and there is room for improvement, especially by taking advantage of the 64 kB block erase commands, which I think LittleFS will never do, it's not going up by a factor of eight.

This is the raw block device read performance. One can see how, compared to the graph in the OP, the external speed is improved by my changes to the ESP-IDF code, while the internal is unchanged, since each flash chip has an separate driver (that is 99% identical to all the other drivers).

[projectgus]

These are really very interesting results! Thanks for taking the time to dig so deeply into this.

I can supply a bit of context on some of this:

Something that makes reads slow is that they are also limited to 64 byte chunks! There's code in ESP-IDF that forces all flash reads to be page aligned, not cross a page boundary, and be 64 bytes or less. And all reads are also first read into a temporary buffer, which is filled with 0xff before each read, and then are copied into another temporary buffer. None of this is necessary. It also reconfigures the flash controller before each read to the same settings it was already at.

I guarantee at least some of this stuff is prioritising correctness over performance. There are some absolutely shocking flash chip controllers out there, there's no standard and every vendor does things a little different (before you even get to the bugs). However it does seem bad to punish all flash performance rather than to add quirks for bad actors.

Unfortunately, this 64 byte limit is inside the secret binary only part of the ESP-IDF code, so it can't be fixed.

There's a couple of things going on here, but there's a bit more to it than this. AFAIK all of these routines are using the hardware SPI state machine in MMIO mode. This has 16 32-bit data registers, which is where the limit comes from:

(Extract from the GPSPI docs in ESP32-S3 TRM.)

I don't particularly understand why the data also has to go via the temp buffer on the stack in spi_flash_chip_generic_read(). That certainly seems like it could be optimised out whenever the buffer is already word aligned, and maybe always.

There is a hardware DMA mode for GP-SPI in ESP32-S3 that I assume can read arbitrary byte lengths, but it seems no one has implemented this in a driver. There's no such DMA mode on original ESP32, and I'm not sure about the other SoCs.

Certainly there is some support work in the ESP-IDF driver for adding DMA later (all the "!direct_read" execution path that accounts for DMA length and alignment requirements), this was done when I still worked at Espressif. I don't remember all the details but I'm 99% sure it's currently only used for encrypted flash read and write.

The high-level workaround code does point to how DMA isn't a silver bullet for performance, as it depends on buffer alignment. Although I think for MicroPython on ESP32-S3 it would often be a fast path as DMA to/from PSRAM supports 16 byte aligned buffers and the MicroPython heap block size is 16 bytes. 🎉

The second thing which ESP-IDF does on these newer chips is link the mask ROM implementations of many flash driver functions to save code size (especially IRAM size). This is may be the "secret" code you run up against, although it looks like this is disabled by default in ESP-IDF and in MicroPython:
https://docs.espressif.com/projects/esp-idf/en/v5.0.6/esp32s3/api-reference/kconfig.html#config-spi-flash-rom-impl

(The spi_flash driver is also pretty complex with the amount of layers and indirection for different host drivers and chips!)

But still, a bunch of bad code can be fixed, and after getting rid of some of this stuff, I was able to get the mpy filesystem read speed up to over 8 MB/sec.

Do you have the patch of what you changed? I think it might be feasible to send some of it upstream, even if it's only added for certain flash chip drivers.

The other thing that could be worth trying on the MP side might be using the ESP-IDF spi_flash mmap feature (i.e. the flash cache) for internal flash reads in some cases. Sequential read isn't going to be any faster (the biggest cache line fill is 32 bytes), but if the mapping is kept around then frequently accessed regions may end up cached, and the newer chips can do hardware lookahead. This might depend on how good the higher layer filesystem is at caching though, IIRC LittleFs supports its own cache already.

[xyzzy42]

I guarantee at least some of this stuff is prioritising correctness over performance

Not enough unfortunately. The code assumes the write enable latch bit and the busy status bit will be cleared simultaneously when a write command completes. The Macronix flash chip we used clears busy first, then WEL later, and it's possible for a status poll to hit the window when busy is clear but WEL is not. Espressif's mxic driver can't handle this quirk.

I don't particularly understand why the data also has to go via the temp buffer on the stack in spi_flash_chip_generic_read(). That certainly seems like it could be optimised out whenever the buffer is already word aligned, and maybe always.

Yes, I got rid of that. Not just the buffer, but all the extra alignment code. Not only is it not necessary for reads to be aligned, there is already a layer above this function that aligns them.

There's a couple of things going on here, but there's a bit more to it than this. AFAIK all of these routines are using the hardware SPI state machine in MMIO mode. This has 16 32-bit data registers, which is where the limit comes from:

So that's where it comes from. I've generally found that Espressif hardware to not be documented well enough that writing code to directly program it is feasible. Not that I actually want to, who has time for that in the modern era? And esp-idf is actually pretty well designed for a vendor HAL system and it's useful to use it.

This is may be the "secret" code you run up against, although it looks like this is disabled by default in ESP-IDF and in MicroPython:

It's different code. The IDF option to use the high level ROM flash functions is indeed off. But eventually it gets to this low level function:

PROVIDE( spi_flash_hal_read = 0x40000d68 );

I believe that's where the GPSPI state machine's 16 registers that you mentioned would appear.

[xyzzy42]

And here is filesystem performance. I used a LittleFS readsize of 256 as this results in the peak performance, regardless of the size of the read() calls used on the files.

While the raw block read speed of the ESP-IDF flash code (in SPI mode) vs the Python EEPROM library is nearly the same, as seen in the previous post, the Python lib has much slower filesystem read speed. The write speed is also much slower, despite the fact that it's mostly waiting for flash sectors to erase.

We also see no jump in write speed at 64kB, like there is with direct block access, because LittleFS will never make use of the faster 64kB block erase. This isn't entirely a LittleFS flaw, it's also a flaw in the Micropython extended AbstractBlockDevice API, which only allows for a single 4kB sector be erased at a time.