micropython port micro:bit v2 : Please add item assignment / referencing to global static memory blocks via bytearray() and memoryview()

[burksfamly]

For the micropython port to the micro:bit v2, please consider adding item assignment / referencing to bytearray() memoryview() for memory blocks that have been instantiated with a global focus for the purpose of using the memory block as a static byte buffer. In other words, allow the ability to peek and poke one or more bytes to a block of memory that, once declared, is not moved by other operations.

_global_ba = bytearray(64)
_mv_global_ba = memoryview(_global_ba)

_global_ba[:] = b'message payload' ... this is not supported
_mv_global_ba[:] = b'message payload' ... this is not supported

_global_ba.extend(b'message payload') ... this moves the memory block and the pointer _mv_global_ba is now invalid

[karfas]

I don't understand each and every python construct, but here are my thoughts about this (someone else please correct me when I got something wrong).

The operations you would like to use are also not supported in standard python 3:

Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> a = bytearray(20)
>>> mv = memoryview(a)
>>> mv[:] = b'foo'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview assignment: lvalue and rvalue have different structures
>>> mv[0] = 0x40
>>>

The last of your use cases (buffer.extend() ):

• changes the semantics of array.extend() - in python, array.extend() is required to return a new object.
• is most likely not possible in any python implementation, as the memory behind the original array will already be in use by some other object.
  If you require a large buffer, you should allocate (declare) it in the first place.
• The memoryview doesn't become "invalid". It still has a pointer to the array it had been created from.

[peterhinch]

This is not valid Python: you can't do this in CPython. This is how to do it:

_global_ba = bytearray(64)
_mv_global_ba = memoryview(_global_ba)
msg = b"My message"
_mv_global_ba[0: len(msg)] = msg

[burksfamly]

@karfas and @peterhinch ... thank you so much for your comments

As I stated in a related post regarding this issue, python is not my forte and I am in a situation where my dev environment is using the micropython port to the micro:bit v2.

Item assignment is not supported in this environment by either bytearray() or memoryview(), which means that
_mv_global_ba[0: len(msg)] = msg
is not supported and will not work, as it generates an error

Having said that and to clarify, what I am seeking is a method to declare [ similar to malloc() ] a static block of memory that I have global access to which I can poke and peek byte(s) of data without micropython moving the location of that buffer when manipulated.

I only used the bytarray.extend() method as there are only three bytearray() methods available in the micropython port to micro:bit v2 and the extend() method is the only one that will permit bytes of data to be copied to the buffer as item assignment is not permitted.

If this is not possible in python or the micropython port to the micro:bit v2, that is very good information for me to know so that I can reassess and go in a different direction.

QUESTION: So...is it possible in the micropython port to the micro:bit v2 to declare a global block of static memory, obtain a pointer to that block of memory, and use that pointer to write and read byte(s) of data (poke and peek) without micropython moving the location of the block of memory and thereby voiding the global pointer to that memory?

Would it be possible to answer this question for me? Yes or No

If yes, would it be possible to provide a simple example of this for theh micropython port to the micro:bit v2.

If no, that is very good for me to know, and I will reconsider alternatives.

Thank you so much for your input.

[andrewleech]

I'm not sure why it would matter if the block of memory gets moved? Is it for speed, or you don't have enough spare ram for copies to succeed? Explaining the problems you're seeing help others come up with the best suggestions!

Regardless, I think you could get what you need with uctypes https://docs.micropython.org/en/latest/library/uctypes.html
You could declare a struct with an array of the appropriate byte type and length.
Generally you would still create a bytearray first, then apply the struct to that buffer. Accessing it via the struct will definitely not cause any moving / resizing of the underlying buffer.

[burksfamly]

@andrewleech Thank you for your reply.
Yes., and as far as I know uctypes is not available in the micropython port to the micro:bit v2.

[burksfamly]

micro:bit help desk reached out to me to let me know what was not supported and what was supported as follows:

------------------------------------------
what is supported and not supported
------------------------------------------

a = bytearray(b'0123456789')

a[0:3] = b'jkl'           ...not supported

m = memoryview(a)
m[0:3] = b'jkl'          ...not supported

a[0] = b'j'                ...supported
a[1] = b'k'
a[2] = b'l'

------------------------------------------
The following is what I was looking for...the buffers are instantiated once and not moved around by micropython operations.
------------------------------------------

_rx_buffer = bytearray(64)
_mv_rx_buffer = memoryview(_rx_buffer)
_tx_buffer = bytearray(64)
_mv_tx_buffer = memoryview(_tx_buffer)

def poke(toBuffer, toOffset, fromBuffer, fromOffset, numBytes):
    for i in range(numBytes):
        toBuffer[toOffset+i] = fromBuffer[fromOffset+i]

def peek(mvBuffer, offset, numBytes):
    return(bytes(mvBuffer[offset:offset+numBytes]))

getRadioMsg = b'017,1234,PING,4059,3945,N,payload 001'       # sim func getting rx bytes
poke(_mv_rx_buffer,  0, getRadioMsg,    0, len(getRadioMsg)) # rcv rx bytes directly into buff 
poke(_mv_tx_buffer,  0, _mv_rx_buffer,  0, len(getRadioMsg)) # copy rx bytes to tx buff
poke(_mv_tx_buffer, 19, _mv_rx_buffer, 14, 4)                # tx dst node = rx snd node 
poke(_mv_tx_buffer, 14, _mv_rx_buffer, 19, 4)                # tx snd node = rx dst node (me)
print(peek(_mv_rx_buffer, 0, len(getRadioMsg)))              # use rx buff
print(peek(_mv_tx_buffer, 0, len(getRadioMsg)))              # use tx buff

getRadioMsg = b'017,4321,PING,9876,0123,N,payload 001'       # sim func getting rx bytes
poke(_mv_rx_buffer,  0, getRadioMsg,    0, len(getRadioMsg)) # rcv rx bytes directly into buff 
poke(_mv_tx_buffer,  0, _mv_rx_buffer,  0, len(getRadioMsg)) # copy rx bytes to tx buff
poke(_mv_tx_buffer, 19, _mv_rx_buffer, 14, 4)                # tx dst node = rx snd node 
poke(_mv_tx_buffer, 14, _mv_rx_buffer, 19, 4)                # tx snd node = rx dst node (me)
print(peek(_mv_rx_buffer, 0, len(getRadioMsg)))              # use rx buff
print(peek(_mv_tx_buffer, 0, len(getRadioMsg)))              # use tx buff

[jimmo]

FYI we are considering adding some sort of memcpy / memmove / memset like primitives to MicroPython. (This came up in the context of implementing USB device class drivers).

CPython does have memmove and memset in ctypes (see https://docs.python.org/3/library/ctypes.html#ctypes.memmove) but unfortunately:
a) we don't implement ctypes (and it would be a bit strange to have both uctypes and ctypes
b) these methods don't support an offset parameter, which limits their use a bit

[burksfamly]

@jimmo Thank you...that would be useful for the micro:bit, especially for fast streams over uart, spi, i2c, etc.
At this time, speed is not critical to my application, but sketch, heap, stack taxation and fragmentation are.
BTW in my previous post re micro:bit help desk guidance, they reached out again to inform me that I may
be better served using directed peeking via buffer[i] rather than using memoryview(), as memoryview() may
be creating another object and copies of data...I will be exploring rewriting my peek() func...

[peterhinch]

QUESTION: So...is it possible in the micropython port to the micro:bit v2 to declare a global block of static memory, obtain a pointer to that block of memory, and use that pointer to write and read byte(s) of data (poke and peek) without micropython moving the location of the block of memory and thereby voiding the global pointer to that memory?

Yes, and the code I provided does exactly that. If you write

buf = bytearray(100)
mvb = memoryview(buf)

You can peek and poke via mvb freely. Don't touch buf and it will remain at its original location. You "peek and poke" using mvb:

a = mvb[42]  # Read from offset 42 (peek)
mvb[20] = 5  # Poke offset 20
msg = b"Here is a message"
mvb[50 : 50 + len(msg)] = msg  # Put the message in locations starting at 50

Python uses this mechanism in place of pointers.

[burksfamly]

@rkompass ... I do not have a make-environment to build the micropython.hex file.

At the web interface https://python.microbit.org/v/3 , where do you let the interface know that you are pointing to an overriding
micropython hex file?

Mu has a feature option under the gear icon settings to point to a custom micropython.hex file.

thank you

[rkompass]

I put the above file unpacked to MICROBIT.HEX onto the MICROBIT drive which appears as soon as the Microbit is plugged in. Later I unplugged and plugged in again.
Then in the Web interface I did a connect and did klicked on show serial and entered a Ctrl-D:

MicroPython v1.20.0-dirty on 2023-12-18; micro:bit v2.1.1 with nRF52833
Type "help()" for more information.
>>> 

[rkompass]

You are right, there's a problem: At the Web-Interface with a klick to "Send to micro:bit" I'm again at the 1.18 MP version.
Because Thonny makes the problems mentioned below I tried out the Mu editor and can now recommend it for the Microbit. There is an option in Mu where you may select the HEX-File that you want to use and you can then open the new firmware file (the location of it, where it is stored on your disk).

[rkompass]

@burksfamly I found also that you still can use Thonny with the above Firmware, you just have to select another (not the Microbit) interpreter setting like e.g. RP2040 Pico or Circuitpython. Then it works and the problems described below are gone. You can upload your scripts and don't automatically upload a big hex file (that includes the whole Micropython interpreter) every time.

Now it seems I'm again in my familiar setup. The Microbit does not allow directories on its file system and has no f-strings though.

[burksfamly]

@rkompass ... thank you ... I will try to use your 1.20 dirty HEX file with Mu and verify ...

I wonder if someone from the micro:bit foundation could elabprate if and when they will update from 1.18 to 1.20 with the item assignment available with bytearray and memoryview so that the web interface will automatically pick that one up ... if so, when ? ... if not, curious as to why ...

[jimmo]

which reports at the line strb(r0, [pcnt, 0]) a SyntaxError: expecting an assembler instruction.

@rkompass See https://docs.micropython.org/en/latest/reference/asm_thumb2_str.html

pcnt is not a register name.

2.: generators return None instead of correct values, as demonstrated by:

This works for me on the standard micro:bit v2 firmware:

MicroPython v1.18 on 2023-10-30; micro:bit v2.1.2 with nRF52833
Type "help()" for more information.
>>> 
paste mode; Ctrl-C to cancel, Ctrl-D to finish
=== def generator_function():
===     yield 42
===     yield 22
=== 
=== generator = generator_function()
=== 
=== # generator yields only 42
=== for value in generator:
===     print(value)
=== 
42
22

[rkompass]

@jimmo Thank you again and Oops, I'm sorry:

I took the assembler example from my PC without further inspecting. You are right of course.
Both problems are still there now, also with correct assembler code, but only when I use Thonny.

On the web interface here, both examples work as expected (with old and with the new compilation).

This is the first time I see such a gross dysfunctionality with Thonny without any indication of error (thrashing bytes, not connecting or such). Now I HAVE TO learn to use the other interfaces. I will report to aivarannamaa of course.