microphone support

[thiswillbeyourgithub]

As per @DonvdH request there

Thanks a lot!

[thiswillbeyourgithub]

@thiswillbeyourgithub: If you open a separate issue regarding the microphone I will share my initial findings, but I don't think it will be easy because Micropython appears to be lacking PDM support and polling the pin using Python is probably not fast enough to obtain proper audio data. Circuitpython does support PDM, so perhaps this could be ported if someone would be willing to do this.

Answering to @DonvdH in #14 (comment)

I am involved in the developpement of Wasp-OS (micropython OS on pinetime from Pine64, based on the nRF52832)

In this code they manage to get a reading of the heart sensor at 24Hz. I do not know if they can get it to be faster. They also succed in doing a bunch of post processing on the watch by applying filters and stuff.

Is that relevant? How do you update from this info regarding the feasability of using the
microphone on micropython?

Additionnaly, when you say you fear that python might not be fast enough, do you take into account the different code emitters that can dramatically improve the speed?

Thanks a lot!

[DonvdH]

Great to hear that you are so actively involved with smartwatch development. I have also looked at the Pinetime and Wasp-OS by the way. While it also looks very nice, I have chosen the T-Watch in this case because of the wifi connectivity.

Regarding the microphone:
The Lilygo T-Watch 2020 v3 is using a PDM MEMS Microphone in the form of the Knowles SPM1423HM4H.

It is connected to the following pins:
Data: IO02
SCLK: IO00

So from what I understand is that the ESP32 needs to generate a clock signal on IO00 and per clock cycle one bit can then be read on IO02. So you would need about 24000 clock cyles to obtain some usable audio.

The code to obtain the PDM data would look something like this:

import machine
import array

def capture_pdm_data(buffer_size):
    pdm_data = array.array('H', [0] * buffer_size)  # Unsigned 16-bit PDM buffer
    
    # Configure PDM clock pin
    pdm_clk_pin = machine.Pin(0, machine.Pin.OUT)
    pdm_clk_pin.value(0)  # Set clock pin to low
    
    # Read PDM data
    for i in range(buffer_size):
        pdm_data[i] = machine.Pin(2).value()  # Read PDM data pin
        
        # Generate PDM clock signal (e.g., using a GPIO toggle)
        pdm_clk_pin.value(1)
        pdm_clk_pin.value(0)
    
    return pdm_data

If I do a little benchmarking, it appears that reading 24000 bits of PDM data takes +/- 0.669 seconds:

import time

starttime = time.time_ns()
pdm_data = capture_pdm_data(24000)
endtime = time.time_ns()
print("Total time taken in milliseconds: "+str((endtime-starttime)/1000000))

So you appear to be right that the required speed is achievable using Python.

I believe the next steps would be to get the timing right and to convert the data into a usable format (WAV) afterwards.

[thiswillbeyourgithub]

Thanks a lot! Really!

WIth a bit of help from GPT-4 I apparently achieved a bit faster still :

from machine import Pin
import micropython
import array

micropython.alloc_emergency_exception_buf(100)

pdm_clk_pin = Pin(0, Pin.OUT)
pdm_data_pin = Pin(2, Pin.IN)

@micropython.native
def capture_pdm_data(buffer_size):
    pdm_data = array.array('H', [0] * buffer_size)
    for i in range(buffer_size):
        pdm_data[i] = pdm_data_pin.value()
        pdm_clk_pin.on()
        pdm_clk_pin.off()
    return pdm_data

Test with

import time

starttime = time.time_ns()
pdm_data = capture_pdm_data(24000)
endtime = time.time_ns()
print("Total time taken in milliseconds: "+str((endtime-starttime)/1000000))

Average result was 0.437s. I also tried a bit with the viper emitter (apparently even faster) but I couldn't make it work.

Regarding the timing, isn't it just a matter of adding some timers in the loop? The heart.py code from wasp-os seems to do that.

Also, what is the unit of the value stored at each clock cycle in pdm_data? Is it directly an amplitude reading of the air pressure? As you can see I'm quite lost (I'm a med student and this is a side project!)

[DonvdH]

Nice job, that's an impressive performance which is being achieved there by the micropython.native decorator.

There are numerous strategies to add delay.
In this specific case I would start out by using time.sleep_us so the benchmark reports +/- 1 second:
https://docs.micropython.org/en/v1.7/wipy/library/time.html#time.sleep_us
Since the firmware and processor are always the same, I expect it to work well enough for the time being. The example you mention may work better, but if you find it too hard then just skip it.

Wikipedia explains how a PDM Mems Microphone works quite nicely:
https://en.m.wikipedia.org/wiki/Pulse-density_modulation

Once the delay has been added, a function is needed to convert the PDM data to PCM/WAV. But I couldn't find a ready to use library for this unfortunately, so this may prove to be the hardest part.

[thiswillbeyourgithub]

I initially went with just adding a timer with time.sleep_us(25) but was not sure what would happen with difference with compiled code so I went with a proper timer instead:

With the following code I managed to get good enough accuracy:

from machine import Pin, Timer
import micropython
import array
import time

pacer = micropython.const(4425)

micropython.alloc_emergency_exception_buf(100)

pdm_clk_pin = Pin(0, Pin.OUT)
pdm_data_pin = Pin(2, Pin.IN)

@micropython.native
def capture_pdm_data(buffer_size):
    pdm_data = array.array('H', [0] * buffer_size)
    for i in range(buffer_size):
        prev = time.ticks_cpu()
        pdm_data[i] = pdm_data_pin.value()
        pdm_clk_pin.on()
        pdm_clk_pin.off()
        while time.ticks_diff(time.ticks_cpu(), prev) <= pacer:
            pass
        #print(time.ticks_diff(time.ticks_cpu(), prev))

        #time.sleep_us(25)
    return pdm_data

starttime = time.time_ns()
pdm_data = capture_pdm_data(24000)
endtime = time.time_ns()
print("Total time taken in milliseconds: "+str((endtime-starttime)/1000000))

The answer is around 1000ms +- 0.03ms

Now regarding the PDM to PCM/WAV, would you mind explaining to me the job of the ADC? I thought I understood that it could be used to turn a PDM signal into PCM.

Also here's some links I stumbled upon last time I tried to understand how to use audio on this watch:

• Here's the ADC of the TWatch

• Also I found this C code

• Here's another code showing how to play a wav file

• Two project related to audio and on ESP32

• A last, a github thread that looked somewhat related

This is way outside of my scope so any hand holding is greatly appreciated!

[DonvdH]

I appreciate your persistence. :) Keep in mind that I'm not an audio specialist and my time is unfortunately rather limited. Also I don't currently have plans to use the microphone myself.

The ADC (Analog to Digital Converter) is unrelated to what you're trying to achieve. The ADC makes it possible to read an analog signal (voltage) on a pin, but a PDM MEMS Microphone emits a digital signal (instead of a analog signal).

If I had to make an educated guess, then you would need something like the code below. Several filters appear to be needed to actually convert PDM to PCM, but unfortunately I couldn't find a Python library for that.
See the following PDF for more details: https://www.nxp.com/docs/en/application-note/AN12590.pdf

from machine import Pin
import micropython
import array
import time
import ustruct

micropython.alloc_emergency_exception_buf(100)

pacer = micropython.const(4425)

pdm_clk_pin = Pin(0, Pin.OUT)
pdm_data_pin = Pin(2, Pin.IN)

@micropython.native
def capture_pdm_data(buffer_size):
    pdm_data = array.array('H', [0] * buffer_size)
    for i in range(buffer_size):
        prev = time.ticks_cpu()
        pdm_clk_pin.on()
        pdm_clk_pin.off()
        pdm_data[i] = pdm_data_pin.value()
        while time.ticks_diff(time.ticks_cpu(), prev) <= pacer:
            pass
        #print(time.ticks_diff(time.ticks_cpu(), prev))

        #time.sleep_us(25)
    return pdm_data

starttime = time.time_ns()
seconds_of_audio = 2 # Record two seconds of audio data (for testing)
pdm_buffer = capture_pdm_data(24000*seconds_of_audio)
endtime = time.time_ns()
print("Total time taken in milliseconds: "+str((endtime-starttime)/1000000))


num_samples = len(pdm_buffer)
pcm_buffer = array.array('H', [0] * num_samples)  # 'H' for unsigned 16-bit PCM samples

# Convert PDM to PCM from pdm_buffer into pcm_buffer here
# Several filters appear to be needed like a CIC filter and low pass filters
# See: https://www.nxp.com/docs/en/application-note/AN12590.pdf

# Define WAV file parameters
sample_width = 2  # 2 bytes for 16-bit PCM
num_channels = 1  # Mono

# Open the WAV file for writing
with open('output.wav', 'wb') as wav_file:
    # Write the WAV file header
    wav_file.write(b'RIFF')
    wav_file.write(b'\x00\x00\x00\x00')  # Placeholder for total file size
    wav_file.write(b'WAVE')
    wav_file.write(b'fmt ')
    wav_file.write(ustruct.pack('<IHHIIHH', 16, 1, num_channels, 16000, 32000, num_channels * sample_width, sample_width * 8))
    # Write the PCM data
    wav_file.write(b'data')
    wav_file.write(ustruct.pack('<I', len(pcm_buffer) * sample_width))
    wav_file.write(bytes(pcm_buffer))
    # Calculate the total file size and update the WAV file header
    total_size = len(pcm_buffer) * sample_width + 36
    wav_file.seek(4)
    wav_file.write(ustruct.pack('<I', total_size - 8))

[DonvdH]

I believe that, given the lack of available Micropython libraries for PDM conversion, the best solution would be to transfer the PDM data to a HTTP API and then convert it to PCM/WAV using this tool:
https://github.com/siorpaes/pdm_playground/tree/master/pdm2pcm

For example FastAPI could be used on the server side.
An advantage of this approach would be processing speed and you could then quite easily run the data through a speech to text API afterwards for example.

[devnoname120]

ESP32 actually contains an Inter-IC Sound (I2S) peripheral that can convert from PDM to PCM by using PDM Mode (RX).
MicroPython has basic I2S support but unfortunately it doesn't support PDM Mode (RX).

There is a repository from 2020 that adds the I2S PDM mode to MicroPython: https://github.com/lemariva/micropython-i2s-driver

I'm not sure how much effort it would require to port it over to a recent MicroPython. @jeffmer Is https://github.com/jeffmer/micropython/tree/GPIO_WAKEUP the right branch for compiling your custom firmware.bin?

See also: https://github.com/lunokjod/watch/blob/b71340b486b68d7682a9021e454779557d7a291c/src/app/Mic.cpp#L60-L82

[thiswillbeyourgithub]

I can't add anything regarding those hardware stuff unfortunately.

I believe that, given the lack of available Micropython libraries for PDM conversion, the best solution would be to transfer the PDM data to a HTTP API and then convert it to PCM/WAV using this tool:

I would prefer to keep this as a last resort option.

Here's my latest trial:

from machine import Pin
import micropython
import array
import time
import ustruct
import math
from tempos import pm
from drivers.axp202 import LD04

micropython.alloc_emergency_exception_buf(100)

pacer = micropython.const(4425)

pdm_clk_pin = Pin(0, Pin.OUT)
pdm_data_pin = Pin(2, Pin.IN)

@micropython.native
def cic_filter(pdm_data, num_stages, decimation_factor, differential_delay):
    # Cascaded Integrator-comb filter
    integrator = [0] * num_stages
    comb = [0] * num_stages
    output = array.array('H', [0] * len(pdm_data))
    for i in range(len(pdm_data)):
        integrator[0] += pdm_data[i]
        for j in range(1, num_stages):
            integrator[j] += integrator[j - 1]
        if i % decimation_factor == 0:
            for j in range(num_stages):
                comb[j] = integrator[j] - comb[j]
            output[i // decimation_factor] = comb[-1]
    return output

@micropython.native
def pdm_to_pcm(pdm_data):
    # PDM to PCM conversion
    pcm_data = array.array('H', [0] * (len(pdm_data) // 16))
    for i in range(0, len(pdm_data), 16):
        pcm_data[i // 16] = sum(pdm_data[i:i+16]) // 16
    return pcm_data

@micropython.native
def low_pass_filter(pcm_data, cutoff_frequency, sample_rate):
    # Low pass decimation filter
    output = array.array('H', [0] * len(pcm_data))
    alpha = 2 * math.pi * cutoff_frequency / sample_rate
    for i in range(1, len(pcm_data)):
        output[i] = int(alpha * pcm_data[i] + (1 - alpha) * output[i - 1])
    return output

@micropython.native
def anti_aliasing_filter(pcm_data, cutoff_frequency, sample_rate):
    # High frequency noise anti aliasing low pass filter
    output = array.array('H', [0] * len(pcm_data))
    alpha = 2 * math.pi * cutoff_frequency / sample_rate
    for i in range(1, len(pcm_data)):
        output[i] = int(alpha * pcm_data[i] + (1 - alpha) * output[i - 1])
    return output


@micropython.native
def capture_pdm_data(buffer_size):
    pdm_data = array.array('H', [0] * buffer_size)
    pdm_clk_pin.off()
    start = time.time()

    for i in range(buffer_size):
        prev = time.ticks_cpu()
        pdm_clk_pin.on()
        pdm_clk_pin.off()
        pdm_data[i] = pdm_data_pin.value()
        while time.ticks_diff(time.ticks_cpu(), prev) <= pacer:
            pass
    end = time.time() - start
    print(end)
    return pdm_data

def record_audio(length_s=1):
    pm.setPower(LD04, 1)
    print("recording")
    pdm_buffer = capture_pdm_data(24000*length_s)
    print("done recording")
    pm.setPower(LD04, 0)
    num_samples = len(pdm_buffer)

    # Convert PDM to PCM from pdm_buffer into pcm_buffer here
    pdm_buffer = cic_filter(pdm_buffer, 4, 16, 1)
    pcm_buffer = pdm_to_pcm(pdm_buffer)
    pcm_buffer = low_pass_filter(pcm_buffer, 4000, 16000)
    pcm_buffer = anti_aliasing_filter(pcm_buffer, 4000, 16000)


    # Define WAV file parameters
    sample_width = 2  # 2 bytes for 16-bit PCM
    num_channels = 1  # Mono

    # Open the WAV file for writing
    with open('output.wav', 'wb') as wav_file:
        # Write the WAV file header
        wav_file.write(b'RIFF')
        wav_file.write(b'\x00\x00\x00\x00')  # Placeholder for total file size
        wav_file.write(b'WAVE')
        wav_file.write(b'fmt ')
        wav_file.write(ustruct.pack('<IHHIIHH', 16, 1, num_channels, 16000, 32000, num_channels * sample_width, sample_width * 8))
        # Write the PCM data
        wav_file.write(b'data')
        wav_file.write(ustruct.pack('<I', len(pcm_buffer) * sample_width))
        for sample in pcm_buffer:
            wav_file.write(sample.to_bytes(2, 'little'))
        # Calculate the total file size and update the WAV file header
        total_size = len(pcm_buffer) * sample_width + 36
        wav_file.seek(4)
        wav_file.write(ustruct.pack('<I', total_size - 8))

def play_audio(filename="output.wav"):
    from machine import I2S
    bck_pin = Pin(26) # Bit clock output
    ws_pin = Pin(25) # Word clock output
    sdout_pin = Pin(33) # Serial data output

    with open(filename, "rb") as f:
        content = f.read()
    wavarray = bytearray(len(content))
    wavarray.extend(content)

    pm.setPower(0x03, 1)    
    #Adjust the shift parameter to increase or decrease the volume
    I2S.shift(buf=wavarray, bits=16, shift=1) # Positive for left shift (volume increase by 6dB), negative for right shift (volume decrease by 6dB).
    audio_out = I2S(0, sck=bck_pin, ws=ws_pin, sd=sdout_pin, mode=I2S.TX, bits=16, format=I2S.MONO, rate=24000, ibuf=2048)
    audio_out.write(wavarray)
    pm.setPower(0x03, 0)
    print("done playing")


record_audio(5)
play_audio()

The function "play_audio" work fine with a regular wav file found on the internet.
But the recordings from the microphone are a simple "POC" sound.

Is there any obvious mistake somewhere?

[DonvdH]

You're using a decimation factor of 16:
pdm_buffer = cic_filter(pdm_buffer, 4, 16, 1)

With a corresponding pcm buffer length:
pcm_data = array.array('H', [0] * (len(pdm_data) // 16))

This results in a pcm_buffer size of 1500 which is then being played at a bitrate of 24000. So the sound will only last 1/16th of a second.

I believe you need a decimation factor of 1 (no decimation).
From what I understand, decimation is only needed if you're recording PDM audio at a higher bitrate then the PCM bitrate you want to convert it to.

[thiswillbeyourgithub]

Thank you as usual.

Btw I also checked and the function play_audio is working perfectly well if given proper audio with the appropriate sample rate.

I notice that the microphone output is somewhat unreliable: I sometimes have a pdm_data full of 0 straight out of the microphone and don't really understand why. Might be related to this :

I stumbled upon this in the datasheet of the microphone:

Doesn't that mean that to be active, the microphone would need to receive a clock signal in the MHz range? Because I don't think we are fast enough with my current delayed loop.

Here's my latest version of the code:

import micropython
import array
import time
import ustruct
import math
from tempos import pm
from drivers.axp202 import LD04
from machine import I2S, Pin

micropython.alloc_emergency_exception_buf(100)

# Slow down audio capture to calibrate time
pacer = micropython.const(4000)
# microphone pins:
pdm_clk_pin = Pin(0, Pin.OUT)
pdm_data_pin = Pin(2, Pin.IN)
# speaker pins:
bck_pin = Pin(26) # Bit clock output
ws_pin = Pin(25) # Word clock output
sdout_pin = Pin(33) # Serial data output
# audio play chunk size
play_chunk_size = micropython.const(1024)
# WAV file parameters
sample_width = micropython.const(2)  # 2 bytes for 16-bit PCM
num_channels = micropython.const(1)  # Mono

# PCM to PDM filters  TODO: use viper emitter
# @micropython.native
def cic_filter(pdm_data, num_stages, decimation_factor):
    # Cascaded Integrator-comb filter
    integrator = [0] * num_stages
    comb = [0] * num_stages
    output = array.array('H', [0] * len(pdm_data))
    for i in range(len(pdm_data)):
        integrator[0] += pdm_data[i]
        for j in range(1, num_stages):
            integrator[j] += integrator[j - 1]
        if i % decimation_factor == 0:
            for j in range(num_stages):
                comb[j] = integrator[j] - comb[j]
            output[i // decimation_factor] = comb[-1]
    return output

@micropython.native
def pdm_to_pcm(pdm_data):
    # PDM to PCM conversion
    pcm_data = array.array('H', [0] * (len(pdm_data) // 16))
    for i in range(0, len(pdm_data), 16):
        pcm_data[i // 16] = sum(pdm_data[i:i+16]) // 16
    return pcm_data

@micropython.native
def low_pass_filter(pcm_data, cutoff_frequency, sample_rate):
    # Low pass decimation filter
    output = array.array('H', [0] * len(pcm_data))
    alpha = 2 * math.pi * cutoff_frequency / sample_rate
    for i in range(1, len(pcm_data)):
        output[i] = int(alpha * pcm_data[i] + (1 - alpha) * output[i - 1])
    return output

@micropython.native
def anti_aliasing_filter(pcm_data, cutoff_frequency, sample_rate):
    # High frequency noise anti aliasing low pass filter
    output = array.array('H', [0] * len(pcm_data))
    alpha = 2 * math.pi * cutoff_frequency / sample_rate
    for i in range(1, len(pcm_data)):
        output[i] = int(alpha * pcm_data[i] + (1 - alpha) * output[i - 1])
    return output


@micropython.native
def capture_pdm_data(length_s):
    pdm_data = array.array('H', [0] * length_s * 16000)
    pm.setPower(LD04, 1)
    pdm_clk_pin.value(0)
    start = time.ticks_ms()

    for i in range(len(pdm_data)):
        prev = time.ticks_cpu()
        pdm_data[i] = pdm_data_pin.value()
        pdm_clk_pin.on()
        pdm_clk_pin.off()
        while time.ticks_diff(time.ticks_cpu(), prev) <= pacer:
            pass
    print(time.ticks_diff(time.ticks_ms(), start) / 1000)
    if not max(pdm_data) == 1 and min(pdm_data) == 0:
        print("INVALID MICROPHONE OUTPUT")
    pm.setPower(LD04, 0)
    return pdm_data

def record_audio(length_s=1):
    print("recording")
    pdm_buffer = capture_pdm_data(length_s)

    # Convert PDM to PCM from pdm_buffer into pcm_buffer here
    print("applying filters")
    pdm_buffer = cic_filter(pdm_buffer, 4, 1)
    pcm_buffer = pdm_to_pcm(pdm_buffer)
    #pcm_buffer = low_pass_filter(pcm_buffer, 4000, 16000)
    #pcm_buffer = anti_aliasing_filter(pcm_buffer, 4000, 16000)

    # Open the WAV file for writing
    print("writing to file")
    with open('output.wav', 'wb') as wav_file:
        # Write the WAV file header
        wav_file.write(b'RIFF')
        wav_file.write(b'\x00\x00\x00\x00')  # Placeholder for total file size
        wav_file.write(b'WAVE')
        wav_file.write(b'fmt ')
        wav_file.write(ustruct.pack('<IHHIIHH', 16, 1, num_channels, 16000, 32000, num_channels * sample_width, sample_width * 8))
        wav_file.write(b'data')
        wav_file.write(ustruct.pack('<I', len(pcm_buffer) * sample_width))
        wav_file.write(pcm_buffer)
        # Calculate the total file size and update the WAV file header
        total_size = len(pcm_buffer) * sample_width + 36
        wav_file.seek(4)
        wav_file.write(ustruct.pack('<I', total_size - 8))

def play_audio(filename="output.wav", volume=-3):
    print("playing audio")
    pm.setPower(0x03, 1)    
    audio_out = I2S(
        0,
        sck=bck_pin,
        ws=ws_pin,
        sd=sdout_pin,
        mode=I2S.TX,
        bits=16,
        format=I2S.MONO,
        rate=16000,
        ibuf=play_chunk_size)
    start = time.ticks_ms()
    with open(filename, "rb") as f:
        while True:
            chunk = f.read(play_chunk_size)
            if not chunk:
                break
            chunk = bytearray(chunk)
            I2S.shift(buf=chunk, bits=16, shift=volume) # Positive for left shift (volume increase by 6dB), negative for right shift (volume decrease by 6dB).
            audio_out.write(chunk)
    print(time.ticks_diff(time.ticks_ms(), start) / 1000)
        
    audio_out.deinit()
    pm.setPower(LD04, 0)
    print("done playing")


record_audio(5)
play_audio()
#play_audio("Wav_868kb_mono_16k.wav")

[DonvdH]

Regarding the play duration, you're still dividing the buffer length by 16 in pdm_to_pcm:
pcm_data = array.array('H', [0] * (len(pdm_data) // 16)) <-- The 16 here means divide by 16
The pdm_to_pcm(pdm_data) appears to geared towards decimation and averaging, you could try skipping it entirely, so leave out this line:
pcm_buffer = pdm_to_pcm(pdm_buffer)

And then just replace pcm_buffer with pdm_buffer in later lines.

The 0's might indeed be caused by the clock speed being too low or the delay before reading the data pin.

Try changing:

        pdm_data[i] = pdm_data_pin.value()
        pdm_clk_pin.on()
        pdm_clk_pin.off()
        while time.ticks_diff(time.ticks_cpu(), prev) <= pacer:
            pass

to this:

        pdm_clk_pin.on()
        pdm_clk_pin.off()
        pdm_data[i] = pdm_data_pin.value()
        while time.ticks_diff(time.ticks_cpu(), prev) <= pacer:
            pdm_clk_pin.on()
            pdm_clk_pin.off()

The first/old code moves the clock line up/down, then waits a while and only then reads the data pin. It's likely that the data pin has then moved to 0 after a while.

[thiswillbeyourgithub]

Regarding the play duration, you're still dividing the buffer length by 16 in pdm_to_pcm:
pcm_data = array.array('H', [0] * (len(pdm_data) // 16)) <-- The 16 here means divide by 16
The pdm_to_pcm(pdm_data) appears to geared towards decimation and averaging, you could try skipping it entirely, so leave out this line:
pcm_buffer = pdm_to_pcm(pdm_buffer)

Oh okay. I thought that the length had to be shorter because 16 1s and 0s are turned into a single value if the recording is 16bits, my bad.

The first/old code moves the clock line up/down, then waits a while and only then reads the data pin. It's likely that the data pin has then moved to 0 after a while.

Very smart thanks a lot.

I'm testing a bunch right now and will report back.

[thiswillbeyourgithub]

Welp here's where I'm at for the day.

I still get noise at the end. I added a few prints to get an idea of the signal and how it's being processed.

import micropython
import array
import time
import ustruct
import math
from tempos import pm
from drivers.axp202 import LD04
from machine import I2S, Pin

micropython.alloc_emergency_exception_buf(100)

# Slow down audio capture to calibrate time
pacer = micropython.const(6400)
wakeup_delay = micropython.const(20)  # needs to be stimulated for 10ms to activate
# microphone pins:
pdm_clk_pin = Pin(0, Pin.OUT)
pdm_data_pin = Pin(2, Pin.IN)
# speaker pins:
bck_pin = Pin(26) # Bit clock output
ws_pin = Pin(25) # Word clock output
sdout_pin = Pin(33) # Serial data output
# audio play chunk size
play_chunk_size = micropython.const(1024)
# WAV file parameters
sample_width = micropython.const(2)  # 2 bytes for 16-bit PCM
num_channels = micropython.const(1)  # Mono

@micropython.native
def capture_pdm_data(length_s):
    pdm_data = array.array('H', [0] * length_s * 16000)
    pm.setPower(LD04, 1)
    pdm_clk_pin.value(0)

    prev = time.ticks_ms()
    while time.ticks_diff(time.ticks_ms(), prev) <= wakeup_delay:
        pdm_clk_pin.on()
        pdm_clk_pin.off()
    
    start = time.ticks_ms()
    for i in range(len(pdm_data)):
        pdm_data[i] = pdm_data_pin.value()
        prev = time.ticks_cpu()
        while time.ticks_diff(time.ticks_cpu(), prev) <= pacer:
            # the microphone is active only above 1MHz
            pdm_clk_pin.on()
            pdm_clk_pin.off()
    print(time.ticks_diff(time.ticks_ms(), start) / 1000)
    
    if not (max(pdm_data) == 1 and min(pdm_data) == 0):
        print("INVALID MICROPHONE OUTPUT")
    pm.setPower(LD04, 0)
    return pdm_data

@micropython.native
def pdm_to_pcm(pdm_data):
    pcm_data = array.array('h', [0]*(len(pdm_data)//16))  # Decimation factor of 16
    for i in range(0, len(pdm_data), 16):
        # Sum up every 16 PDM samples
        pcm_data[i//16] = sum(pdm_data[i:i+16])
    # Remove DC offset
    mean = int(sum(pcm_data) / len(pcm_data))
    for i in range(len(pcm_data)):
        pcm_data[i] -= mean
    return pcm_data

@micropython.native
def interpolate(pcm_data):
    interpolated_data = array.array('h', [0]*(len(pcm_data)*16))
    for i in range(len(pcm_data)-1):
        for j in range(16):
            interpolated_data[i*16+j] = int(pcm_data[i] + j*(pcm_data[i+1]-pcm_data[i])/16)
    return interpolated_data

@micropython.native
def low_pass_filter(pcm_data, cutoff_freq):
    filtered_data = array.array('h', [0]*len(pcm_data))
    for i in range(1, len(pcm_data)):
        filtered_data[i] = int((pcm_data[i-1] * (1 - cutoff_freq)) + (pcm_data[i] * cutoff_freq))
    return filtered_data

@micropython.native
def anti_aliasing_filter(pcm_data):
    filtered_data = array.array('h', [0]*len(pcm_data))
    for i in range(2, len(pcm_data)):
        filtered_data[i] = int((pcm_data[i-2] + 2*pcm_data[i-1] + pcm_data[i]) / 4)
    return filtered_data

@micropython.native
def record_audio(length_s=1):
    print("recording")
    buffer = capture_pdm_data(length_s)

    # Convert PDM to PCM from pdm_buffer into pcm_buffer here
    print("applying filters")
    print("0 len: {} max: {} min: {} mean: {}".format(len(buffer), max(buffer), min(buffer), sum(buffer) / len(buffer)))
    buffer = pdm_to_pcm(buffer)
    print("1 len: {} max: {} min: {} mean: {}".format(len(buffer), max(buffer), min(buffer), sum(buffer) / len(buffer)))
    buffer = interpolate(buffer)
    print("2 len: {} max: {} min: {} mean: {}".format(len(buffer), max(buffer), min(buffer), sum(buffer) / len(buffer)))
    buffer = low_pass_filter(buffer, 2000)
    print("3 len: {} max: {} min: {} mean: {}".format(len(buffer), max(buffer), min(buffer), sum(buffer) / len(buffer)))
    buffer = anti_aliasing_filter(buffer)
    print("4 len: {} max: {} min: {} mean: {}".format(len(buffer), max(buffer), min(buffer), sum(buffer) / len(buffer)))

    # Open the WAV file for writing
    print("writing to file")
    with open('output.wav', 'wb') as wav_file:
        # Write the WAV file header
        wav_file.write(b'RIFF')
        wav_file.write(b'\x00\x00\x00\x00')  # Placeholder for total file size
        wav_file.write(b'WAVE')
        wav_file.write(b'fmt ')
        wav_file.write(ustruct.pack('<IHHIIHH', 16, 1, num_channels, 16000, 32000, num_channels * sample_width, sample_width * 8))
        wav_file.write(b'data')
        wav_file.write(ustruct.pack('<I', len(buffer) * sample_width))
        wav_file.write(buffer)
        # Calculate the total file size and update the WAV file header
        total_size = len(buffer) * sample_width + 36
        wav_file.seek(4)
        wav_file.write(ustruct.pack('<I', total_size - 8))

@micropython.native
def play_audio(filename="output.wav", volume=-1):
    print("playing audio")
    pm.setPower(0x03, 1)    
    audio_out = I2S(
        0,
        sck=bck_pin,
        ws=ws_pin,
        sd=sdout_pin,
        mode=I2S.TX,
        bits=16,
        format=I2S.MONO,
        rate=16000,
        ibuf=play_chunk_size)
    start = time.ticks_ms()
    with open(filename, "rb") as f:
        while True:
            chunk = f.read(play_chunk_size)
            if not chunk:
                break
            chunk = bytearray(chunk)
            I2S.shift(buf=chunk, bits=16, shift=volume) # Positive for left shift (volume increase by 6dB), negative for right shift (volume decrease by 6dB).
            audio_out.write(chunk)
    print(time.ticks_diff(time.ticks_ms(), start) / 1000)
        
    audio_out.deinit()
    pm.setPower(LD04, 0)
    print("done playing")


record_audio(5)
play_audio()
#play_audio("Wav_868kb_mono_16k.wav")

Output:

>>> %Run -c $EDITOR_CONTENT
recording
5.064
applying filters
0 len: 80000 max: 1 min: 0 mean: 0.9943875 <- out of the mic
1 len: 5000 max: 1 min: -6 mean: 0.9102 <- after pdm_to_pcm
2 len: 80000 max: 1 min: -6 mean: 0.829275 <- after interpolate
3 len: 80000 max: 2000 min: -2000 mean: 0.9792749 <- after low pass filter
4 len: 80000 max: 1000 min: -999 mean: 0.9729375 <- after anti aliasing
writing to file
playing audio
5.006
done playing

My guess is that my filters are not functionnal. I tried a lot of variation of filter order, arguments, rewriting from scratch, etc but I'm giving up for now. Any pointers? What kind of values should I actually expect as output of each filter? Am I supposed to get frequencies at some point? Isn't the interpolation upsampling stupidly space-wasting?

[DonvdH]

Best course of action would be to first use the PDM2PCM tool I mentioned to validate whether the PDM data is valid at all:
https://github.com/siorpaes/pdm_playground/tree/master/pdm2pcm

Now you basically don't know whether the PDM is valid or whether the filters are failing.

Also I wonder where you found the Micropython filters, the context may tell something about the suitability of the filters and how they should be used.

[thiswillbeyourgithub]

I'll give it a go thank you.

Also I wonder where you found the Micropython filters, the context may tell something about the suitability of the filters and how they should be used.

You're not going to like it but it's a mix of intuition and GPT-4 prompting. Unfortunately it's getting worse and worse as they RLHF it but it's still faster than trying to compensate my lack of background.

[DonvdH]

You're not going to like it but it's a mix of intuition and GPT-4 prompting. Unfortunately it's getting worse and worse as they RLHF it but it's still faster than trying to compensate my lack of background.

ChatGPT can be a great tool when it has seen a lot of examples. But when it hasn't, it can convincingly give incorrect results. So that's something to keep in mind at least.

If I look at https://github.com/siorpaes/pdm_playground/blob/master/pdm2pcm/pdm2pcm.c then appears that it requires at least a decimation factor of 64. So it looks that for 16kbit PCM, 1024000 PDM samples would be needed.

When I look at https://github.com/siorpaes/pdm_playground/blob/master/pdm2pcm/OpenPDMFilter.c then it appears that it is really reconstructing audio data from the bit stream:

    for (i = 0; i < decimation; i++) {
      coef[j][i] = sinc[j * decimation + i];
      sum += sinc[j * decimation + i];
    }
  }

  sub_const = sum >> 1;
  div_const = sub_const * Param->MaxVolume / 32768 / FILTER_GAIN;
  div_const = (div_const == 0 ? 1 : div_const);

Also I read the following here:
https://www.cuidevices.com/blog/pdm-vs-i2s-comparing-digital-interfaces-in-mems-microphones
" If the audio sample rate is the industry standard of 44.1 kHz with 8 bits of precision, then a mono channel will need a clock speed of at least 352.8 kHz."

So in contrast to what I previously believed, a higher PDM bitrate really appears to be required and the highest bitrate that can be achieved using Micropython is about 55khz as we previously learned.

So then I have to agree with @devnoname120 that the best (or only) possible route would be to compile the I2S-PDM driver he mentioned into the firmware that @jeffmer has provided.

[thiswillbeyourgithub]

Alright. Well thank you immensely @DonvdH for helping me along the way.

The next step is then to wait for @jeffmer to answer this :

I'm not sure how much effort it would require to port it over to a recent MicroPython. @jeffmer Is https://github.com/jeffmer/micropython/tree/GPIO_WAKEUP the right branch for compiling your custom firmware.bin?

[thiswillbeyourgithub]

Hello @jeffmer

I was wondering if you had time to answer the question raised by @devnoname120 here? That would save me plausibly liters of tears and sweat :)

My enthusiasm about the chatgpt watch has not faded in the least

[jeffmer]

Apologies, I missed the question from @devnoname120.

The answer is yes, you have to build the GPIO_WAKEUP branch to make firmware.bin.

[thiswillbeyourgithub]

Great. And any guidance you mught have or specific warnings or helpful tips before I dive into adding i2s microphone into it? The other repo linked is making me optimistic but I have no idea what i'm doing frankly.

[jeffmer]

Try building the firmware without your mods first and check the firmware works as before:-)

[thiswillbeyourgithub]

Hello again. So I did build the firmware from your repo using those instructions.

It flashes fine then running "verify_flash" shows no error.

But when trying install.sh the script just hangs at Copying root files (first step) with no more output regardless of how long I wait.

I did flash back your firmware and ran install.sh successfuly so the issue seems to be with the build.

What should I do :)? Are there special arguments to pass when building?

EDIT: also I can't seem to get a prompt when using mpremote with this build.

edit : I may have fixed it. Will post a new message if help is needed :)

[thiswillbeyourgithub]

Hello again again. So I managed to get a working REPL from the watch using the firmware using those instructions.

But when doing import loader I get issues caused by the decorator micropython.viper. So apparently this build has no viper? I disabled every occurence of this decorator to see if that would work and I get another error in drivers/axp202.py caused by Pin not having any attribute called IRQ_LOW_LEVEL.

So I decided to stop digging until I get more information from you @jeffmer because I'm guessing lots more might be missing.

Any idea what's happening? Also to make the REPL flashing work I had to flash using idf.py flash instead of mpremote. mpremote would not raise any issue when called to flash the firmware but I never got a REPL whereas idf worked but was surprisingly faster (like 10x faster than mpremote) and I couldn't find an equivalent to mpremote verify_flash so couldn't check further.

When building, I'm assuming the default board is the right one. But is there another board I should specify like ESP32_GENERIC_S3 for example?

[jeffmer]

Hi, a better approach might be to clone the official Micropython repository and build it. I have only changed one file which you can find here https://github.com/jeffmer/micropython/blob/master/ports/esp32/machine_pin.c
I have not updated my repository for some time. You need to build the general ESP32 with the PSRAM option.

In summary, clone the official repository and replace the machine_pin.c file which has the gpio_wakeup stuff. You should have viper etc.

[thiswillbeyourgithub]

Thanks for the quick reply. So I did try with the recent micropython and putting the file machine_pin.c then running idf.py fullclean && idf.py build.
(I also did try idf.py build -D BOARD_VARIANT=SPIRAM and it does not fix this specific issue but I think I should deal with this only after successfuly building without arguments first)
Here's the error that I get:

MPY umqtt/simple.py
MPY upysh.py
GEN /USERPATH/twatch_microphone/micropython/ports/esp32/build/frozen_content.c
[984/1216] Generating ../../genhdr/moduledefs.collectedModule registrations updated
[1181/1216] Building C object esp-idf/main_esp32/CMakeFiles/__idf_main_esp32.dir/__/machine_pin.c.objFAILED: esp-idf/main_esp32/CMakeFiles/__idf_main_esp32.dir/__/machine_pin.c.obj
/USERPATH/.espressif/tools/xtensa-esp32-elf/esp-2022r1-11.2.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc -DFFCONF_H=\"/USERPATH/twatch_microphone/micropython/lib/oofatfs/ffconf.h\" -DLFS1_NO_ASSERT -DLFS1_NO_DEBUG -DLFS1_NO_ERROR -DLFS1_NO_MALLOC -DLFS1_NO_WARN -DLFS2_NO_ASSERT -DLFS2_NO_DEBUG -DLFS2_NO_ERROR -DLFS2_NO_MALLOC -DLFS2_NO_WARN -DMBEDTLS_CONFIG_FILE=\"mbedtls/esp_config.h\" -DMICROPY_ESP_IDF_4=1 -DMICROPY_MODULE_FROZEN_MPY="(1)" -DMICROPY_PY_BTREE=1 -DMICROPY_QSTR_EXTRA_POOL=mp_qstr_frozen_const_pool -DMICROPY_VFS_FAT=1 -DMICROPY_VFS_LFS2=1 -DSOC_MMU_PAGE_SIZE=CONFIG_MMU_PAGE_SIZE -D__DBINTERFACE_PRIVATE=1 -Dvirt_fd_t="void*" -I/USERPATH/twatch_microphone/micropython/ports/esp32/build/config -I/USERPATH/twatch_microphone/micropython -I/USERPATH/twatch_microphone/micropython/lib/berkeley-db-1.xx/PORT/include -I/USERPATH/twatch_microphone/micropython/ports/esp32 -I/USERPATH/twatch_microphone/micropython/ports/esp32/boards/ESP32_GENERIC -I/USERPATH/twatch_microphone/micropython/ports/esp32/build -I/USERPATH/twatch_microphone/esp-idf/components/bt/host/nimble/nimble -I/USERPATH/twatch_microphone/esp-idf/components/newlib/platform_include -I/USERPATH/twatch_microphone/esp-idf/components/freertos/FreeRTOS-Kernel/include -I/USERPATH/twatch_microphone/esp-idf/components/freertos/esp_additions/include/freertos -I/USERPATH/twatch_microphone/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/include -I/USERPATH/twatch_microphone/esp-idf/components/freertos/esp_additions/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_hw_support/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_hw_support/include/soc -I/USERPATH/twatch_microphone/esp-idf/components/esp_hw_support/include/soc/esp32 -I/USERPATH/twatch_microphone/esp-idf/components/esp_hw_support/port/esp32/. -I/USERPATH/twatch_microphone/esp-idf/components/esp_hw_support/port/esp32/private_include -I/USERPATH/twatch_microphone/esp-idf/components/heap/include -I/USERPATH/twatch_microphone/esp-idf/components/log/include -I/USERPATH/twatch_microphone/esp-idf/components/soc/include -I/USERPATH/twatch_microphone/esp-idf/components/soc/esp32/. -I/USERPATH/twatch_microphone/esp-idf/components/soc/esp32/include -I/USERPATH/twatch_microphone/esp-idf/components/hal/esp32/include -I/USERPATH/twatch_microphone/esp-idf/components/hal/include -I/USERPATH/twatch_microphone/esp-idf/components/hal/platform_port/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_rom/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_rom/include/esp32 -I/USERPATH/twatch_microphone/esp-idf/components/esp_rom/esp32 -I/USERPATH/twatch_microphone/esp-idf/components/esp_common/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_system/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_system/port/soc -I/USERPATH/twatch_microphone/esp-idf/components/esp_system/port/include/private -I/USERPATH/twatch_microphone/esp-idf/components/xtensa/include -I/USERPATH/twatch_microphone/esp-idf/components/xtensa/esp32/include -I/USERPATH/twatch_microphone/esp-idf/components/lwip/include -I/USERPATH/twatch_microphone/esp-idf/components/lwip/include/apps -I/USERPATH/twatch_microphone/esp-idf/components/lwip/include/apps/sntp -I/USERPATH/twatch_microphone/esp-idf/components/lwip/lwip/src/include -I/USERPATH/twatch_microphone/esp-idf/components/lwip/port/esp32/include -I/USERPATH/twatch_microphone/esp-idf/components/lwip/port/esp32/include/arch -I/USERPATH/twatch_microphone/esp-idf/components/app_update/include -I/USERPATH/twatch_microphone/esp-idf/components/bootloader_support/include -I/USERPATH/twatch_microphone/esp-idf/components/bootloader_support/bootloader_flash/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_app_format/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_partition/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/include/esp32/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/common/osi/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/common/api/include/api -I/USERPATH/twatch_microphone/esp-idf/components/bt/common/btc/profile/esp/blufi/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/common/btc/profile/esp/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/host/nimble/nimble/nimble/host/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/host/nimble/nimble/nimble/host/services/ans/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/host/nimble/nimble/nimble/host/services/bas/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/host/nimble/nimble/nimble/host/services/dis/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/host/nimble/nimble/nimble/host/services/gap/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/host/nimble/nimble/nimble/host/services/gatt/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/host/nimble/nimble/nimble/host/services/ias/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/host/nimble/nimble/nimble/host/services/ipss/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/host/nimble/nimble/nimble/host/services/lls/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/host/nimble/nimble/nimble/host/services/tps/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/host/nimble/nimble/nimble/host/util/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/host/nimble/nimble/nimble/host/store/ram/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/host/nimble/nimble/nimble/host/store/config/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/porting/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/host/nimble/nimble/porting/nimble/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/host/nimble/port/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/host/nimble/nimble/porting/npl/freertos/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/host/nimble/nimble/nimble/include -I/USERPATH/twatch_microphone/esp-idf/components/bt/host/nimble/esp-hci/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_timer/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_wifi/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_event/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_phy/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_phy/esp32/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_netif/include -I/USERPATH/twatch_microphone/esp-idf/components/driver/include -I/USERPATH/twatch_microphone/esp-idf/components/driver/deprecated -I/USERPATH/twatch_microphone/esp-idf/components/driver/esp32/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_pm/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_ringbuf/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_adc/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_adc/interface -I/USERPATH/twatch_microphone/esp-idf/components/esp_adc/esp32/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_adc/deprecated/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_eth/include -I/USERPATH/twatch_microphone/esp-idf/components/esp_psram/include -I/USERPATH/twatch_microphone/esp-idf/components/mbedtls/port/include -I/USERPATH/twatch_microphone/esp-idf/components/mbedtls/mbedtls/include -I/USERPATH/twatch_microphone/esp-idf/components/mbedtls/mbedtls/library -I/USERPATH/twatch_microphone/esp-idf/components/mbedtls/esp_crt_bundle/include -I/USERPATH/twatch_microphone/esp-idf/components/nvs_flash/include -I/USERPATH/twatch_microphone/esp-idf/components/sdmmc/include -I/USERPATH/twatch_microphone/esp-idf/components/spi_flash/include -I/USERPATH/twatch_microphone/esp-idf/components/ulp/ulp_common/include -I/USERPATH/twatch_microphone/esp-idf/components/ulp/ulp_common/include/esp32 -I/USERPATH/twatch_microphone/esp-idf/components/ulp/ulp_fsm/include -I/USERPATH/twatch_microphone/esp-idf/components/ulp/ulp_fsm/include/esp32 -I/USERPATH/twatch_microphone/esp-idf/components/vfs/include -I/USERPATH/twatch_microphone/micropython/ports/esp32/managed_components/espressif__mdns/include -I/USERPATH/twatch_microphone/esp-idf/components/console -mlongcalls -Wno-frame-address  -fdiagnostics-color=always -ffunction-sections -fdata-sections -Wall -Werror=all -Wno-error=unused-function -Wno-error=unused-variable -Wno-error=deprecated-declarations -Wextra -Wno-unused-parameter -Wno-sign-compare -Wno-enum-conversion -gdwarf-4 -ggdb -O2 -fmacro-prefix-map=/USERPATH/twatch_microphone/micropython/ports/esp32=. -fmacro-prefix-map=/USERPATH/twatch_microphone/esp-idf=/IDF -fstrict-volatile-bitfields -Wno-error=unused-but-set-variable -fno-jump-tables -fno-tree-switch-conversion -DconfigENABLE_FREERTOS_DEBUG_OCDAWARE=1 -std=gnu17 -Wno-old-style-declaration -D_GNU_SOURCE -DIDF_VER=\"v5.0.2\" -DESP_PLATFORM -DNDEBUG -D_POSIX_READER_WRITER_LOCKS -Wno-clobbered -Wno-deprecated-declarations -Wno-missing-field-initializers -MD -MT esp-idf/main_esp32/CMakeFiles/__idf_main_esp32.dir/__/machine_pin.c.obj -MF esp-idf/main_esp32/CMakeFiles/__idf_main_esp32.dir/__/machine_pin.c.obj.d -o esp-idf/main_esp32/CMakeFiles/__idf_main_esp32.dir/__/machine_pin.c.obj -c /USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c
In file included from /USERPATH/twatch_microphone/micropython/py/mpstate.h:33,
                 from /USERPATH/twatch_microphone/micropython/py/runtime.h:29,
                 from /USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c:36:
/USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c: In function 'machine_pin_find':
/USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c:98:33: error: 'machine_pin_type' undeclared (first use in this function); did you mean 'machine_dac_type'?
   98 |     if (mp_obj_is_type(pin_in, &machine_pin_type)) {
      |                                 ^~~~~~~~~~~~~~~~
/USERPATH/twatch_microphone/micropython/py/misc.h:54:61: note: in definition of macro 'MP_STATIC_ASSERT'
   54 | #define MP_STATIC_ASSERT(cond) ((void)sizeof(char[1 - 2 * !(cond)]))
      |                                                             ^~~~
/USERPATH/twatch_microphone/micropython/py/obj.h:931:5: note: in expansion of macro 'MP_STATIC_ASSERT_NONCONSTEXPR'
  931 |     MP_STATIC_ASSERT_NONCONSTEXPR((t) != &mp_type_bool), assert((t) != &mp_type_bool),         \
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/USERPATH/twatch_microphone/micropython/py/obj.h:937:31: note: in expansion of macro 'mp_type_assert_not_bool_int_str_nonetype'
  937 | #define mp_obj_is_type(o, t) (mp_type_assert_not_bool_int_str_nonetype(t) && mp_obj_is_exact_type(o, t))
      |                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c:98:9: note: in expansion of macro 'mp_obj_is_type'
   98 |     if (mp_obj_is_type(pin_in, &machine_pin_type)) {
      |         ^~~~~~~~~~~~~~
/USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c:98:33: note: each undeclared identifier is reported only once for each function it appears in
   98 |     if (mp_obj_is_type(pin_in, &machine_pin_type)) {
      |                                 ^~~~~~~~~~~~~~~~
/USERPATH/twatch_microphone/micropython/py/misc.h:54:61: note: in definition of macro 'MP_STATIC_ASSERT'
   54 | #define MP_STATIC_ASSERT(cond) ((void)sizeof(char[1 - 2 * !(cond)]))
      |                                                             ^~~~
/USERPATH/twatch_microphone/micropython/py/obj.h:931:5: note: in expansion of macro 'MP_STATIC_ASSERT_NONCONSTEXPR'
  931 |     MP_STATIC_ASSERT_NONCONSTEXPR((t) != &mp_type_bool), assert((t) != &mp_type_bool),         \
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/USERPATH/twatch_microphone/micropython/py/obj.h:937:31: note: in expansion of macro 'mp_type_assert_not_bool_int_str_nonetype'
  937 | #define mp_obj_is_type(o, t) (mp_type_assert_not_bool_int_str_nonetype(t) && mp_obj_is_exact_type(o, t))
      |                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c:98:9: note: in expansion of macro 'mp_obj_is_type'
   98 |     if (mp_obj_is_type(pin_in, &machine_pin_type)) {
      |         ^~~~~~~~~~~~~~
In file included from /USERPATH/twatch_microphone/micropython/py/mpstate.h:35,
                 from /USERPATH/twatch_microphone/micropython/py/runtime.h:29,
                 from /USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c:36:
/USERPATH/twatch_microphone/micropython/py/obj.h:931:86: error: left-hand operand of comma expression has no effect [-Werror=unused-value]
  931 |     MP_STATIC_ASSERT_NONCONSTEXPR((t) != &mp_type_bool), assert((t) != &mp_type_bool),         \
      |                                                                                      ^
/USERPATH/twatch_microphone/micropython/py/obj.h:937:31: note: in expansion of macro 'mp_type_assert_not_bool_int_str_nonetype'
  937 | #define mp_obj_is_type(o, t) (mp_type_assert_not_bool_int_str_nonetype(t) && mp_obj_is_exact_type(o, t))
      |                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c:98:9: note: in expansion of macro 'mp_obj_is_type'
   98 |     if (mp_obj_is_type(pin_in, &machine_pin_type)) {
      |         ^~~~~~~~~~~~~~
/USERPATH/twatch_microphone/micropython/py/obj.h:932:84: error: left-hand operand of comma expression has no effect [-Werror=unused-value]
  932 |     MP_STATIC_ASSERT_NONCONSTEXPR((t) != &mp_type_int), assert((t) != &mp_type_int),           \
      |                                                                                    ^
/USERPATH/twatch_microphone/micropython/py/obj.h:937:31: note: in expansion of macro 'mp_type_assert_not_bool_int_str_nonetype'
  937 | #define mp_obj_is_type(o, t) (mp_type_assert_not_bool_int_str_nonetype(t) && mp_obj_is_exact_type(o, t))
      |                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c:98:9: note: in expansion of macro 'mp_obj_is_type'
   98 |     if (mp_obj_is_type(pin_in, &machine_pin_type)) {
      |         ^~~~~~~~~~~~~~
/USERPATH/twatch_microphone/micropython/py/obj.h:933:84: error: left-hand operand of comma expression has no effect [-Werror=unused-value]
  933 |     MP_STATIC_ASSERT_NONCONSTEXPR((t) != &mp_type_str), assert((t) != &mp_type_str),           \
      |                                                                                    ^
/USERPATH/twatch_microphone/micropython/py/obj.h:937:31: note: in expansion of macro 'mp_type_assert_not_bool_int_str_nonetype'
  937 | #define mp_obj_is_type(o, t) (mp_type_assert_not_bool_int_str_nonetype(t) && mp_obj_is_exact_type(o, t))
      |                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c:98:9: note: in expansion of macro 'mp_obj_is_type'
   98 |     if (mp_obj_is_type(pin_in, &machine_pin_type)) {
      |         ^~~~~~~~~~~~~~
/USERPATH/twatch_microphone/micropython/py/obj.h:934:94: error: left-hand operand of comma expression has no effect [-Werror=unused-value]
  934 |     MP_STATIC_ASSERT_NONCONSTEXPR((t) != &mp_type_NoneType), assert((t) != &mp_type_NoneType), \
      |                                                                                              ^
/USERPATH/twatch_microphone/micropython/py/obj.h:937:31: note: in expansion of macro 'mp_type_assert_not_bool_int_str_nonetype'
  937 | #define mp_obj_is_type(o, t) (mp_type_assert_not_bool_int_str_nonetype(t) && mp_obj_is_exact_type(o, t))
      |                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c:98:9: note: in expansion of macro 'mp_obj_is_type'
   98 |     if (mp_obj_is_type(pin_in, &machine_pin_type)) {
      |         ^~~~~~~~~~~~~~
In file included from /USERPATH/twatch_microphone/micropython/py/mpstate.h:35,
                 from /USERPATH/twatch_microphone/micropython/py/runtime.h:29,
                 from /USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c:36:
/USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c: In function 'machine_pin_irq':
/USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c:342:29: error: 'machine_pin_irq_obj_table' undeclared (first use in this function); did you mean 'machine_pin_obj_table'?
  342 |     return MP_OBJ_FROM_PTR(&machine_pin_irq_obj_table[PIN_OBJ_INDEX(self)]);
      |                             ^~~~~~~~~~~~~~~~~~~~~~~~~
/USERPATH/twatch_microphone/micropython/py/obj.h:321:40: note: in definition of macro 'MP_OBJ_FROM_PTR'
  321 | #define MP_OBJ_FROM_PTR(p) ((mp_obj_t)(p))
      |                                        ^
/USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c: In function 'machine_pin_irq_call':
/USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c:62:44: error: 'machine_pin_irq_obj_table' undeclared (first use in this function); did you mean 'machine_pin_obj_table'?
   62 | #define PIN_IRQ_OBJ_INDEX(self) ((self) - &machine_pin_irq_obj_table[0])
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~
/USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c:419:60: note: in expansion of macro 'PIN_IRQ_OBJ_INDEX'
  419 |     machine_pin_isr_handler((void *)&machine_pin_obj_table[PIN_IRQ_OBJ_INDEX(self)]);
      |                                                            ^~~~~~~~~~~~~~~~~
/USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c: In function 'machine_pin_irq_trigger':
/USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c:62:44: error: 'machine_pin_irq_obj_table' undeclared (first use in this function); did you mean 'machine_pin_obj_table'?
   62 | #define PIN_IRQ_OBJ_INDEX(self) ((self) - &machine_pin_irq_obj_table[0])
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~
/USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c:425:35: note: in expansion of macro 'PIN_IRQ_OBJ_INDEX'
  425 |     uint32_t orig_trig = GPIO.pin[PIN_IRQ_OBJ_INDEX(self)].int_type;
      |                                   ^~~~~~~~~~~~~~~~~
/USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c: In function 'machine_pin_irq':
/USERPATH/twatch_microphone/micropython/ports/esp32/machine_pin.c:343:1: error: control reaches end of non-void function [-Werror=return-type]
  343 | }
      | ^
cc1: some warnings being treated as errors
[1190/1216] Building C object esp-idf/main_esp32/CMakeFiles/__id.../Downloads/twatch_microphone/micropython/lib/littlefs/lfs2.c.objninja: build stopped: subcommand failed.
ninja failed with exit code 1, output of the command is in the /USERPATH/twatch_microphone/micropython/ports/esp32/build/log/idf_py_stderr_output_2559722 and /USERPATH/twatch_microphone/micropython/ports/esp32/build/log/idf_py_stdout_output_2559722

So knowing elementary grade C I poked around with adding some lines from machine_pin.h and it seems to somewhat advance a bit but I'd like your opinion as that would save me from a very slow random walk :).

[jeffmer]

It looks like there have been changes to the Micropython repository that are incompatible. It will take me some time to sort out - hopefully in the next week or so ...

Yes there have been some changes to machine_pin.c in the main repository. It seems to have some code related to GPIO_WAKEUP so my mods may not be necessary. In any case the file I suggested is out of date - sorry!

I will have a look when I get a chance.

[thiswillbeyourgithub]

Thank you so much!

I can't wait to have that AI watch :)

[jeffmer]

OK - my micropython repository now has a GPIO_WAKEUP branch with an updated machine_pin.c which I have tested on a V1 watch. A bit of a struggle as for some reason modmachine.c cancelled all wakeup sources - now fixed.

To make it you need to clone my repository and choose the GPIO_WAKEUP branch.

The make command is:

make BOARD=ESP32_GENERIC BOARD_VARIANT=SPIRAM

Good luck!

PS re AI: Espruino Banglejs2 watch has a working TensorFlow module.

[thiswillbeyourgithub]

Thank you so much! I'll take a look monday

[thiswillbeyourgithub]

Hi!

So building works, flashing works, install.sh works. But when I get to the REPL and type "import loader" I get can't import name viper.

I'm using the GPIO branch of your repo.

Any idea what's the issue?

[jeffmer]

No idea, that did not happen with me. Maybe, rebuild your mpy-cross compiler and recompile the modules that have the viper decorator.

[thiswillbeyourgithub]

OMG I got your build working :)!

I think the issue was that I was not using the mpy-cross compiler when compiling the apps or something.

Thank you immensely.

[thiswillbeyourgithub]

Okay so I then tried to merge the commit from that lemariva repo who apparently got PDM support on esp32 to the recent micropython version.

I managed to make a build and flash it but am not sure I did something wrong so I documented the whole thing in this PR.

I added a few questions at the end and tried to be extra clear to make it as painless as possible to help the clown that I am 🤡 .

Pinging @devnoname120 @jeffmer @DonvdH

Thank you again everyone for getting me this far :)

[thiswillbeyourgithub]

I knew it wasn't a good idea to ask for help 2 days before christmas :)

Can anyone give me a few pointers? My end project will basically be like a super cheapt rabbit r1 or openwearables

Thanks!

Pinging @devnoname120 @jeffmer @DonvdH

[jeffmer]

Sorry I missed this and I am away for the next month so cannot be of much help. To date, I have had very little success with ESP32 audio devices and Micropython:-)

[DonvdH]

@thiswillbeyourgithub:
Nice to see you're still working on this :)

Based on the errors you are getting, have you got I2S_NUM_MAX defined in driver/i2s.h?

Like this:
#define I2S_NUM_MAX 2

[thiswillbeyourgithub]

Thank you very much @DonvdH, indeed that simple line was missing and adding it fixed the first issue I was getting.

The second hurdle is about this section from machine_i2s.c:

const mp_obj_type_t machine_hw_i2s_type = {
{ &mp_type_type },
.name = MP_QSTR_I2S,
.print = machine_hw_i2s_print,
.make_new = machine_hw_i2s_make_new,
.locals_dict = (mp_obj_dict_t *) &machine_hw_i2s_locals_dict,
};

which returned errors like error: 'mp_obj_type_t' {aka 'const struct _mp_obj_type_t'} has no member named 'locals_dict' also for .make_new and .print.

Noticing that the end of machine_i2c.c ended up with a similar looking bunch of lines I intuitively merged it to:

MP_DEFINE_CONST_OBJ_TYPE(
machine_hw_i2s_type,
MP_QSTR_I2S,
MP_TYPE_FLAG_NONE,
make_new, machine_hw_i2s_make_new,
print, machine_hw_i2s_print,
locals_dict, (mp_obj_dict_t *) &machine_hw_i2s_locals_dict
);

which succesfully builds but I don't see PDM in dir(machine.I2S) (is this a sane way of checking if my frankenbuild even worked?)

Thanks!

[jonnor]

There is now a pull request in mainline MicroPython for PDM support for the I2S. It should work on the T-Watch V3
micropython/micropython#14176

[thiswillbeyourgithub]

Thank you so much @jonnor for the heads up. I haven't had the time to give it a try yet but I saw you owned a T-Watch V3 like me. Would you by any chance have any ready-to-use testing code for micropython to play sounds and record stuff? That would be very helpful to me.

[jonnor]

I have not yet had time to try out audio with MicroPython yet. But the MR uses the I2S peripheral, which has been around for a long time. There is a set of examples here that look very good, https://github.com/miketeachman/micropython-i2s-examples/tree/master

[thiswillbeyourgithub]

I had to jump through a few hoops but I apparently successfuly merged the two commits (the one from the PR and the one from @jeffmer for GPIO wakeup).

It ended up vastly simpler to fork the latest micropython, remove the latest commits since one about berkeley-db that didn't seem important but wouldn't build, then apply the PR patch, then manually apply the modification from jeffmer.

You can keep track of that in my fork.

I successfuly build the watch and I see this:

>>> import machine
>>> dir(machine.I2S)
['__class__', '__name__', 'readinto', 'write', 'MONO', 'PDM_RX', 'RX', 'STEREO', 'TX', '__bases__', '__del__', '__dict__', 'deinit', 'init', 'irq', 'shift']

So I think that something went right becauseIIRC PDM_RX was not there before :) :

which succesfully builds but I don't see PDM in dir(machine.I2S) (is this a sane way of checking if my frankenbuild even worked?)

Next steps I assume are testing the microphone using the link from @jonnor

[thiswillbeyourgithub]

I went as far as I can afford in my random walk today and for the weeks to come. I updated my fork to mention everything to get started where I left off in the README : https://github.com/thiswillbeyourgithub/micropython

Current status

All current files are in AUDIO_TEST. There, I have a working script to play wav files on the watch (you can test it with the recording.wav file that I found somewhere online), and my plan is to modify the script that is supposed to record until it finally works. Both scripts are initially derived from that repo.

The latest issue is: the created file is empty of audio: it produces a file called mic.wav but I couldn't play it even on my computer and the hexdump showed it was actually full of 0000s. I won't have time to test some more for a bit so please join in!

To help: Take a look at the folder AUDIO_TEST, and tell me why the recordings are empty.

Here are links that can be relevant by increasing order of importance:
https://github.com/Xinyuan-LilyGO/TTGO_TWatch_Library/blob/master/docs/watch_2020_v3.md
https://github.com/Xinyuan-LilyGO/TTGO_TWatch_Library/
https://docs.micropython.org/en/latest/library/machine.I2S.html#machine-i2s
https://github.com/uraich/twatch2020_firmware/blob/6258eee0021351521da70d63a00d063e7a0acde7/hardware/sound/play-mono-wav-uasyncio.py
OPHoperHPO/lilygo-ttgo-twatch-2020-micropython#5
https://github.com/miketeachman/micropython-esp32-i2s-examples
Was updated to this repo:
https://github.com/miketeachman/micropython-i2s-examples
https://github.com/lunokjod/watch/blob/b71340b486b68d7682a9021e454779557d7a291c/src/app/Mic.cpp#L60-L82
This apparently is a working implementation in C so should contain all the answers?

See below for the steps I had to take to make the build.

I'm especially eager to hear about @jonnor and @DonvdH when and if you get a chance of course. As I reminder I know next to nothing about C and am just a lost medical student wanting this project too bad :) so there's a real chance that a 15 minute look from you can save me weeks of hair pulling! Thanks!