Music synthesis and spectrum visualization on RP2040 with CMSIS-Stream, CMSIS-DSP and Arm-2D .
There is now a related blog post on Hackster.
The demo is made of 2 CMSIS-Stream graphs.
- Core 0 executes an Audio graph
- Core 1 executes an LCD graph
The Audio graph synthesizes the sound and sends it to the PWM audio driver and Core 1.
The LCD graph reads the audio data from Core 0 and generates content and position of some layers:
- The spectrum layer
- The amplitude layer
- The CMSIS logo layer
Arm-2D is used as output node of the LCD graph to compose the layers with blending and create the final scene.
PicoMusicVideoShort.mov
With a serial console connected though USB you can control the digital gain of the Audio graph:
+or=key to increase the digital gain-key to decrease the digital gainq(quiet) to set the gain to0.0
The console also displays the name of the song being played. If you have configured several songs in the demo, you can move to the next song with n key.
You can also use the buttons below the screen:
To add new songs, you can modify the arrays defined in Songs/songs.c and add a reference to your new song:
static const uint8_t* _allSongs[];
static const char* _names[];The songs must be converted from a MIDI into a C array file using the miditones tools by Len Shustek.
You can look at the Songs folder to see some converted songs.
The audio graphs synthesizes the music and sends the audio samples to the PWM audio driver and to core 1.
The tool miditones from the Playtune_synth_project can be used to convert MIDI files and is configured to use 6 channels by default. That's why the Audio graph only has 6 channels. But it is configurable in the Python scripts and the graph can be modified to have as many channels as needed.
When a channel is not used by a song it has a low overhead (but memory copies are still occurring because buffers can be shared between different "FIFOs" for memory optimization purposes).
A channel is made of a very simple waveform synthesizer followed by a node that changes the envelope of the waveform.
The channels are mixed with saturating additions.
The sequencer nodes generates commands and control the waveform and envelope nodes.
The audio node sends the audio samples to the PWM driver.
The core1 node sends data to core 1 in a non blocking way : if core 1 is not fast enough it means some audio packets will be missed.
The LCD graph processes the audio coming from core 0 to display the amplitude and spectrum of the synthesized music. It is waiting for audio packets and processing will be halted until a new audio packet is received.
The received audio samples are decimated by a factor of 16 because we only have 240 samples to display and also to avoid having a too high refresh rate of the LCD.
The sound signal is filtered and decimated using the CMSIS-DSP function arm_fir_decimate_q15.
The filter coefficients are defined in App/Sources/main.cpp : coefficients of the filter are in the coefs array in q15_t format. Current filter is a simple comb filter.
A FFT of the decimated signal is computed. The FFT is windowed using a Hanning window. The window is computed in App/Sources/main.cpp using the CMSIS-DSP function arm_hanning_f32 and arm_float_to_q15.
A FFT amplitude with scaling is computed. The scaling has been chosen in an arbitrary way so that the FFT spectrum is well visible on the display.
A widget nodes takes a layer and some samples and draws something into the layer. Two widgets are defined:
- A spectrum widget that displays a spectrum in a layer from the FFT amplitudes at the input of this widget
- An amplitude widget that displays the sound waveform in a layer
Due to the high decimation factor, don't expect to see high frequencies on the spectrum widget.
The layers are then composed with blending by the Arm-2D node to render the final scene on the LCD.
The color coding is reflecting what I used on my hardware and unfortunately I did not have enough wires for the grounds. So the ground connections are not displayed in black.
And the 3V connection is not always represented in red.
Here is a photo of the final experiment:
The transistor is a NPN 2222A. But any other NPN transistor would work. It just need to have a higher bandwidth than audio bandwidth.
Speaker has a resistance of 4 Ω. You can also put a resistance in series with the speaker is you want to decrease the amount of current drawn from the USB.
The pin number used is coming from
pico-extras/blob/master/src/rp2_common/pico_audio_pwm/include/pico/audio_pwm.h
and can be changed by redefining some macros:
#ifndef PICO_AUDIO_PWM_L_PIN
#define PICO_AUDIO_PWM_L_PIN 0
#endif
#ifndef PICO_AUDIO_PWM_R_PIN
#define PICO_AUDIO_PWM_R_PIN 1
#endif
#ifndef PICO_AUDIO_PWM_MONO_PIN
#define PICO_AUDIO_PWM_MONO_PIN PICO_AUDIO_PWM_L_PIN
#endifThe LCD is https://www.waveshare.com/wiki/Pico-LCD-1.3
And we are using a https://www.waveshare.com/pico-quad-expander.htm to connect the LCD
The pins used comes from the Arm2D_RP2040 project and are defined in RP2040/st7789_lcd.h. They can be changed by redefining some macros:
#ifndef PIN_DIN
#define PIN_DIN 11
#endif
#ifndef PIN_CLK
#define PIN_CLK 10
#endif
#ifndef PIN_CS
#define PIN_CS 9
#endif
#ifndef PIN_DC
#define PIN_DC 8
#endif
#ifndef PIN_RESET
#define PIN_RESET 12
#endif
#ifndef PIN_BL
#define PIN_BL 13
#endif First, you need to be able to build the Pico examples and have the pico-extra since the demo is using audio drivers from pico-extra. Refer to the Pico documentation to know what to install.
PICO_SDK_PATH must be set.
PICO_EXTRAS_PATH must be set.
Once you can build some Pico examples, you can then build this project:
git clone --recurse-submodules https://github.com/christophe0606/PicoMusic.git
cd PicoMusicmkdir build.tmp
cd build.tmp
cmake -G "Unix Makefiles" ..
makeYou can also use the following options on the cmake command line if you don't want to use the git submodules and have already cloned the repositories:
-DARM2D=/ArmSoftware/Arm-2D \
-DCMSISDSP=/ArmSoftware/CMSIS-DSP \
-DCMSISSTREAM=/ArmSoftware/CMSIS-Stream \
-DCMSISCORE=/ArmSoftware/CMSIS_5/CMSIS/Core \
-DARM2D_RP2040=/ArmSoftware/Arm2D_RP2040/RP2040Where ArmSoftware is the path where you cloned the libraries.
For CMSIS-DSP and Arm-2D, use the develop branch.
The last project Arm2D_RP2040 is providing RP2040 specific optimizations for Arm-2D and is also providing a CMakeLists.txt to make it easier to build both Arm-2D + CMSIS-DSP in the Pico environment.
The MIDI part is coming from the Playtune_synth_project Copyright (C) 2016, Len Shustek
The Playtune_synth files have been strongly modified and refactored to be integrated in the CMSIS-Stream framework. They are clearly identified (always inside a Playtune_synth folder) and Copyright (C) 2016, Len Shustek.
Playtune_synth files are covered by their original MIT license.
The Arm2D_RP2040 project is using an example of PIO driver for LCD coming from the pico-examples. The example has been modified to be integrated in CMSIS-Stream.
It is Copyright (c) 2020 Raspberry Pi (Trading) Ltd and covered by a BSD-3 license.
The PWM node is based upon a Pico playground audio example using the drivers from pico-extra. The example has been modified to be integrated in CMSIS-Stream.
It is Copyright (c) 2020 Raspberry Pi (Trading) Ltd and covered by a BSD-3 license.
Other files are covered by an Apache license and have been developed by Arm.