Whisper v29@main for Raspberry PI

[nettlep]

Build of whisper for Raspberry Pi devices running Raspbian on armv7 or better hardware. This was built/tested on a rPI model 4.

Prerequisites

In order to run, you will need to install some dependencies and setup a few files. Do this in a terminal on your Raspberry Pi.

Your best bet is to clone the repo and drop the pre-built whisper binary into the root of that directory.

Then, from within that directory, do these things:

# First, install the repository for the Swift language (see https://github.com/futurejones/swift-arm64)
$ curl -s https://archive.swiftlang.xyz/install.sh | sudo bash

# Next install the dependencies
$ sudo apt install libpng-dev libncurses5-dev libaa1-dev libavcodec-dev libavformat-dev swiftlang

# Next, setup your config files
$ ln -s decks.json ~/.decks.json
$ ln -s whisper.conf ~/.whisper.conf

$ Finally, run a test
$ ./whisper -6 Sources/Steve/Steve/Reference/inter6.quick-random-movements.mp4

If all goes well, that last command should produce a text-based GUI showing you the playback of the video along with stats of the scanning process.

---

This discussion was created from the release Whisper v29@main for Raspberry PI.

[dlbroadfoot]

Thanks @nettlep !

I followed your instructions and got whisper running on a Raspberry 3B (32bit Raspberry Pi OS Lite). The sample video worked great!

I've enabled the NoIR camera on the Pi but when I run ./whisper -6 I get a "no media provider" error.

I've checked the camera works as I can capture an image using raspistill.

Are there any other settings or command line switches that need to be enabled to get whisper working with the Pi camera?

ThankS

[nettlep]

There was a build issue preventing Whisper from including MMAL support for video capture. I've resolved this issue and produced another release:

https://github.com/nettlep/magic/releases/tag/whisper-v29%40main%2Bvideo-rpi

[dlbroadfoot]

Thanks @nettlep . I've confirmed that works with the Pi cam

[nettlep]

That's great news and thanks for letting me know. Not sure, but you might be the first person besides myself to run Whisper on a PI. Congrats?

Be aware that the PI cameras, with their fixed-focus, are finicky. In order for the code to be large enough in frame to be read, it has to be relatively close (~14" in my experience.) But when the focus is adjusted to that range, then things get out of focus quite easily as they drift from that ideal distance.

How to test without a deck

Get a code from Cardbars. Print or store the image(s) on your phone. Then use the image instead of a deck. I keep a few codes on my phone for just this purpose. Also, make sure the code you print/store is the same type of code that your whisper is setup to use.

Other helpful tips working with rPI cameras

If you have an iOS device, load up Abra (it's on the App Store now, search for Abra Magic). If your rPI and iOS devices are on the same network, Abra will discover and connect to the rPI server automatically. When this happens, the rPI's video feed will instantly show up in Abra's viewport. The viewport is low-resolution, so tap the (+) icon in the viewport to see a 1:1 pixel (zoomed-in) view of the center of the viewport. Using this close-up view, you now have a detailed view to fine-tune your camera's focus. Once setup, tap the (-) icon to return to normal.

[Chtwin]

Everything worked for me but I can't get past 80%. Do you have any conditions that would improve the percentage with the phone such as the brightness of the phone or the ambient lighting?

Here is what I get with my tests:

[Chtwin]

I could improve the detection by changing the focus of the raspberry pi camera. I found an STL file that will allow you to print a small gear to change the focus:
https://cults3d.com/fr/mod%C3%A8le-3d/outil/key-for-adjusting-lens-of-raspicam-v2-1

Result after changing the focus:

[nettlep]

In your screenshot, you will see a (+) icon in the bottom-right of the viewport.

That is to help focus - it zooms in so you can see your true focus.

[Chtwin]

Yes thank you I had used that in addition to the manual setting focus and it works pretty well.

[Chtwin]

When I ran whisper on my raspberry pi 4 I got this error message:

X Error of failed request: BadName (named color or font does not exist) Major opcode of failed request: 45 (X_OpenFont) Serial number of failed request: 7 Current serial number in output stream: 8

Do you have any idea where this could be coming from?

Thanks

[Chtwin]

Yes from the terminal on Raspbian Bullseye.

[nettlep]

That's a new one by me. Maybe somebody else can offer advice?

[Chtwin]

It didn't work for me because I was trying to run whisper from the linux terminal, but in ssh everything works fine.

[Chtwin]

When I run whisper with the video I have no problem but when I run to use my camera I have a black screen and directly the performance info.

Did I do something wrong?

By the way, thank you very much for all your help and your wonderful project!

[Chtwin]

I solved my problem, so I'll put the solution here for those of you who have the same problem (it was a bit lame).

I had run the command libcamera-still -o test.jpg to see if my camera was working and it worked fine. But on whisper it didn't work.

To fix this, I had to activate the legacy cam by doing:
sudo raspi-config

Then you have to reboot the raspberry pi and everything works!

[Chtwin]

Hi @nettlep,

Two months ago I used an RPI 4 to run Whisper and everything worked. I wanted to miniaturize it and it took me 2 months to buy a Pi zero 2 W(I don't know if it's a documentation problem but on the README it says Pi Zero W and not 2 W, I couldn't get it to run on a W). On the Pi Zero 2 W everything runs but the application blocks on "Waiting for confidence".

I tested again with my PI 4 and everything works fine. (I dropped the cam so the screen is black)

With the camera of the computer or the iphone it works. But on my Pi 3B I have the same problem as the Pi Zero 2 W. On the other hand, when I use Steve's videos everything works fine on all devices.

I just use my phone to display the code with different level of brightness and even angle. I really don't understand, especially since the focus is well adjusted and everything works on the Pi 4. Do you have an explanation or a way to unblock me?

Thanks a lot!

[nettlep]

Getting stuck waiting for confidence can be frustrating until you understand what it really means in order to troubleshoot it. This is something I ran into periodically, so bear with me as I cover this in some detail for others that may run across this.

As you probably already know, the software doesn't just report the order of the deck from the first scan it gets. It has to build up a history of valid reads so it can analyze that history to build a confident result. How much history is defined by analysis.MinHistoryEntries in the whisper.conf file. The default is 15, which is to say that if you're capturing at 30Hz, it will take a minimum of 0.5 seconds to get that first result. The key here is that a history entry requires valid reads of the deck. Not every frame will produce a read (motion blur, something obscures the deck, etc.) You also have a deadline to contend with, which is defined by analysis.MaxHistoryAgeMS. This value says that any history entry older than [default = 4 seconds] will be removed from the history.

I'll assume defaults going forward.

Okay, so that's the backgrounder to help you in diagnosing the problem. The fact that it says "Waiting for confidence" means that it is indeed scanning the deck. If it weren't getting any reads, then it would report "not found" or "move closer" or some other suggestion. So this means it is having trouble collecting 15 valid reads in a 4-second period, OR the decks in the history do not resolve to a single deck order within a minimum confidence.

Whenever I see this, it is one of these:

1. You're not using the correct deck format (ex: you have it set for inter6 when it should be something else.)
2. Too much motion blur. This is a challenge when reading IR decks because those are low-light conditions.
3. The device is dropping frames. Make sure you're running a release build and you don't have a bunch of debug options enabled to slow it down. To highlight this, if you're dropping frames and only getting 5 FPS, then you only have 20 frames to work with within the deadline. Unless every one of those frames is a valid read, it's easy to miss the 15-history-entries requirement. See the section below on one way to resolve this without losing quality or confidence.
4. It's a focus problem: Are you using the "+" icon to set the focus correctly? What appears in focus may not actually be in focus until you enable that focus zoom.
5. Deck detection (specifically tracing the marks.) In your last screenshot where you are using a video from Steve, you can see the rectangle around the deck and the lines trough the bit marks. The left/right edges of that rectangle follow the contour of the wide landmarks. The trace marks process is what follows that contour and is the most finicky part of the scanning process. Sometimes it doesn't trace far enough (the marks on the deck are not consistent so it is fooled into thinking it reached the edge of the deck) or traces too far (the background fools it into thinking the marks continue.) Make sure your deck has good marks and find the correct position (with practice) and you should largely avoid this problem.

Dealing with dropped frames

@Chtwin's problem is # 3. Look at the stats at the bottom of your first picture:

FPS: 2 (413.8ms) : 28.6ms

In English, this means: "The time from the start of one frame to the start of the next frame was 413 ms, which works out to about 2 FPS. Of those 413 ms, only 28 ms was spent actually locating/scanning/decoding/analyzing the deck."

2 FPS is not enough to get past the dreaded "waiting for confidence". You might think that you need to lower analysis.MinHistoryEntries, but this reduces your overall confidence. Alternatively, you could increase analysis.MaxHistoryAgeMS but it may take a while to get that first truly confident result.

There are better solutions:

Try setting debug.ViewportDebugView to false

If 28ms was spent doing the actual work, then where are those other 385 ms going? That's all the overhead of getting that frame from the camera, copying the memory, creating the debug viewport, doing any image processing (rotating the image, etc.)

The most expensive operation is creating debug viewport. That's taking a 2 MB (monochrome 1080p) frame and creating a new 8MB (32-bit color 1080p) frame from it. Reading/writing that much memory is expensive, especially for realtime on a Pi Zero.

By setting debug.ViewportDebugView to false, you opt out of the fancy debug information (so you won't see those lines drawn on the viewport) but you skip that expensive operation entirely.

This also speeds up the time to send the viewport to the clients, since it will package up a viewport that is 1/4th the memory.

This may be all you need to do, but there's more...

Try increasing capture.ViewportFrequencyFrames

Setting this value to n will only generate a viewport and send it to clients every n frames. The default is 1, which means we do it every frame. Increasing this value to something like 10 will make the viewport choppy (only about 3 FPS) but we can still see what's going on and we spend 1/10th the time packaging up that viewport to send to clients.

This is not a magic bullet. Let's say we use a larger value of 30. If that viewport burns 300ms (about 1/3rd of a second) then every 30 frames we are going to burn 300ms. During that time, approximately 9 frames would get missed (dropped.)

Finally, for those that prefer excruciating detail: a value of 30 (and a camera recording at 30Hz) does not mean we do this once per second. It's actually longer because those frames that were dropped while we were busy packaging up the viewport are not counted. So we'll capture 30 frames, then spend some amount of time on the viewport, and then capture 30 more frames before doing it again.

[nettlep]

Also, in reference to not being able to get it to run on the Zero W (not the 2 variant) - did you build using the armv6 argument? (just add armv6 to your args to the build script.) That flag was added specifically for the Zero W. The Zero 2 W has an armv7 processor, which is probably why your builds are working on it.

Disclaimer: It has been years since I've tried building/running this on a Zero W, so if it doesn't work I may have broken something without realizing it.

[Chtwin]

Thank you for this very very very complete answer. I will test this tomorrow and keep you posted.

To answer to the questions:

1 - Yes I do use the correct inter6 marking on the whisper, abra and deck side.

2 - The blur is ok.

3 - I will check this

4 - Yes I use this to set the focus

5 - I will change the settings to see if I can improve this

Thank you for all your hard work.

[Chtwin]

I put debug.ViewportDebugView to false.

I increased capture.ViewportFrequencyFrames to 10.

And finally I incresead analysis.MaxHistoryAgeMS to 8000 too.

It works perfectly.

Thank you so much for all your help it's really amazing.