Any possiblity to reduce the SPI PSRam used up

[vishalvisd]

Thanks for helping with the codebase and I really liked the implementation!

I need to put a text overlay on the video and for that purpose, I was following CameraWebServer, where this statement: dl_matrix3du_t *image_matrix = dl_matrix3du_alloc(1, fb->width, fb->height, 3); needs a good amount of memory about 2.3-2.5 MB, but it fails to allocate memory needed, since about 1.3 MB remains after camera config. Please see below output of print statement:-
`

Before camera config ...Internal Total heap 285952, internal Free Heap 175184, SPIRam Total heap   4194252, SPIRam Free Heap   4194252
...
cam_err = esp_camera_init(&config);
...
After  camera config ...Internal Total heap 285300, internal Free Heap 117048, SPIRam Total heap   4194204, SPIRam Free Heap   1314204

Is there any was to reduce the amount of SPI RAM used up during esp_camera_init?

[jameszah]

Hmmm ... good question.

I've thought about the "text-over-image" issue. I think the way that code works is to convert the jpg to an rgb, then write the text onto the image, and then convert it back to a jpg. That process needs loads of memory and cpu to do all of that work -- which would make it non-viable to write something on every frame like a timestamp or watermark.

Normally the camera hardware does the first convert to jpeg, and then your pc or phone does the convert back to rgb for display. I went through a phase of converting every jpeg to rgb to look for jpeg errors, but it was very slow on esp32.

So I've thought of taking a little frame like a qvga, writing some text on that, then blowing it up of hd, and inserting it into the stream of hd frames. Or maybe taking a hd, squueze it down to qvga (losing 90% of detail), writing the text,, and turning it back to hd. So you could have a title page within the video with the time and trademark etc. without needing the memory and cpu to expand, write time, and shrink every frame back to jpg.

I've been meaning to work on that.

To your direct question - reducing memory from camera_init(). Not really. It is set up to receive the largest available jpeg from the camera, and 3 copies of it. One copy moving from camera to psram, one copy moving from psram to sd, and the third to use for psram to wifi if the streaming it working faster than the sd writer. You could drop the streaming, or change the buffer allocation for just "vga buffer size" instead of "uxga buffer size", and change the buffer size for very high quality to lower quality (i think framesizeconfig, qualityconfig variables which are more aggressive than framesize and quality) ... but it would run very slowly.

So I'd say write text a a few frames at the beginning/end, or like the CameraWebServer example you would have to restrict yourself to a much smaller frame size.

Maybe you could interleve a text frame every 10-50 frames. ??? Or I suppose if you are doing a timelapse you would have cpu between frames, but still not much memory.

So no option to convert a hd jpg to rgb inside the 4mb psram.

Maybe I'll give it a try at my qvga solution. 😄

[vishalvisd]

Thanks for your suggestion, however, I am bit consfused with your fourth paragraph that says to have a title page on the video. Did you mean that there would a quick overlay at the beginning of the video and then the video will be having no overlay...

However, I was wondering if there can a way by which we can take only a small portion of the fb-buf and put overlay on it and then replace that portion of fb-buf with the overlayed portion. I am expecting that this approach will be quite fast and relatively very little memory use

Something like this:-
dl_matrix3du_t *image_matrix = dl_matrix3du_alloc(1, x(width), y(height), 3);

fmt2rgb888(fb->buf, length***, fb->format, image_matrix->item); *** don't really know how to find the length correctly but guess there should be a way

rgb_print(image_matrix, 0x000000FF, "test"); // rgb_print function taken from CameraWebServer

fmt2jpg(image_matrix->item, x*y*3, x, y, PIXFORMAT_RGB888, 90, &_jpg_buf, &_jpg_buf_len);

// THEN, replacing the portion of fb->buf with _jpg_buf

I am more from application development so I may sound novice, but please let me know your suggestion....

[jameszah]

Just working on this ...

You said this "SPIRam Free Heap 1314204" earlier, which is what I recall, but now I'm getting "SPIRam Free Heap 3006143" before I have done anything. I seem to have discovered 1.7 mb of memory without doing anything ????

What versions of Arduino IDE and esp32-arduino are you using.
I have 1.8.19 and 2.0.2.
They may have improved the memory allocation ????

[jameszah]

Using a VGA frame 640x480 or 921600 bytes for the rgb888, it takes about a second to covert it to rgb and write some text, then convert it back to jpg.

So that is 1 frame per second, on a small frame.

I will see if I can put a small qvga frame into a stream of hd frames ....

The jpg is huffman coding so you have to step through every bit to unravel which block is where in the file, and if you modify that block (with text), then the size of the huffman coding will change, so you cannot drop it back into the old file or memory buffer ... you have to start over.

14:44:53.997 -> rgb888_buf  3F990208
14:44:54.372 -> to rgb 372 ms
14:44:54.372 -> rgb888 worked 1
14:44:54.888 -> to jpg 543 ms
14:44:54.888 -> fmt2rgb888 worked 1

[vishalvisd]

Just working on this ...

You said this "SPIRam Free Heap 1314204" earlier, which is what I recall, but now I'm getting "SPIRam Free Heap 3006143" before I have done anything. I seem to have discovered 1.7 mb of memory without doing anything ????

What versions of Arduino IDE and esp32-arduino are you using. I have 1.8.19 and 2.0.2. They may have improved the memory allocation ????

Sorry for the late reply as I was travelling...
I too using the same version of Arduino IDE: 1.8.19 and esp32-arduino 2.0.2. The issue of memory with which I started this question is resolved. The issue that came up later was the slow processing.