Build problem with OpenCV IDF component (IEC-85)

[Ncouch64]

Answers checklist.

• I have read the documentation of the component in question and the issue is not addressed there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report

Hello! I am reaching out here with a build problem with the OpenCV component mentioned here in this PR: #94. At first I was thinking it was a problem with my idf but I have been able to get projects build with various other components, as soon as I add the OpenCV component as a dependency I get the build failure. This may be something super simple that I am overlooking but after all my searching around I have not been able to find a solution. Thank you for your time!

Edit: Also should probably mention that this is while doing just one of the examples "features2d" that you can find mentioned on the idf component registry here: https://components.espressif.com/components/espressif/opencv.

The build failure I am getting when using the component:
-- Found Git: C:/Users/na258588/.espressif/tools/idf-git/2.39.2/cmd/git.exe (found version "2.39.2.windows.1")
-- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- The ASM compiler identification is GNU
-- Found assembler: C:/Users/na258588/.espressif/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/xtensa-esp32s3-elf-gcc.exe
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Users/na258588/.espressif/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/xtensa-esp32s3-elf-gcc.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Users/na258588/.espressif/tools/xtensa-esp32s3-elf/esp-12.2.0_20230208/xtensa-esp32s3-elf/bin/xtensa-esp32s3-elf-g++.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- git rev-parse returned 'fatal: not a git repository (or any of the parent directories): .git'
-- Could not use 'git describe' to determine PROJECT_VER.
-- Building ESP-IDF components for target esp32s3
Processing 3 dependencies:
[1/3] espressif/esp32-camera (2.0.6)
[2/3] espressif/opencv (4.7.0~3)
[3/3] idf (5.1.2)
-- Found Git: C:/Users/na258588/.espressif/tools/idf-git/2.39.2/cmd/git.exe (found version "2.39.2.windows.1")
CMake Error at C:/Users/na258588/esp/esp-idf/tools/cmake/component.cmake:224 (message):
CMake Warning (dev) at build_properties.temp.cmake:8:

Syntax Warning in cmake code at column 47

Argument not separated from preceding token by whitespace.

Call Stack (most recent call first):

C:/Users/na258588/esp/esp-idf/tools/cmake/scripts/component_get_requirements.cmake:3 (include)

This warning is for project developers. Use -Wno-dev to suppress it.

fatal: not a git repository (or any of the parent directories): .git

CMake Error at
C:/Users/na258588/.espressif/tools/cmake/3.24.0/share/cmake-3.24/Modules/ExternalProject.cmake:1253
(define_property):

define_property command is not scriptable

Call Stack (most recent call first):

C:/Users/na258588/esp/esp-idf/tools/cmake/kconfig.cmake:1 (include)
C:/Users/na258588/esp/esp-idf/tools/cmake/idf.cmake:43 (include)
C:/Users/na258588/esp/esp-idf/tools/cmake/project.cmake:29 (include)
E:/work/Project/features2d/CMakeLists.txt:5 (include)
C:/Users/na258588/esp/esp-idf/tools/cmake/scripts/component_get_requirements.cmake:106 (include)
C:/Users/na258588/esp/esp-idf/tools/cmake/scripts/component_get_requirements.cmake:124 (__component_get_requirements)

Call Stack (most recent call first):
C:/Users/na258588/esp/esp-idf/tools/cmake/build.cmake:574 (__component_get_requirements)
C:/Users/na258588/esp/esp-idf/tools/cmake/project.cmake:547 (idf_build_process)
CMakeLists.txt:6 (project)

-- Configuring incomplete, errors occurred!
See also "E:/work/Project/features2d/build/CMakeFiles/CMakeOutput.log".

The terminal process "C:\Windows\System32\cmd.exe /d /c cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DESP_PLATFORM=1 -B e:\work\Project\features2d\build -S e:\work\Project\features2d" terminated with exit code: 1.

[dmitry1945]

@Ncouch64 the new version of IDF opencv component released. Please take a look.

[Ncouch64]

Ok, I was able to get the example features2d project to build with 4.7.0~4 - YAY! Now I am running into a separate problem. So creating the project from the example using idf.py create-project-from-example "espressif/opencv^4.7.0~4:features2d" it is unable to find any of these source files -
#include <esp_log.h>
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "esp_system.h"
#include "esp_timer.h"
#include "esp_camera.h"
and for whatever reason when it is flashed to my ai-thinker board I get this out

I (2717) gpio: GPIO[5]| InputEn: 1| OutputEn: 0| OpenDrain: 0| Pullup: 1| Pulldown: 0| Intr:2
I (2727) cam_hal: cam init ok
I (2727) sccb: pin_sda 18 pin_scl 23
I (2727) sccb: sccb_i2c_port=1
E (2747) camera: Camera probe failed with error 0x105(ESP_ERR_NOT_FOUND)
E (2747) CAM: Camera Init Failed
I (2747) main_task: Returned from app_main()
None

I am assuming that I am doing something wrong here. It has been a good few months since I was fully dug in to ESP work so there is likely some processes that I am overlooking. I thank you @igrr & @dmitry1945 for the continued assistance.

[dmitry1945]

@Ncouch64 let me check...

[dmitry1945]

@Ncouch64 I've now created a project from idf.py create-project-from-example "espressif/opencv^4.7.0~4:features2d"
and it's compiled without problems....
Could you please update to the latest IDF version, if it's possible? Or do you have one already?

[Ncouch64]

Hmm, now this is odd. I am pretty sure I am on the latest IDF version, I will check now and make sure. Ill then try and create another project from the example. Is there anything I should be doing outside of creating the project and building it?

Edit: Currently on 5.1.2 using VSCode.
Edit 2: Ok, did a fresh install of IDF just as a sanity check. Using the above command created a features2d example project, set my target for my ESP32 device and build the project. It completes the build without errors, but none of those header files are found and when I flash the build to my device I get the output mentioned above.

[dmitry1945]

@Ncouch64 just to be sure, did you run git submodule update --init --recursive after switch to 5.2 (I always forger to do this...)?
And, did you select the AI-Thinker in menuconfig?

And, see you tomorrow.

[Ncouch64]

You are totally right these are things I should look into tomorrow. Thanks for the continued assistance. I will be going back at the ESP work tomorrow morning so I will test and post my findings here.

[Ncouch64]

@Ncouch64 just to be sure, did you run git submodule update --init --recursive after switch to 5.2 (I always forger to do this...)? And, did you select the AI-Thinker in menuconfig?

And, see you tomorrow.

Ok, updated to 5.2 (was previously 5.1.2), running the submodule update in the features2d project doesn't work for me as it says it is not a git repository - this is after creating a brand new project from the features2d example. After attempting to run the example again I finally got an output of detected features! Weirdly enough it is telling me it still cannot locate the source files. For example its saying it cannot open esp_err.h source file (dependency of esp_camera.h). I will try to do some of my own testing using the CV component now that I got an output to see if I can get a simple project to work.

Edit: I also want to mention here, I am not sure if its just my IDF or what but if I have the CV component added into the project it takes anywhere from 10-20 minutes just to open the sdk config and to build the project a similar amount of time. Is this expected or have I done something wrong? Also fixed the above paragraph, something happened in the edit that messed it up.

[Ncouch64]

@dmitry1945 Have you tested the FAST algorithm running this component? I am having an issue where features are not being detected.

[dmitry1945]

@Ncouch64 no, I'be tested with some default one... What kind of problem do you have with a FAST?

[Ncouch64]

@Ncouch64 no, I'be tested with some default one... What kind of problem do you have with a FAST?

Its really interesting actually. So if I use for example goodFeaturesToTrack() it is able to detect features in images. If I use Fast() it is never able to detect features in the image. I am not quite sure what the problem is.

[Ncouch64]

I also did want to throw out another question, would using the findEssentialMat() function and related functions be too intense for an esp chip to handle? I am wondering if there are too many operations to properly run on ESP. Currently when trying to use this function I get this during build: region `dram0_0_seg' overflowed by 465872 bytes.

[dmitry1945]

@Ncouch64 it's possible to calculate the findEssentialMat(), but for that you will need to make some changes in opencv. And anyway, it will not be very fast. It will be possible with new Espressif chips.
If you want, you can try with esp32. For that, you will need to change double type to float, and remove some big arrays to the constant memory.

[Ncouch64]

@Ncouch64 it's possible to calculate the findEssentialMat(), but for that you will need to make some changes in opencv. And anyway, it will not be very fast. It will be possible with new Espressif chips. If you want, you can try with esp32. For that, you will need to change double type to float, and remove some big arrays to the constant memory.

Do you feel like an S3 would have a better chance at this? I have one of those on the way.

Also thank you for the continued assistance with this component.

[Ncouch64]

So if I could throw out another question as I am trying to understand use of CV on esp, when I see this during my build process region 'dram0_0_seg' overflowed by 354872 bytes what exactly does this mean? So for reference I have only seen this when trying to build using the findEssentialMat() function. Is this telling me that the program is attempting to use more memory than is available? Is this what you meant by needing to changing the way the CV component functions? In your honest opinion, using an esp32s3, is this even feasible to attempt to do? Is it just currently too much for the device to handle?

[dmitry1945]

@Ncouch64 the findEssentialMat() using inside some tables (arrays), that should be located in constant memory (like const uint8_t sometable[];) and they will be located in the flash, but they are located in the RAM ((just uint8_t sometable[];)).

[dmitry1945]

@Ncouch64 please look to the repo from Peter Marcisovsky (I'm not sure if it's open, let me know if not):
https://github.com/peter-marcisovsky/opencv/tree/feature/slam_float

He did some changes to the opencv for visual odometry.

[Ncouch64]

@Ncouch64 please look to the repo from Peter Marcisovsky (I'm not sure if it's open, let me know if not): https://github.com/peter-marcisovsky/opencv/tree/feature/slam_float

He did some changes to the opencv for visual odometry.

The Github seems to be open, I will look through this resource thank you!

[Ncouch64]

@Ncouch64 please look to the repo from Peter Marcisovsky (I'm not sure if it's open, let me know if not): https://github.com/peter-marcisovsky/opencv/tree/feature/slam_float

He did some changes to the opencv for visual odometry.

So if I want to use this in an IDF project I know I need to add it as a submodule. Do I need to add something to the Cmakelists? I am not entirely sure how to make sure it builds using this.

[dmitry1945]

@Ncouch64 just replace the opencv and opencv_contrib with copy from Peter repo. It should work (I'm not sure). Or, just compare opncv code in opencv component folder with code from Peter, and make some changes by hands.
You don't need to change the CMakeLists.txt file.

[Ncouch64]

@Ncouch64 just replace the opencv and opencv_contrib with copy from Peter repo. It should work (I'm not sure). Or, just compare opncv code in opencv component folder with code from Peter, and make some changes by hands. You don't need to change the CMakeLists.txt file.

Understood, thanks for this info. I will be working on testing this out!

[Ncouch64]

Just tested it out and it seems when I switch the files over to the ones from Peter's repo and then build the project they are replaced again with the files from the component. Is there some way to make it so that this will not be the case? I am assuming on build it is checking for component differences from the server and replacing them, but I do not know this for sure.

[Ncouch64]

@dmitry1945 I have encountered another problem and I am going to post it here as I am not seeing this anywhere else and am not sure if this has something to do with the component or not. So I am trying to capture an image, perform a simple canny detection on the image and then save it back to the SDCard. I have done this many times before using OpenCV and ESP. However using this component when I initialize the SDCard the camera fails to capture a frame - see output.

--- WARNING: GDB cannot open serial ports accessed as COMx
--- Using \.\COM5 instead...
--- esp-idf-monitor 1.4.0 on \.\COM5 115200 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
I (98) esp_image: segment 0: paddr=00010020 vaddr=3c13002ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x15 (USB_UART_CHIP_RESET),boot:0x8 (SPI_FAST_FLASH_BOOT)
Saved PC:0x40379faa
0x40379faa: esp_cpu_wait_for_intr at C:/Users/na258588/esp/v5.2/esp-idf/components/esp_hw_support/cpu.c:145

SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce3820,len:0x1918
load:0x403c9700,len:0x4
load:0x403c9704,len:0xe5c
load:0x403cc700,len:0x3024
entry 0x403c993c
I (26) boot: ESP-IDF v5.2-beta2-190-g4ecc978bd6 2nd stage bootloader
I (27) boot: compile time Feb 8 2024 09:51:33
I (27) boot: Multicore bootloader
I (31) boot: chip revision: v0.1
I (35) qio_mode: Enabling default flash chip QIO
I (41) boot.esp32s3: Boot SPI Speed : 80MHz
I (45) boot.esp32s3: SPI Mode : QIO
I (50) boot.esp32s3: SPI Flash Size : 8MB
I (55) boot: Enabling RNG early entropy source...
I (60) boot: Partition Table:
I (64) boot: ## Label Usage Type ST Offset Length
I (71) boot: 0 nvs WiFi data 01 02 00009000 00006000
I (79) boot: 1 phy_init RF data 01 01 0000f000 00001000
I (86) boot: 2 factory factory app 00 00 00010000 00300000
I (94) boot: End of partition table
I (98) esp_image: segment 0: paddr=00010020 vaddr=3c130020 size=82da0h (535968) map
I (187) esp_image: segment 1: paddr=00092dc8 vaddr=3fc96e00 size=044b0h ( 17584) load
I (191) esp_image: segment 2: paddr=00097280 vaddr=40374000 size=08d98h ( 36248) load
I (200) esp_image: segment 3: paddr=000a0020 vaddr=42000020 size=12d6c4h (1234628) map
I (389) esp_image: segment 4: paddr=001cd6ec vaddr=4037cd98 size=09fc8h ( 40904) load
I (405) boot: Loaded app from partition at offset 0x10000
I (405) boot: Disabling RNG early entropy source...
I (417) cpu_start: Multicore app
I (417) octal_psram: vendor id : 0x0d (AP)
I (417) octal_psram: dev id : 0x02 (generation 3)
I (420) octal_psram: density : 0x03 (64 Mbit)
I (426) octal_psram: good-die : 0x01 (Pass)
I (431) octal_psram: Latency : 0x01 (Fixed)
I (436) octal_psram: VCC : 0x01 (3V)
I (441) octal_psram: SRF : 0x01 (Fast Refresh)
I (447) octal_psram: BurstType : 0x01 (Hybrid Wrap)
I (453) octal_psram: BurstLen : 0x01 (32 Byte)
I (458) octal_psram: Readlatency : 0x02 (10 cycles@Fixed)
I (465) octal_psram: DriveStrength: 0x00 (1/1)
I (470) MSPI Timing: PSRAM timing tuning index: 4
I (475) esp_psram: Found 8MB PSRAM device
I (480) esp_psram: Speed: 80MHz
I (929) esp_psram: SPI SRAM memory test OK
I (939) cpu_start: Pro cpu start user code
I (939) cpu_start: cpu freq: 240000000 Hz
I (939) cpu_start: Application information:
I (942) cpu_start: Project name: revised_cv
I (947) cpu_start: App version: 1
I (952) cpu_start: Compile time: Feb 8 2024 09:49:48
I (958) cpu_start: ELF file SHA256: 2c56e1fa2...
I (963) cpu_start: ESP-IDF: v5.2-beta2-190-g4ecc978bd6
I (970) cpu_start: Min chip rev: v0.0
I (974) cpu_start: Max chip rev: v0.99
I (979) cpu_start: Chip rev: v0.1
I (984) heap_init: Initializing. RAM available for dynamic allocation:
I (991) heap_init: At 3FCB6978 len 00032D98 (203 KiB): RAM
I (997) heap_init: At 3FCE9710 len 00005724 (21 KiB): RAM
I (1003) heap_init: At 3FCF0000 len 00008000 (32 KiB): DRAM
I (1010) heap_init: At 600FE010 len 00001FD8 (7 KiB): RTCRAM
I (1016) esp_psram: Adding pool of 8192K of PSRAM memory to heap allocator
I (1024) spi_flash: detected chip: gd
I (1028) spi_flash: flash io: qio
W (1033) i2c: This driver is an old driver, please migrate your application code to adapt driver/i2c_master.h
I (1044) sleep: Configure to isolate all GPIO pins in sleep state
I (1050) sleep: Enable automatic switching of GPIO sleep configuration
I (1057) main_task: Started on CPU0
I (1067) esp_psram: Reserving pool of 32K of internal memory for DMA/internal allocations
I (1067) main_task: Calling app_main()
I (1077) s3 ll_cam: DMA Channel=4
I (1077) cam_hal: cam init ok
I (1077) sccb: pin_sda 40 pin_scl 39
I (1087) sccb: sccb_i2c_port=1
I (1097) camera: Detected camera at address=0x30
I (1097) camera: Detected OV2640 camera
I (1097) camera: Camera PID=0x26 VER=0x42 MIDL=0x7f MIDH=0xa2
I (1177) cam_hal: buffer_size: 16384, half_buffer_size: 1024, node_buffer_size: 1024, node_cnt: 16, total_cnt: 15
I (1177) cam_hal: Allocating 15360 Byte frame buffer in PSRAM
I (1187) cam_hal: cam config ok
I (1187) ov2640: Set PLL: clk_2x: 0, clk_div: 0, pclk_auto: 0, pclk_div: 8
Finished initializing the camera!
I (1267) SDCARD: Initializing MicroSD Card
I (1267) SDCARD: Using SPI Peripheral
I (1267) SDCARD: Mounting filesystem
I (1267) gpio: GPIO[21]| InputEn: 0| OutputEn: 1| OpenDrain: 0| Pullup: 0| Pulldown: 0| Intr:0
I (1277) sdspi_transaction: cmd=52, R1 response: command not supported
I (1327) sdspi_transaction: cmd=5, R1 response: command not supported
I (1357) SDCARD: Filesystem mounted
Name: SM32G
Type: SDHC/SDXC
Speed: 20.00 MHz (limit: 20.00 MHz)
Size: 30436MB
CSD: ver=2, sector_size=512, capacity=62333952 read_bl_len=9
SSR: bus_width=1
Finished initializing the MicroSD card!
About to start the task!
I (1377) main_task: Returned from app_main()
W (5387) cam_hal: Failed to get the frame on time!
Camera capture has failed!
W (9397) cam_hal: Failed to get the frame on time!
Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled.

Core 1 register dump:
PC : 0x42009da8 PS : 0x00060e30 A0 : 0x8037dff8 A1 : 0x3fcd2a30
0x42009da8: videoCapture(void*) at E:/work/Projects/revised_cv/main/app_main.cpp:301

A2 : 0x00000000 A3 : 0x00000000 A4 : 0x00000000 A5 : 0x00060023
A6 : 0x00000001 A7 : 0x00000000 A8 : 0x82009da6 A9 : 0x3fcd2a10
A10 : 0x00000000 A11 : 0x00000003 A12 : 0x00060023 A13 : 0x3fcd2d5c
A14 : 0x3fcbc4a0 A15 : 0x00000001 SAR : 0x00000004 EXCCAUSE: 0x0000001c
EXCVADDR: 0x0000000c LBEG : 0x400556d5 LEND : 0x400556e5 LCOUNT : 0xfffffffd
0x400556d5: strlen in ROM
0x400556e5: strlen in ROM

Backtrace: 0x42009da5:0x3fcd2a30 0x4037dff5:0x3fcd2bf0
0x42009da5: videoCapture(void*) at E:/work/Projects/revised_cv/main/app_main.cpp:298
0x4037dff5: vPortTaskWrapper at C:/Users/na258588/esp/v5.2/esp-idf/components/freertos/FreeRTOS-Kernel/portable/xtensa/port.c:134

If I comment out the function to initialize the sdcard I do not have this issue and the camera is able to capture the images. I am not sure what I am doing wrong in this situation. I am using the same code I was using when using my own built version of opencv on esp. Is this something that could be caused by this component? I am thinking maybe not but I dont understand why it wouldnt be working here now?

In addition to this I am wondering if it is possible to prevent the component files from being overwritten when attempting to build. If I replace something in the OpenCV files as we discussed above it is overwritten during the build process.

[dmitry1945]

Hi @Ncouch64,

If you have your opencv version, which is working, and the problem that it's overwritten, you can do next.
Download the archive of opencv compoent, and then in your project idf_component.yml file change

  espressif/opencv:
    version: '*'  

to the

  espressif/opencv:
    override_path: "path to your opencv component"
    version: "*"

then, you will be able to change the opencv and opencv_contrib folders.

If the problem inside the opencv, then it will solve the problem.

[Ncouch64]

Hi @Ncouch64,

If you have your opencv version, which is working, and the problem that it's overwritten, you can do next. Download the archive of opencv compoent, and then in your project idf_component.yml file change

  espressif/opencv:
    version: '*'  

to the

  espressif/opencv:
    override_path: "path to your opencv component"
    version: "*"

then, you will be able to change the opencv and opencv_contrib folders.

If the problem inside the opencv, then it will solve the problem.

Ok awesome I appreciate this, I was assuming something like this could be done. The issue with the version of opencv that I was using is that it can only be built using a container of linux with a very specific IDF version and tons of functionality needed to be removed to get it working. It is far too much hassle to use the old opencv build which I why I am very happy now about the cv component.

I will spend some time today getting peter's cv files transferred over and test, I spent a great deal of time yesterday trying to trouble shoot the problem with the camera and I really am not sure why it is happening. Like I said it only happens when I initialize the MicroSD card which makes me think that there is some sort of pin assignment problem or something like this but I have already tested this code to ensure it works so I haven't changed anything except using esp32-camera component and opencv components from the idf component manager.

Edit: @dmitry1945 have you tested sdcard functionality utilizing this component. I am really scratching my head at the problem. I think I will try making a blank project with only esp32-cam dependency to see if it may be something I am doing incorrectly. I have tested this on 3 different esp32s3 boards and am encountering the same issue.

[Ncouch64]

@dmitry1945 I have not tested switching of opencv component yet as I am troubleshooting the cam and sd card at the moment.

I did just finish putting together a small script to capture a single image and save to an sd_card without the CV component and the problem of camera failing is not occurring here. I will now try and do the most basic script using the CV component such as maybe turning an image to grayscale and try and save it to sd_card.

Could this really be something cause by the component? I am trying to think what would cause that and I just cant figure it out. Regardless, I will let you know what happens with my additional test. Hopefully it can maybe help catch a bug?

Edit : Just added in the opencv component as a dependency, and without even adding any of the header-files or using anything from the component, I encounter the frame capture failing. W (5184) cam_hal: Failed to get the frame on time!

[Ncouch64]

Wanted to throw another note here, for TermCriteria they use the variable EPS in opencv which is also defined in ESP. I am still working of figuring out the issue with FAST and the MicroSD Card. I think I might have figured out what I did wrong with FAST but I really cannot figure out what is wrong with the microsd. Have you been able to get it working? For something as simple as saving a cv image to the SD?

Edit: After all the testing I did I was unable to get any feature detection to work using FAST. I am not sure what the problem is there. It doesnt fail to build or run, but doesnt return any detected features.

[dmitry1945]

Hi @Ncouch64, I don't know the details of implementation for the FAST.
From my experience, I think the problem could be somewhere with combination of PSRAM+SD card + cache + camera.

[Ncouch64]

Hi @Ncouch64, I don't know the details of implementation for the FAST. From my experience, I think the problem could be somewhere with combination of PSRAM+SD card + cache + camera.

Gotcha, I will try and figure it out but I am not really even sure where to begin. Its really strange to me that saving to SD card works fine at any other point but as soon as I add the CV component as a dependency it just fails to write. Not sure why that would be. Ill mess around with it.