Spinnaker 2.3.0.77: Segmentation fault upon creating system instance

[rgov]

System Description
Camera Name and Model: FLIR Blackfly S BFS-U3-50S5C-C
Operating System: Ubuntu 18.04 "Bionic"
Spinnaker Version: 2.3.0.77
ROS Version: Melodic

Computer details
Processor: Jetson TX2 (aarch64)
RAM: 4 GB?

Do the cameras work with SpinView?

They work with the Python examples provided with the Spinnaker SDK.

Describe the bug

I am using the Feature/spinnaker_update branch with a few patches from #89.

Shortly after the node loads, it crashes calling System::GetInstance() in acquisition::Capture::init_variables_register_to_ros().

Error Messages

[ INFO] [1611688595.430654747]: *** SYSTEM INFORMATION ***
[ INFO] [1611688595.430699675]: Creating system instance...

Thread 1 "acquisition_nod" received signal SIGSEGV, Segmentation fault.
0x0000007fb7cdc560 in std::istream::sentry::sentry(std::istream&, bool) () from /usr/lib/aarch64-linux-gnu/libstdc++.so.6
(gdb) bt
#0  0x0000007fb7cdc560 in std::istream::sentry::sentry(std::istream&, bool) () at /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#1  0x0000007fb7cddc28 in std::istream::sync() () at /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#2  0x0000007fb0340868 in boost::archive::basic_text_iprimitive<std::istream>::~basic_text_iprimitive() () at /usr/lib/aarch64-linux-gnu/libboost_serialization.so.1.65.1
#3  0x0000007f9184c688 in  () at /opt/spinnaker/lib/flir-gentl/FLIR_GenTL.cti
#4  0x0000007f9184caf0 in  () at /opt/spinnaker/lib/flir-gentl/FLIR_GenTL.cti
#5  0x0000007f91821060 in  () at /opt/spinnaker/lib/flir-gentl/FLIR_GenTL.cti
#6  0x0000007f9181c214 in  () at /opt/spinnaker/lib/flir-gentl/FLIR_GenTL.cti
#7  0x0000007f917eed58 in TLOpen () at /opt/spinnaker/lib/flir-gentl/FLIR_GenTL.cti
#8  0x0000007fa12d4ed0 in  () at /opt/spinnaker/lib/libSpinnaker.so.2
#9  0x0000007fa1365320 in  () at /opt/spinnaker/lib/libSpinnaker.so.2
#10 0x0000007fa13fcfb0 in Spinnaker::System::GetInstance() () at /opt/spinnaker/lib/libSpinnaker.so.2
#11 0x0000007fb04808ac in acquisition::Capture::init_variables_register_to_ros() () at /home/user/my-proj/devel/lib//libacquilib.so

[rgov]

I have contacted FLIR about this issue.

[ghost]

I'd suggest trying older versions of their driver, possibly 2.0.0.147, we have been using it with TX2 successfully.

[rgov]

Yes, 2.0.0.147 works. Thanks for that tip.

[PushyamiKaveti]

Seems resolved. closing the issue.

[rgov]

Well, it is not resolved because the module is seemingly incompatible with the latest version of Spinnaker. The fact that it crashes—especially so early in initialization—suggests that it is violating some (perhaps unstated) API contract.

I tried reaching out to FLIR but it is difficult for me to make a reduced test case, and they weren't interested even in looking at the crash log. Since they stripped symbols from their library, we cannot even figure out where it is crashing.

They suggested maybe we are initializing the library more than once from different threads, which might make sense when dealing with nodelets, but now that I have my system in working order, I'm hesitant to revert back to broke to investigate.

[ntomer]

Where do you download older versions of spinnaker?

[ghost]

You can find some of the older versions under archive folder in spinnakers download page on box.lenovo.com at https://meta.box.lenovo.com/v/link/view/a1995795ffba47dbbe45771477319cc3

Are you looking for a specific version?

[ntomer]

Ah, thanks!
No, I am not looking for a particular version. I had a similar issue as 'rgov' setting up a new computer (Ubuntu 20.04 desktop), and was able to find spinnaker 2.2.0.48 on a different machine I had set up. I transferred it it to the new computer and that fixed the problem for me. I also contacted FLIR so hopefully they will take a closer look at the issue if there are multiple tickets.

[ghost]

@ntomer It is good to have feedback on compatibility from different versions.
Are you using ROS Noetic on Ubuntu 20.04? Is this an x64 architecture? So just to confirm, are you saying that 2.3.0.77 didn't work and 2.2.0.48 did work?

[ntomer]

@shahvi I should clarify that I am using a slightly modified fork of the repo (though not too extensively I think), but can install this one to see if it works. However yes, I am running on Ubuntu 20.04, x64 with 2.2.048 working, and 2.3.0.77 did not work.

Also just heard back from FLIR support and they said that they don't support the ROS platform or this code base, and can't help debugging.

[ghost]

@ntomer that is what I'd have expected from FLIR. In the past we saw a similar issue, probably because of shared pointers used by Spinnaker. I am sure someone else will have an issue with it too and they will get it resolved in one of the subsequent releases.

One thing to try might be the node version instead of the nodelet which is default. You can launch that using the node_acquisition.launch

[ntomer]

@shahvi I was able to recreate the problem with the current code base with some modification (changing opencv import, removing display functionality) on Noetic Ubuntu 20.04 x64, using the node launch file.
Upon installing the 2.3.0.77 version of spinnaker, spinview failed to load and displayed an error stating that it was unable to get a system instance, stating that the genTL paths may not be setup correctly.
Restarting the computer solved the spinview crash problem, but the node still crashes. Running "source /etc/profile.d/setup_flir_gentl_64.sh 64" did not help the node run so I am at a bit of a loss. Might just have to wait for a later release as you say.