Some D455 not woring as slave in sync system

[FrGrQuim]

Required Info
Camera Model	D455
Firmware Version	5.16.0.1
Operating System & Version	Linux (Ubuntu 22)
Kernel Version (Linux Only)	5.15.0
Platform	PC
SDK Version	2.55.1
Language
Segment	Robot

Issue Description

For a robotic project, we are utilizing the RealSense D455 with synchronization. However, for the past few weeks, we've encountered a new issue: when switching between master and slave roles, the "new" slave camera ceases to send framesets. Once this occurs, a hardware reset is necessary to resume data transmission. This problem arises exclusively when depth frames are configured at 5Hz (we've also tested at 15Hz and 30Hz, where it functions correctly). Upon reverting the roles, the cameras operate correctly with synchronized depth frames (although a hard reset is needed if the slave camera was "broken").

If the depth stream is disabled on the master camera, the slave camera functions properly. Similarly, if the sync cable is disconnected, frames from both cameras are received but remain unsynchronized (which is normal).

We've replicated this issue using both a complete sync cable and a cable with only pins 5 and 9 connected.

Additionally, we've measured the sync signal:



M2 and S2 represent the initial Master and Slave cameras (when the system functions properly).

This behavior has been precisely replicated using the RealSense Viewer program by solely adjusting the frequency of the depth frame and the role of the two cameras (all other parameters remain unchanged). Therefor it seems that the problem don't come from our hardware or our application.

Is it possible that we've damaged a camera, rendering it unable to function as a slave (crashing upon receiving a sync signal)? If so, why does this occur only at 5Hz, and how can we prevent it from happening to other cameras? If not, do you have any other insights into why we're experiencing this behavior?

If you require any further information, please don't hesitate to ask, and I'll do my best to provide it.

[MartyG-RealSense]

Hi @FrGrQuim Changing a camera's Inter Cam Sync Mode setting from master to slave should not damage it and it does not sound as though there is damage since it works normally at 15 and 30 Hz.

The ideal situation is if the master is enabled first and the slave(s) secondly afterwards.

Are you changing the camera from master to slave whilst depth streaming is still enabled? If you are, does the problem still occur at 5 Hz if you disable the depth streams, change the Inter Cam Sync Modes and then re-enable the depth streams?

[FrGrQuim]

Hi @MartyG-RealSense, thanks for your response. We always enable the master before the slave, and in our app, we disable the stream before configuring the cameras. (In the RealSense Viewer, it's not possible to change the Inter-Cam Sync Mode when the stream is enabled).

If the issue isn't with the hardware, another hypothesis is that there may be corruption in the persistent memory. I've updated the cameras, so I don't believe it could be from the firmware, but perhaps there's a persistent configuration that causes the cameras to crash under certain conditions. Is there a way to factory reset the camera, not just the calibration parameters?

Also, is there a method to retrieve error codes, logs, or any other type of information that could help us understand why the camera isn't sending frames?

[MartyG-RealSense]

A factory-reset of the calibration creates a new calibration table inside the camera. The camera can not function fully correctly if the calibration table has somehow become corrupted, so writing a new table through a calibration reset can rectify problems caused by a damaged cable. The calibration reset can be performed in the Viewer using the instructions at #10182 (comment)

Also in the Viewer, you can expand open a debug console panel that displays a continuously updating message log by clicking on a small upward-pointing arrow icon in the bottom corner of the Viewer window.

You can set a path on your computer for the log to be sent to a file by clicking on the gear-wheel icon at the top corner of the Viewer window and selecting the Settings option from its drop-down menu, then selecting the General tab on the settings window that pops up and clicking on the box beside 'Output librealsense log to file'.

Select a path on your computer to save the log file to. Finally, click the Apply button on the bottom of the Settings pop-up to confirm the enabling of logging to file.

The link below provides information about accessing other types of logging such as kernel logs.

https://github.com/IntelRealSense/librealsense/blob/master/doc/troubleshooting.md

[FrGrQuim]

I have attempted to factory reset the calibration table of the two cameras, but this didn't resolve the issue. However, when I checked the logs in the RealSense Viewer app, I observed the following logs when I started the depth stream of the slave camera (the master camera's depth stream was already enabled without any error logs):

After that, I encountered the following log that loops indefinitely:

[MartyG-RealSense]

Do the ** Stream Start Failure** and Left IC2 Config Error errors display every time that this particular camera has its depth stream enabled?

'Left IC2 Config Error' is a very rare error that typically indicates that an interposer cable that joins two circuit boards together inside the camera may not be fully seated in its connectors. It is unlikely that there would be a problem with this internal cable though if that particular camera is able to start the depth stream normally most of the time.

[FrGrQuim]

The Left IC2 Config Error occurs each time I initiate the depth stream on this camera under the following conditions:

• The camera is in slave mode
• The stream is set to 5Hz
• The sync cable is connected (with both the master and slave cameras)
• The depth stream of the master camera is enabled

If any of these conditions are not met, I don't encounter the Left IC2 Config Error, and the stream starts correctly. Additionally, if the camera is already stuck in this issue, attempting to restart the stream (even without meeting the aforementioned conditions) will still fail with the Left IC2 Config Error. A hardware reset is necessary to successfully restart the stream.

The occurrence of Stream Start Failure is not consistent, and I haven't identified the specific conditions that trigger it.

[MartyG-RealSense]

There is likely not a physical hardware fault, then.

I would recommend avoiding use of 5 Hz if possible and using a minimum of 15 Hz. At 5 Hz new frames arrive in the frame pipeline slowly enough that the risk of a problem occurring increases compared to 15 Hz and above, where such problems experienced at 5 Hz disappear.

[FrGrQuim]

Okay, so currently, there is no explanation for the root cause of this issue? Is there something we can do to help you understand this problem? Our robot cannot handle 15Hz streams, and even if we limit the rate by requesting a framerate of 5Hz, we still want to keep the camera's framerate consistent with our application.

Additionally, we would like to retrieve the camera logs (similar to the ones available on the RealSense Viewer) directly on the robot. Is there an API that allows us to do this? I tried using the rs2::device::get_info with this parameter:

However, I couldn't find where I can locate the meaning of the debug OP code.

[MartyG-RealSense]

I do not have hardware sync equipment to replicate and test your sync setup, unfortunately.

It may be worth testing to see whether the problem still occurs when using genlock hardware sync. This will involve setting 5 FPS and Inter Cam Sync Mode '1' on your master camera and Inter Cam Sync Mode '4' (instead of 2) on your slave camera, with the slave camera's FPS set to 15 instead of 5. This is because when using a genlock sync mode with a master camera, the slave FPS must be 3x the master FPS. The slave camera will actually operate at the master's 5 FPS, not 15.

If you are programming your own application with script code then a workaround in the past for problems with a particular FPS speed has been to only use every "nth" frame - for example, setting 30 FPS but only using every 6th frame in order to achieve an actual FPS of 5. A RealSense user developed a Python script to demonstrate this principle at #3169 - it sounds as though you are using C++ but the Python script might provide insights about how to implement a similar custom FPS mechanism in C++.

There is not a list of meanings provided by Intel for the debug op codes as the purpose of the logging mechanism was for RealSense end-users to provide the log to Intel so that an Intel RealSense firmware engineer could interpret it.

The RealSense SDK provides a C++ firmware logging tool called rs-fw-logger that provides similar functionality to the RealSense Viewer's firmware log mechanism.

https://github.com/IntelRealSense/librealsense/tree/master/tools/fw-logger

Further information about this logging tool can be found at #1215

[FrGrQuim]

Hi @MartyG-RealSense,

I've tested genlock 3 mode, and it appears to be functioning correctly. I hope this result can help you understand the root cause of this issue.
Thank you for your assistance. However, we remain interested in understanding the underlying issue and how to address it (or at least prevent its recurrence with new cameras).

[MartyG-RealSense]

Genlock mode '3' is known as Full Slave mode and follows the same rules as slave mode 2 except that it attempts to sync both depth and RGB instead of only depth. The RGB sync in this mode was intended for use with the D415 camera model and it did not work well, but I'm pleased to hear that it helped to resolve your depth syncing between master and slave.

[FrGrQuim]

Sorry I made a mistake, it's the Inter Cam Sync Mode '4' that I have used.

[MartyG-RealSense]

No problem, thanks very much for the clarification.

A key difference between mode 2 and mode 4 (other than mode 4 not syncing RGB) is that mode 2 listens for a sync trigger pulse on each frame and if it does not recognize a pulse within a certain time period then it proceeds with initiating an unsynced capture on that frame. So the camera is always capturing regardless of whether a sync trigger is received or not.

With mode 4 though, the camera waits indefinitely for a trigger pulse and then initiates capture on that frame once the pulse is received. Then it starts waiting indefinitely again until the next trigger pulse arrives.

[MartyG-RealSense]

Hi @FrGrQuim Do you have an update about this case that you can provide, please? Thanks!

[FrGrQuim]

Hi @MartyG-RealSense,

After testing the genlock mode for two weeks, I can provide some feedback.

The genlock mode seems to work well for several consecutive hours, but after this period, the frames from the slave camera start to be consistently +/- 66ms more recent than the frames from the master camera (66 ms corresponds to a time period equivalent to 15Hz). Performing a hard reset of the camera did not correct the problem, but unplugging and re-plugging it did.

So far, I have tested only one set of cameras, and I will try another set today. I will update you with the results.

[MartyG-RealSense]

Thank you, @FrGrQuim - I look forward to your next update.