With a custom dataset, the locations diverge.

[Z-Xiong]

We use mpu6000 to obtain the accelerometer and gyroscope values at 100Hz while collecting the frequency at 20Hz. And used the method in the documentation to calibrate the camera and imu and between the camera and imu. Below are the configuration files used.
imu.yaml

#Accelerometer
accelerometer_noise_density: 0.0006739720604502373 
accelerometer_random_walk: 8.573760135961954e-06 

#Gyroscope
gyroscope_noise_density: 0.00497167224696044 
gyroscope_random_walk: 0.00014068225293967367 

rostopic: '/imu0' #Make sure this is correct
update_rate: 100.0 #Make sure this is correct

camchain-imucam.yaml

cam0:
  T_cam_imu:
  - [-0.15648538643630197, 0.9862054960014432, -0.0539540868556353, 0.0005347724669216995]
  - [0.1594151437326083, 0.07913042358480787, 0.9840351558821452, -0.00012586161076496793]
  - [0.9747302887366279, 0.14538602312407162, -0.16959884580849552, -0.00036736308899636025]
  - [0.0, 0.0, 0.0, 1.0]
  camera_model: pinhole
  distortion_coeffs: [0.36261619267439066, -1.4153139694196566, -0.003438440763239287, 0.0032504558613163]
  distortion_model: radtan
  intrinsics: [723.8666745706989, 726.821998039784, 321.1721901213624, 160.84603758249207]
  line_delay: -7.007640624981265e-05
  resolution: [640, 360]
  rostopic: /cam0/image_raw
  timeshift_cam_imu: -0.06350724145105457

imu-report-imucam.pdf

Calibration results
===================
Normalized Residuals
----------------------------
Reprojection error (cam0): mean 0.9636449149800216, median 0.7609756045271983, std: 0.7468461912975807
Gyroscope error (imu0):
mean 3.150741835498185, median 2.80902840927942, std: 2.0930817173652922
Accelerometer error (imu0): mean 2.088231434501006, median 1.6305300620780163, std: 1.6210970086298837
Residuals
----------------------------
Reprojection error (cam0) [px]: mean 0.9636449149800216, median 0.7609756045271983, std: 0.7468461912975807
Gyroscope error (imu0) [rad/s]: mean 0.276911071949382, median 0.24687870621010133, std: 0.1839558847707472
Accelerometer error (imu0) [m/s^2]: mean 0.07037048213037986, median 0.054946585278238695, std:
0.05462870455479994
Transformation (cam0):
-----------------------
T_ci: (imu0 to cam0):
[[-0.26800681 0.95576049 -0.12121976 0.34365608]
[ 0.79429383 0.29041006 0.53362844 -0.58104977]
[ 0.54522442 0.04673195 -0.83698653 0.11096242]
[ 0. 0. 0. 1.]]
T_ic: (cam0 to imu0):
[[-0.26800681 0.79429383 0.54522442 0.49312699]
[ 0.95576049 0.29041006 0.04673195 -0.1648957 ]
[-0.12121976 0.53362844 -0.83698653 0.44459664]
[ 0. 0. 0. 1.]]
timeshift cam0 to imu0: [s] (t_imu = t_cam + shift)
-0.04036013561890319
Gravity vector in target coords: [m/s^2]
[-5.44247206 1.2940873 -8.05439376]

Calibration configuration
=========================
cam0
-----
Camera model: pinhole
Focal length: [723.8666745706989, 726.821998039784]
Principal point: [321.1721901213624, 160.84603758249207]
Distortion model: radtan
Distortion coefficients: [0.36261619267439066, -1.4153139694196566, -0.003438440763239287, 0.0032504558613163]
Type: checkerboard
Rows
Count: 9
Distance: 0.024 [m]
Cols
Count: 6
Distance: 0.024 [m]

IMU configuration
=================
IMU0:
----------------------------
Model: calibrated
Update rate: 100.0
Accelerometer:
Noise density: 0.0033698603022511864
Noise density (discrete): 0.03369860302251186
Random walk: 8.573760135961954e-05
Gyroscope:
Noise density: 0.008788757899156717
Noise density (discrete): 0.08788757899156717

Random walk: 0.0004973868752312219
T_ib (imu0 to imu0)
[[1. 0. 0. 0.]
[0. 1. 0. 0.]
[0. 0. 1. 0.]
[0. 0. 0. 1.]]
time offset with respect to IMU0: 0.0 [s]

newest-imu-report-imucam.pdf
How should I troubleshoot the problem?