Client exception in call to com.sitewhere.grpc.service

[ATM006]

error 1、Client exception in call to com.sitewhere.grpc.service.DeviceManagement/ListDeviceSummaries.CLOSE
error 2、Client exception in call to com.sitewhere.grpc.service.AssetManagement/GetAssetById.CLOSE

[derekadams]

Can you provide details regarding the Kubernetes cluster where you are running SiteWhere? In particular:

• What provider (Minikube, Google Cloud, etc)?
• How many k8s nodes and how much CPU/RAM for each node?
• What version of Kubernetes?
• Which version of SiteWhere?
• Was SiteWhere installed via swctl install or directly via Helm?

[ATM006]

• What provider (Minikube, Google Cloud, etc)?
  （Naked metal deployment）
  https://github.com/kubeoperator/kubeoperator/
• How many k8s nodes and how much CPU/RAM for each node?
  3nodes（total：CPU12C、RAM24G）
• What version of Kubernetes?
  v1.18.8
• Which version of SiteWhere?
  -v3.0.5
• Was SiteWhere installed via swctl install or directly via Helm?
  swctl install（v0.9.2）

[derekadams]

Thanks, that configuration should work well. I was just making sure that the services were not starved for CPU or other resources. It looks like the root of the issue is failed interactions with the device management microservice when getting device and/or listing assignments. Are there exceptions in the logs for that microservice. Also, is there any particular use case that triggers these issues?

[ATM006]

Thanks for your reply.The current bug will appear after entering the tenant after each new login. This is the error log.
logs-from-device-management-in-device-management-65c6795c77-w58qp.txt
logs-from-instance-management-in-instance-management-59dd64f84b-hwt9d.txt

[derekadams]

The root of this error seems to be the following lines:

2021-03-28 02:47:11,541 WARN  [org.hib.eng.jdb.spi.SqlExceptionHelper] (grpc-default-executor-7) SQL Error: 0, SQLState: 08006
2021-03-28 02:47:11,542 ERROR [org.hib.eng.jdb.spi.SqlExceptionHelper] (grpc-default-executor-7) An I/O error occurred while sending to the backend.
2021-03-28 02:47:11,542 WARN  [org.hib.eng.jdb.spi.SqlExceptionHelper] (grpc-default-executor-8) SQL Error: 0, SQLState: 08006
2021-03-28 02:47:11,542 ERROR [org.hib.eng.jdb.spi.SqlExceptionHelper] (grpc-default-executor-8) An I/O error occurred while sending to the backend.

It seems like in this case, the database had an error, which caused the device management APIs to fail. If you look in the sitewhere-system namespace at the logs for PostgreSQL, are there any indications of what is failing in the database? This could be another issue with running out of space on the PVC.