[DataStorage] Runtime error

[tiokim]

Describe the bug
A runtime error when edgex foundry servers are not running,

level=ERROR ts=2021-01-19T01:51:50.815737266Z app=datastorage source=init.go:154 msg="Get \"http://localhost:48080/api/v1/ping\": dial tcp 127.0.0.1:48080: connect: connection refused"
level=INFO ts=2021-01-19T01:51:51.817181193Z app=datastorage source=init.go:144 msg="Check Metadata service's status by ping..."
level=INFO ts=2021-01-19T01:51:51.818287982Z app=datastorage source=init.go:144 msg="Check Data service's status by ping..."
level=ERROR ts=2021-01-19T01:51:51.822577012Z app=datastorage source=init.go:154 msg="Get \"http://localhost:48081/api/v1/ping\": dial tcp 127.0.0.1:48081: connect: connection refused"
level=ERROR ts=2021-01-19T01:51:51.824049381Z app=datastorage source=init.go:154 msg="Get \"http://localhost:48080/api/v1/ping\": dial tcp 127.0.0.1:48080: connect: connection refused"
level=INFO ts=2021-01-19T01:51:52.825768174Z app=datastorage source=init.go:144 msg="Check Metadata service's status by ping..."
level=INFO ts=2021-01-19T01:51:52.826907105Z app=datastorage source=init.go:144 msg="Check Data service's status by ping..."
level=ERROR ts=2021-01-19T01:51:52.830784824Z app=datastorage source=init.go:154 msg="Get \"http://localhost:48081/api/v1/ping\": dial tcp 127.0.0.1:48081: connect: connection refused"
level=ERROR ts=2021-01-19T01:51:52.83209855Z app=datastorage source=init.go:154 msg="Get \"http://localhost:48080/api/v1/ping\": dial tcp 127.0.0.1:48080: connect: connection refused"
INFO[2021-01-19T01:51:53Z]discovery.go:833 activeDiscovery [discoverymgr] activeDiscovery!!!
INFO[2021-01-19T01:51:53Z]discovery.go:571 func1 [deviceDetectionRoutine] edge-orchestration-3125da9e-1e9a-41aa-ac83-004725eb2d1e
level=ERROR ts=2021-01-19T01:51:53.83359109Z app=datastorage source=init.go:139 msg="dependency Metadata service checking time out"
level=ERROR ts=2021-01-19T01:51:53.834663766Z app=datastorage source=init.go:139 msg="dependency Data service checking time out"
level=INFO ts=2021-01-19T01:51:53.840074015Z app=datastorage source=httpserver.go:116 msg="Web server shutting down"
level=INFO ts=2021-01-19T01:51:53.841736032Z app=datastorage source=httpserver.go:107 msg="Web server stopped"
level=INFO ts=2021-01-19T01:51:54.341966491Z app=datastorage source=httpserver.go:118 msg="Web server shut down"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x8d1a0e]

goroutine 44 [running]:
github.com/edgexfoundry/device-sdk-go/internal/autoevent.(*manager).StopAutoEvents(0x0)
        /home/t25kim/edge-home-orchestration-go/vendor/github.com/edgexfoundry/device-sdk-go/internal/autoevent/manager.go:69 +0x4e
github.com/edgexfoundry/device-sdk-go/pkg/service.(*DeviceService).Stop(0xc0004c2780, 0xc0005ae600)
        /home/t25kim/edge-home-orchestration-go/vendor/github.com/edgexfoundry/device-sdk-go/pkg/service/service.go:134 +0x45
github.com/edgexfoundry/device-sdk-go/pkg/service.Main(0xe2dbda, 0xb, 0xe3a91d, 0x1a, 0xda46e0, 0xc0005a5de0, 0xf46e20, 0xc0005ae600, 0xc0000330a0, 0xc0005ac300, ...)
        /home/t25kim/edge-home-orchestration-go/vendor/github.com/edgexfoundry/device-sdk-go/pkg/service/main.go:69 +0x6fa
github.com/edgexfoundry/device-sdk-go/pkg/startup.Bootstrap(0xe2dbda, 0xb, 0xe3a91d, 0x1a, 0xda46e0, 0xc0005a5de0)
        /home/t25kim/edge-home-orchestration-go/vendor/github.com/edgexfoundry/device-sdk-go/pkg/startup/bootstrap.go:19 +0x117
created by github.com/lf-edge/edge-home-orchestration-go/src/controller/storagemgr.StorageImpl.StartStorage
        /home/t25kim/edge-home-orchestration-go/src/controller/storagemgr/storage.go:51 +0xef

To Reproduce

1. Put necessary configuration files in /var/edge-orchestration/datastorage/
2. Run edge-home-orchestration-go

Expected behavior
Check the edgex foundry server in advance before starting Data Storage.

Test environment configuration (please complete the following information):

• Firmware version: Ubuntu 18.04
• Hardware: x86-64
• Edge Orchestration Release: Coconut

[MoonkiHong]

@suresh-lc PTAL. Thank you in advance.

[suresh-lc]

Hi Sunchit,

I've figured out why it crashed. It's a bug, and it's not so hard to fix.

The AutoEvent Manager isn't initialized when the Service is stopped by the dependency time out (EdgeX Core Data and Core Metadata).

In this line, https://github.com/edgexfoundry/device-sdk-go/blob/v1.4.0/pkg/service/service.go#L134

We should check whether the AutoEvent Manager is nil before calling StopAutoEvents.

We have resolved this issue in V2 (Ireland).

Hi Jim and Iain,

May we create a hanoi branch to fix this issue and tag 1.4.1 for them?

[tiokim]

Hi Sunchit,

I've figured out why it crashed. It's a bug, and it's not so hard to fix.

The AutoEvent Manager isn't initialized when the Service is stopped by the dependency time out (EdgeX Core Data and Core Metadata).

In this line, https://github.com/edgexfoundry/device-sdk-go/blob/v1.4.0/pkg/service/service.go#L134

We should check whether the AutoEvent Manager is nil before calling StopAutoEvents.

We have resolved this issue in V2 (Ireland).

Hi Jim and Iain,

May we create a hanoi branch to fix this issue and tag 1.4.1 for them?

@suresh-lc Who wrote this email?

[suresh-lc]

Hi Sunchit,
I've figured out why it crashed. It's a bug, and it's not so hard to fix.
The AutoEvent Manager isn't initialized when the Service is stopped by the dependency time out (EdgeX Core Data and Core Metadata).
In this line, https://github.com/edgexfoundry/device-sdk-go/blob/v1.4.0/pkg/service/service.go#L134
We should check whether the AutoEvent Manager is nil before calling StopAutoEvents.
We have resolved this issue in V2 (Ireland).
Hi Jim and Iain,
May we create a hanoi branch to fix this issue and tag 1.4.1 for them?

@suresh-lc Who wrote this email?

Cloud Tsai from EdgeX- Device Service team

[tiokim]

This issue will be resolved by using the edgexfoundry 'Ireland' version, but it has not been officially released.
I'd like to leave this as a known issue and upgrade edgexfoundry when Ireland is released.

[sun-sharma]

Post communication with EdgeX members, they have provided us with a fork : github.com/hahattan/device-sdk-go v1.4.1 with the fix on the Hanoi branch, causing this crash.

I had been testing the fix by using the replace directive in our go.mod. That seems to work well.

As you have mentioned, we can wait till the Ireland release or we can also clone the fix to our repo till the time EdgeX releases Ireland.

Please share your thoughts on the same ?
@t25kim

Thanks !

[tiokim]

@sun-sharma Thank you for letting us know.
Could you modify the go.mod file with the replace flag?
It would be nice to add a note that it is temporary and will be removed.