license didn’t get recognized

[roundssai]

Describe the bug
Licenses were created in the ambassador cloud and applied to the pod but continuous restarts were observed and upon seeing the logs it was the license server that was shutting down due to an invalid license.
Timeline - 19th OCT to 20th OCT.

To Reproduce
Steps to reproduce the behavior:

1. Create a license on Ambassador cloud
2. Start the pod with this license (version 3.8.x)
3. wait for a few days to observe the license failure.
4. Attaching logs here for your reference

{"CMD":"entrypoint","PID":1,"THREAD":"/envoy","file":"/go/vendor/github.com/datawire/dlib/dgroup/group.go:375","func":"github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1","level":"debug","msg":"goroutine \"/envoy\" exited without error","time":"2023-10-20 03:50:17.0160"}
[2023-10-20 03:50:17 +0000] [28] [INFO] Worker exiting (pid: 28)
[2023-10-20 03:50:17 +0000] [14] [INFO] Shutting down: Master
{"CMD":"entrypoint","PID":1,"THREAD":"/diagd","dexec.pid":14,"file":"/go/vendor/github.com/datawire/dlib/dexec/cmd.go:255","func":"github.com/datawire/dlib/dexec.(*Cmd).Wait","level":"info","msg":"finished successfully: exit status 0","time":"2023-10-20 03:50:17.2662"}
{"CMD":"entrypoint","PID":1,"THREAD":"/diagd","file":"/go/vendor/github.com/datawire/dlib/dgroup/group.go:375","func":"github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1","level":"debug","msg":"goroutine \"/diagd\" exited without error","time":"2023-10-20 03:50:17.2664"}
{"CMD":"entrypoint","PID":1,"THREAD":":shutdown_status","file":"/go/vendor/github.com/datawire/dlib/dgroup/group.go:84","func":"github.com/datawire/dlib/dgroup.logGoroutineStatuses","level":"info","msg":"  final goroutine statuses:","time":"2023-10-20 03:50:17.2666"}
{"CMD":"entrypoint","PID":1,"THREAD":":shutdown_status","file":"/go/vendor/github.com/datawire/dlib/dgroup/group.go:95","func":"github.com/datawire/dlib/dgroup.logGoroutineStatuses","level":"info","msg":"    /amb-sidecar             : exited with error","time":"2023-10-20 03:50:17.2667"}
{"CMD":"entrypoint","PID":1,"THREAD":":shutdown_status","file":"/go/vendor/github.com/datawire/dlib/dgroup/group.go:95","func":"github.com/datawire/dlib/dgroup.logGoroutineStatuses","level":"info","msg":"    /ambex                   : exited without error","time":"2023-10-20 03:50:17.2667"}
{"CMD":"entrypoint","PID":1,"THREAD":":shutdown_status","file":"/go/vendor/github.com/datawire/dlib/dgroup/group.go:95","func":"github.com/datawire/dlib/dgroup.logGoroutineStatuses","level":"info","msg":"    /diagd                   : exited without error","time":"2023-10-20 03:50:17.2667"}
{"CMD":"entrypoint","PID":1,"THREAD":":shutdown_status","file":"/go/vendor/github.com/datawire/dlib/dgroup/group.go:95","func":"github.com/datawire/dlib/dgroup.logGoroutineStatuses","level":"info","msg":"    /envoy                   : exited without error","time":"2023-10-20 03:50:17.2667"}
{"CMD":"entrypoint","PID":1,"THREAD":":shutdown_status","file":"/go/vendor/github.com/datawire/dlib/dgroup/group.go:95","func":"github.com/datawire/dlib/dgroup.logGoroutineStatuses","level":"info","msg":"    /external_snapshot_server: exited without error","time":"2023-10-20 03:50:17.2668"}
{"CMD":"entrypoint","PID":1,"THREAD":":shutdown_status","file":"/go/vendor/github.com/datawire/dlib/dgroup/group.go:95","func":"github.com/datawire/dlib/dgroup.logGoroutineStatuses","level":"info","msg":"    /healthchecks            : exited without error","time":"2023-10-20 03:50:17.2668"}
{"CMD":"entrypoint","PID":1,"THREAD":":shutdown_status","file":"/go/vendor/github.com/datawire/dlib/dgroup/group.go:95","func":"github.com/datawire/dlib/dgroup.logGoroutineStatuses","level":"info","msg":"    /memory                  : exited without error","time":"2023-10-20 03:50:17.2668"}
{"CMD":"entrypoint","PID":1,"THREAD":":shutdown_status","file":"/go/vendor/github.com/datawire/dlib/dgroup/group.go:95","func":"github.com/datawire/dlib/dgroup.logGoroutineStatuses","level":"info","msg":"    /snapshot_server         : exited without error","time":"2023-10-20 03:50:17.2669"}
{"CMD":"entrypoint","PID":1,"THREAD":":shutdown_status","file":"/go/vendor/github.com/datawire/dlib/dgroup/group.go:95","func":"github.com/datawire/dlib/dgroup.logGoroutineStatuses","level":"info","msg":"    /watcher                 : exited without error","time":"2023-10-20 03:50:17.2669"}
{"CMD":"entrypoint","PID":1,"file":"/go/pkg/busy/busy.go:87","func":"github.com/emissary-ingress/emissary/v3/pkg/busy.Main","level":"error","msg":"shut down with error error: exit status 1","time":"2023-10-20 03:50:17.2669"}

Expected behavior
Smooth running of ambassador pods

Versions (please complete the following information):

• Ambassador - 3.8.0 (is what we tried)
• Kubernetes environment - EKS AWS
• Version - 1.27

[cindymullins-dw]

Hi @roundssai , when was the license created - believe you said in our Slack channel that was Oct 9th. Were you continually connected to Ambassador Cloud between Oct 9th and when you started observing these errors on Oct 19th?

[roundssai]

@cindymullins-dw Yes, between OCT 9th to 18th we faced no issues. we suddenly started facing issues on that day and had to revert.

[cindymullins-dw]

Hi @roundssai , I'm not finding an Ambassador Cloud outage during the time in question. Also, if Edge Stack cannot reach Cloud because of an outage, Edge Stack won’t shut itself down - it continues to operate as normal and will check again for Cloud availability. So I think we can rule out Cloud server availability as the cause.

Considering other possibilities, did you try to reapply your existing license after getting errors and did it work or continue to fail? Did you create a new license? Were you using a Community license - how did you create the license that failed? If you can share (DM me if you prefer) the license key I may be able to check whether it perhaps expired or see the status history.

[roundssai]

@cindymullins-dw Right now we have rolled back to the older version after that issue. I could definitely try it out again and let you know.
The license was created by following the exact same steps mentioned in the document. One difference I noticed is that in the dashboard I see that it says the key never expires but during the outage period, I was able to see the expiry as 29th OCT or so. I can DM you the account details and you could probably check if there an issue.

[andrewnazarov]

We are experiencing restarts every 2 day or so. In Ambassador Cloud it's stated that our licences are valid for two days as well. In logs of the edge-stack pod we see the following messages:

level=error msg=“license manager shutdown with err: unlicensed: a valid unexpired license is required”

and

license-manager	licensemgt/manager.go:227	license manager shutdown initiated

So, a pod is definitely restarted due to the licence issues.

The following is shown in our Ambassador Cloud(I made a screenshot a few minutes ago):