0.3.11 fails to start when a schedule definition exists in the DB

[bvandewe]

What happened:

Synapse server 0.3.11 fails to boot and throws the following logs:

[11/07/2022 16:47:57] warn: ProtoBuf.Grpc.Server.ServicesExtensions.CodeFirstServiceMethodProvider[0]
      Type cannot be serialized; ignoring: Synapse.Integration.Commands.Workflows.V1UploadWorkflowCommand
[11/07/2022 16:47:57] warn: ProtoBuf.Grpc.Server.ServicesExtensions.CodeFirstServiceMethodProvider[0]
      Signature not recognized for Synapse.Apis.Management.Grpc.ISynapseGrpcManagementApi.UploadWorkflowAsync; method will not be bound
[11/07/2022 16:47:57] warn: ProtoBuf.Grpc.Server.ServicesExtensions.CodeFirstServiceMethodProvider[0]
      Signature not recognized for Synapse.Apis.Management.Grpc.ISynapseGrpcManagementApi.ArchiveWorkflowAsync; method will not be bound
[11/07/2022 16:47:57] info: ProtoBuf.Grpc.Server.ServicesExtensions.CodeFirstServiceMethodProvider[0]
      RPC services being provided by Synapse.Apis.Management.Grpc.SynapseGrpcManagementApi: 41
[11/07/2022 16:47:57] info: ProtoBuf.Grpc.Server.ServicesExtensions.CodeFirstServiceMethodProvider[0]
      RPC services being provided by Synapse.Apis.Runtime.Grpc.SynapseGrpcRuntimeApi: 22
[11/07/2022 16:47:57] info: Synapse.Application.Services.PluginManager[0]
      Scanning directory'/app/plugins' for plugins...
[11/07/2022 16:47:57] info: Synapse.Application.Services.PluginManager[0]
      Found 2 matching plugin files in directory '/app/plugins'
[11/07/2022 16:47:57] warn: Microsoft.AspNetCore.Server.Kestrel[0]
      Overriding address(es) 'http://+:80'. Binding to endpoints defined via IConfiguration and/or UseKestrel() instead.
[11/07/2022 16:47:57] info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://[::]:41387
[11/07/2022 16:47:57] info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://[::]:42286
[11/07/2022 16:47:57] info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
[11/07/2022 16:47:57] info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
[11/07/2022 16:47:57] info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app/
[11/07/2022 16:47:57] info: Synapse.Application.Services.PluginManager[0]
      2 plugins have been found in '/app/plugins' directory
[11/07/2022 16:47:57] info: Synapse.Application.Services.PluginManager[0]
      Loading plugin 'EventStore:0.2.0'...
[11/07/2022 16:47:57] info: Synapse.Application.Services.PluginManager[0]
      The plugin 'EventStore:0.2.0' has been successfully loaded
[11/07/2022 16:47:57] info: Synapse.Application.Services.PluginManager[0]
      Loading plugin 'MongoDB:0.2.0'...
[11/07/2022 16:47:57] info: Synapse.Application.Services.PluginManager[0]
      The plugin 'MongoDB:0.2.0' has been successfully loaded
[11/07/2022 16:47:58] fail: Microsoft.Extensions.Hosting.Internal.Host[9]
      BackgroundService failed
      System.ArgumentException: The specified schedule does not define a next occurence (Parameter 'schedule')
         at Synapse.IBackgroundJobManagerExtensions.ScheduleJobAsync(IBackgroundJobManager backgroundJobManager, V1Schedule schedule, CancellationToken cancellationToken) in /src/src/core/Synapse.Application/Extensions/IBackgroundJobManagerExtensions.cs:line 53
         at Synapse.Application.Services.WorkflowScheduler.ExecuteAsync(CancellationToken stoppingToken) in /src/src/core/Synapse.Application/Services/WorkflowScheduler.cs:line 76
         at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
[11/07/2022 16:47:58] crit: Microsoft.Extensions.Hosting.Internal.Host[10]
      The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted.
      System.ArgumentException: The specified schedule does not define a next occurence (Parameter 'schedule')
         at Synapse.IBackgroundJobManagerExtensions.ScheduleJobAsync(IBackgroundJobManager backgroundJobManager, V1Schedule schedule, CancellationToken cancellationToken) in /src/src/core/Synapse.Application/Extensions/IBackgroundJobManagerExtensions.cs:line 53
         at Synapse.Application.Services.WorkflowScheduler.ExecuteAsync(CancellationToken stoppingToken) in /src/src/core/Synapse.Application/Services/WorkflowScheduler.cs:line 76
         at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
[11/07/2022 16:47:58] info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...
[11/07/2022 16:47:58] fail: Neuroglia.Eventing.Services.CloudEventBus[0]
      An error occured while dequeuing and publishing pending cloud events: System.OperationCanceledException: The operation was canceled.
         at System.Threading.Channels.AsyncOperation`1.GetResult(Int16 token)
         at Neuroglia.Eventing.Services.CloudEventBus.DequeueAndPublishPendingEventsAsync()

What you expected to happen: Synapse to start regardless of whether there is a schedule already defined or not.

How to reproduce it: Create a schedule definition for an existing workflow prior to attempt the upgrade to latest release...

Anything else we need to know?:

Environment:

[bvandewe]

FYI - 0.3.11 starts just fine when the persistance layer is empty/reset.

[cdavernas]

I believe this is due to a schedule being invalid (with an validUntil value set, right?) which therefore throws at startup when reconciling the scheduler.

I will need to catch exceptions atomically for each schedule.

[cdavernas]

Closed as fixed in #366