Crash when running the latest docker image

[FranzBusch]

We are seeing a crash happening quite frequently which happens inside the Postgres stack. I am trying to investigate further where this crash is coming from.

The crash log:

0x7f9b8248f97f
0x55e0ea5038df, Swift runtime failure: Unexpectedly found nil while unwrapping an Optional value at /build/.build/checkouts/swift-nio/Sources/NIO/ByteBuffer-core.swift:0
0x55e0ea5038df, static NIO.ByteBuffer._Storage.(allocateAndPrepareRawMemory in _243B594EFDF92ACEC7911DAF9CEF250E)(bytes: Swift.UInt32, allocator: NIO.ByteBufferAllocator) -> Swift.UnsafeMutableRawPointer at /build/.build/checkouts/swift-nio/Sources/NIO/ByteBuffer-core.swift:243
0x55e0ea5038df, NIO.ByteBuffer._Storage.(allocateStorage in _243B594EFDF92ACEC7911DAF9CEF250E)(capacity: Swift.UInt32) -> NIO.ByteBuffer._Storage at /build/.build/checkouts/swift-nio/Sources/NIO/ByteBuffer-core.swift:255
0x55e0ea5038df, NIO.ByteBuffer._Storage.allocateStorage() -> NIO.ByteBuffer._Storage at /build/.build/checkouts/swift-nio/Sources/NIO/ByteBuffer-core.swift:250
0x55e0ea5038df, NIO.ByteBuffer.clear() -> () at /build/.build/checkouts/swift-nio/Sources/NIO/ByteBuffer-core.swift:706
0x55e0ea52983d, NIO.MessageToByteHandler.write(context: NIO.ChannelHandlerContext, data: NIO.NIOAny, promise: Swift.Optional<NIO.EventLoopPromise<()>>) -> () at /build/.build/checkouts/swift-nio/Sources/NIO/Codec.swift:795
0x55e0ea51942f, NIO.ChannelHandlerContext.(invokeWrite in _EEC863903996E9F191EBAFEB0FB0DFDD)(_: NIO.NIOAny, promise: Swift.Optional<NIO.EventLoopPromise<()>>) -> () at /build/.build/checkouts/swift-nio/Sources/NIO/ChannelPipeline.swift:1424
0x55e0ea51b2fd
0x55e0ea7207b3, function signature specialization <Arg[3] = Dead> of generic specialization <PostgresNIO.PostgresRequestHandler> of (extension in PostgresNIO):NIO.ChannelInboundHandler.write(context: NIO.ChannelHandlerContext, items: Swift.Array<A.OutboundOut>, promise: Swift.Optional<NIO.EventLoopPromise<()>>) -> () at /build/.build/checkouts/postgres-nio/Sources/PostgresNIO/Connection/PostgresConnection+Database.swift:120
0x55e0ea71d131, generic specialization <PostgresNIO.PostgresRequestHandler> of (extension in PostgresNIO):NIO.ChannelInboundHandler.write(context: NIO.ChannelHandlerContext, items: Swift.Array<A.OutboundOut>, promise: Swift.Optional<NIO.EventLoopPromise<()>>) -> () at /build/<compiler-generated>:0
0x55e0ea71d131, PostgresNIO.PostgresRequestHandler.write(context: NIO.ChannelHandlerContext, data: NIO.NIOAny, promise: Swift.Optional<NIO.EventLoopPromise<()>>) -> () at /build/.build/checkouts/postgres-nio/Sources/PostgresNIO/Connection/PostgresConnection+Database.swift:94
0x55e0ea51942f, NIO.ChannelHandlerContext.(invokeWrite in _EEC863903996E9F191EBAFEB0FB0DFDD)(_: NIO.NIOAny, promise: Swift.Optional<NIO.EventLoopPromise<()>>) -> () at /build/.build/checkouts/swift-nio/Sources/NIO/ChannelPipeline.swift:1424
0x55e0ea519456, NIO.ChannelHandlerContext.(invokeWrite in _EEC863903996E9F191EBAFEB0FB0DFDD)(_: NIO.NIOAny, promise: Swift.Optional<NIO.EventLoopPromise<()>>) -> () at /build/.build/checkouts/swift-nio/Sources/NIO/ChannelPipeline.swift:1426
0x55e0ea515dd4
0x55e0ea50c8c8
0x55e0ea50c816, protocol witness for NIO.ChannelOutboundInvoker.write(_: NIO.NIOAny, promise: Swift.Optional<NIO.EventLoopPromise<()>>) -> () in conformance NIO.BaseSocketChannel<A1> : NIO.ChannelOutboundInvoker in NIO at /build/<compiler-generated>:0
0x55e0ea511887, function signature specialization <Arg[1] = Dead, Arg[2] = Dead> of (extension in NIO):NIO.ChannelOutboundInvoker.write(_: NIO.NIOAny, file: Swift.StaticString, line: Swift.UInt) -> NIO.EventLoopFuture<()> at /build/.build/checkouts/swift-nio/Sources/NIO/ChannelInvoker.swift:132
0x55e0ea50cb38
0x55e0ea71feca, function signature specialization <Arg[0] = Existential To Protocol Constrained Generic> of PostgresNIO.PostgresConnection.send(_: PostgresNIO.PostgresRequest, logger: Logging.Logger) -> NIO.EventLoopFuture<()> at /build/.build/checkouts/postgres-nio/Sources/PostgresNIO/Connection/PostgresConnection+Database.swift:11
0x55e0ea48d50a, FluentPostgresDriver._FluentPostgresDatabase.send(_: PostgresNIO.PostgresRequest, logger: Logging.Logger) -> NIO.EventLoopFuture<()> at /build/.build/checkouts/fluent-postgres-driver/Sources/FluentPostgresDriver/FluentPostgresDatabase.swift:128
0x55e0ea48d50a, protocol witness for PostgresNIO.PostgresDatabase.send(_: PostgresNIO.PostgresRequest, logger: Logging.Logger) -> NIO.EventLoopFuture<()> in conformance FluentPostgresDriver._FluentPostgresDatabase : PostgresNIO.PostgresDatabase in FluentPostgresDriver at /build/<compiler-generated>:127
0x55e0ea74a0d1, (extension in PostgresNIO):PostgresNIO.PostgresDatabase.query(_: Swift.String, _: Swift.Array<PostgresNIO.PostgresData>, onMetadata: (PostgresNIO.PostgresQueryMetadata) -> (), onRow: (PostgresNIO.PostgresRow) throws -> ()) -> NIO.EventLoopFuture<()> at /build/.build/checkouts/postgres-nio/Sources/PostgresNIO/PostgresDatabase+Query.swift:32
0x55e0ea48bce1, FluentPostgresDriver._FluentPostgresDatabase.execute(query: FluentKit.DatabaseQuery, onOutput: (FluentKit.DatabaseOutput) -> ()) -> NIO.EventLoopFuture<()> at /build/.build/checkouts/fluent-postgres-driver/Sources/FluentPostgresDriver/FluentPostgresDatabase.swift:28
0x55e0ea46bd65, FluentKit.QueryBuilder.run((FluentKit.DatabaseOutput) -> ()) -> NIO.EventLoopFuture<()> at /build/.build/checkouts/fluent-kit/Sources/FluentKit/Query/Builder/QueryBuilder.swift:286
0x55e0ea46c07d, FluentKit.QueryBuilder.run() -> NIO.EventLoopFuture<()> at /build/.build/checkouts/fluent-kit/Sources/FluentKit/Query/Builder/QueryBuilder.swift:201
0x55e0ea46c07d, FluentKit.QueryBuilder.create() -> NIO.EventLoopFuture<()> at /build/.build/checkouts/fluent-kit/Sources/FluentKit/Query/Builder/QueryBuilder.swift:96
0x55e0ea441bb3, closure #3 () -> NIO.EventLoopFuture<()> in (extension in FluentKit):Swift.Collection< where A.Element: FluentKit.Model>.create(on: FluentKit.Database) -> NIO.EventLoopFuture<()> at /build/.build/checkouts/fluent-kit/Sources/FluentKit/Model/Model+CRUD.swift:171
0x55e0e9f45801, reabstraction thunk helper from @escaping @callee_guaranteed () -> (@owned NIO.EventLoopFuture<()>) to @escaping @callee_guaranteed (@in_guaranteed ()) -> (@owned NIO.EventLoopFuture<()>) at /build/<compiler-generated>:0
0x55e0e9f45801, function signature specialization <Arg[2] = Owned To Guaranteed> of function signature specialization <Arg[1] = [Closure Propagated : reabstraction thunk helper from @escaping @callee_guaranteed () -> (@owned NIO.EventLoopFuture<()>) to @escaping @callee_guaranteed (@in_guaranteed ()) -> (@owned NIO.EventLoopFuture<()>), Argument Types : [@escaping @callee_guaranteed () -> (@owned NIO.EventLoopFuture<()>)]> of generic specialization <(), ()> of closure #1 () -> NIO.CallbackList in NIO.EventLoopFuture.flatMap<A>(file: Swift.StaticString, line: Swift.UInt, _: (A) -> NIO.EventLoopFuture<A1>) -> NIO.EventLoopFuture<A1> at /build/<compiler-generated>:0
0x55e0e9f45c7a, function signature specialization <Arg[1] = [Closure Propagated : reabstraction thunk helper from @escaping @callee_guaranteed () -> (@owned NIO.EventLoopFuture<()>) to @escaping @callee_guaranteed (@in_guaranteed ()) -> (@owned NIO.EventLoopFuture<()>), Argument Types : [@escaping @callee_guaranteed () -> (@owned NIO.EventLoopFuture<()>)]> of generic specialization <(), ()> of closure #1 () -> NIO.CallbackList in NIO.EventLoopFuture.flatMap<A>(file: Swift.StaticString, line: Swift.UInt, _: (A) -> NIO.EventLoopFuture<A1>) -> NIO.EventLoopFuture<A1> at /build/<compiler-generated>:0
0x55e0e9f45c7a, function signature specialization <Arg[0] = Dead, Arg[1] = Dead, Arg[3] = Owned To Guaranteed> of function signature specialization <Arg[2] = [Closure Propagated : reabstraction thunk helper from @escaping @callee_guaranteed () -> (@owned NIO.EventLoopFuture<()>) to @escaping @callee_guaranteed (@in_guaranteed ()) -> (@owned NIO.EventLoopFuture<()>), Argument Types : [@escaping @callee_guaranteed () -> (@owned NIO.EventLoopFuture<()>)]> of generic specialization <(), ()> of NIO.EventLoopFuture.flatMap<A>(file: Swift.StaticString, line: Swift.UInt, _: (A) -> NIO.EventLoopFuture<A1>) -> NIO.EventLoopFuture<A1> at /build/<compiler-generated>:0
0x55e0ea43ec8a, function signature specialization <Arg[2] = [Closure Propagated : reabstraction thunk helper from @escaping @callee_guaranteed () -> (@owned NIO.EventLoopFuture<()>) to @escaping @callee_guaranteed (@in_guaranteed ()) -> (@owned NIO.EventLoopFuture<()>), Argument Types : [@escaping @callee_guaranteed () -> (@owned NIO.EventLoopFuture<()>)]> of generic specialization <(), ()> of NIO.EventLoopFuture.flatMap<A>(file: Swift.StaticString, line: Swift.UInt, _: (A) -> NIO.EventLoopFuture<A1>) -> NIO.EventLoopFuture<A1> at /build/<compiler-generated>:0
0x55e0ea43ec8a, (extension in FluentKit):Swift.Collection< where A.Element: FluentKit.Model>.create(on: FluentKit.Database) -> NIO.EventLoopFuture<()> at /build/.build/checkouts/fluent-kit/Sources/FluentKit/Model/Model+CRUD.swift:168
0x55e0eabdbb54, closure #1 () -> NIO.EventLoopFuture<()> in closure #6 (Swift.Array<XCMetricsBackendLib.BuildNote>) -> () in XCMetricsBackendLib.PostgreSQLMetricsRepository.(insert in _E018F61C537BC4C1E5E8249F972C0535)(on: FluentKit.Database, buildMetrics: XCMetricsBackendLib.BuildMetrics) -> NIO.EventLoopFuture<()> at /build/Sources/XCMetricsBackendLib/Common/Repositories/Postgress/PostgreSQLMetricsRepository.swift:174
0x55e0eabdbb54, partial apply forwarder for closure #1 () -> NIO.EventLoopFuture<()> in closure #6 (Swift.Array<XCMetricsBackendLib.BuildNote>) -> () in XCMetricsBackendLib.PostgreSQLMetricsRepository.(insert in _E018F61C537BC4C1E5E8249F972C0535)(on: FluentKit.Database, buildMetrics: XCMetricsBackendLib.BuildMetrics) -> NIO.EventLoopFuture<()> at /build/<compiler-generated>:0
0x55e0eabda580, reabstraction thunk helper from @escaping @callee_guaranteed () -> (@owned NIO.EventLoopFuture<()>) to @escaping @callee_guaranteed (@in_guaranteed ()) -> (@owned NIO.EventLoopFuture<()>) at /build/<compiler-generated>:0
0x55e0eabda580, partial apply forwarder for reabstraction thunk helper from @escaping @callee_guaranteed () -> (@owned NIO.EventLoopFuture<()>) to @escaping @callee_guaranteed (@in_guaranteed ()) -> (@owned NIO.EventLoopFuture<()>) at /build/<compiler-generated>:0
0x55e0e9f52d88, generic specialization <(), ()> of closure #1 () -> NIO.CallbackList in NIO.EventLoopFuture.flatMap<A>(file: Swift.StaticString, line: Swift.UInt, _: (A) -> NIO.EventLoopFuture<A1>) -> NIO.EventLoopFuture<A1> at /build/<compiler-generated>:0
0x55e0ea558673, reabstraction thunk helper from @escaping @callee_guaranteed () -> (@owned NIO.CallbackList) to @escaping @callee_guaranteed () -> (@out NIO.CallbackList) at /build/<compiler-generated>:0
0x55e0ea558673, partial apply forwarder for reabstraction thunk helper from @escaping @callee_guaranteed () -> (@owned NIO.CallbackList) to @escaping @callee_guaranteed () -> (@out NIO.CallbackList) at /build/<compiler-generated>:0
0x55e0ea55cf98
0x55e0ea54710a, reabstraction thunk helper from @escaping @callee_guaranteed () -> (@out NIO.CallbackList) to @escaping @callee_guaranteed () -> (@owned NIO.CallbackList) at /build/<compiler-generated>:0
0x55e0ea54710a, NIO.CallbackList._run() -> () at /build/.build/checkouts/swift-nio/Sources/NIO/EventLoopFuture.swift:108
0x55e0e9f43185, generic specialization <()> of NIO.EventLoopPromise._resolve(value: Swift.Result<A, Swift.Error>) -> () at /build/<compiler-generated>:0
0x55e0ea71cf5d, generic specialization <()> of NIO.EventLoopPromise.succeed(A) -> () at /build/<compiler-generated>:0
0x55e0ea71cf5d, PostgresNIO.PostgresRequestHandler.(_channelRead in _EC8377CE6023773E62AFAEB7D7896A11)(context: NIO.ChannelHandlerContext, data: NIO.NIOAny) throws -> () at /build/.build/checkouts/postgres-nio/Sources/PostgresNIO/Connection/PostgresConnection+Database.swift:74
0x55e0ea71cfcc, PostgresNIO.PostgresRequestHandler.channelRead(context: NIO.ChannelHandlerContext, data: NIO.NIOAny) -> () at /build/.build/checkouts/postgres-nio/Sources/PostgresNIO/Connection/PostgresConnection+Database.swift:81
0x55e0ea519805, NIO.ChannelHandlerContext.(invokeChannelRead in _EEC863903996E9F191EBAFEB0FB0DFDD)(NIO.NIOAny) -> () at /build/.build/checkouts/swift-nio/Sources/NIO/ChannelPipeline.swift:1344
0x55e0ea519823, NIO.ChannelHandlerContext.(invokeChannelRead in _EEC863903996E9F191EBAFEB0FB0DFDD)(NIO.NIOAny) -> () at /build/.build/checkouts/swift-nio/Sources/NIO/ChannelPipeline.swift:1346
0x55e0ea515582, NIO.ChannelHandlerContext.fireChannelRead(NIO.NIOAny) -> () at /build/.build/checkouts/swift-nio/Sources/NIO/ChannelPipeline.swift:1157
0x55e0ea748417, PostgresNIO.PostgresMessageDecoder.decode(context: NIO.ChannelHandlerContext, buffer: inout NIO.ByteBuffer) throws -> NIO.DecodingState at /build/.build/checkouts/postgres-nio/Sources/PostgresNIO/Message/PostgresMessageDecoder.swift:56
0x55e0ea7484bf, protocol witness for NIO.ByteToMessageDecoder.decode(context: NIO.ChannelHandlerContext, buffer: inout NIO.ByteBuffer) throws -> NIO.DecodingState in conformance PostgresNIO.PostgresMessageDecoder : NIO.ByteToMessageDecoder in PostgresNIO at /build/<compiler-generated>:0
0x55e0ea5288c6, closure #1 (inout A, inout NIO.ByteBuffer) throws -> NIO.DecodingState in NIO.ByteToMessageHandler.(decodeLoop in _F2A740607FEBA425AF6F8C49A24C0AD7)(context: NIO.ChannelHandlerContext, decodeMode: NIO.ByteToMessageHandler<A>.(DecodeMode in _F2A740607FEBA425AF6F8C49A24C0AD7)) throws -> NIO.(B2MDBuffer in _F2A740607FEBA425AF6F8C49A24C0AD7).BufferProcessingResult at /build/.build/checkouts/swift-nio/Sources/NIO/Codec.swift:567
0x55e0ea528661, NIO.ByteToMessageHandler.(withNextBuffer in _F2A740607FEBA425AF6F8C49A24C0AD7)(allowEmptyBuffer: Swift.Bool, _: (inout A, inout NIO.ByteBuffer) throws -> NIO.DecodingState) throws -> NIO.(B2MDBuffer in _F2A740607FEBA425AF6F8C49A24C0AD7).BufferProcessingResult at /build/.build/checkouts/swift-nio/Sources/NIO/Codec.swift:529
0x55e0ea528661, NIO.ByteToMessageHandler.(decodeLoop in _F2A740607FEBA425AF6F8C49A24C0AD7)(context: NIO.ChannelHandlerContext, decodeMode: NIO.ByteToMessageHandler<A>.(DecodeMode in _F2A740607FEBA425AF6F8C49A24C0AD7)) throws -> NIO.(B2MDBuffer in _F2A740607FEBA425AF6F8C49A24C0AD7).BufferProcessingResult at /build/.build/checkouts/swift-nio/Sources/NIO/Codec.swift:563
0x55e0ea528e86, NIO.ByteToMessageHandler.channelRead(context: NIO.ChannelHandlerContext, data: NIO.NIOAny) -> () at /build/.build/checkouts/swift-nio/Sources/NIO/Codec.swift:636
0x55e0ea519805, NIO.ChannelHandlerContext.(invokeChannelRead in _EEC863903996E9F191EBAFEB0FB0DFDD)(NIO.NIOAny) -> () at /build/.build/checkouts/swift-nio/Sources/NIO/ChannelPipeline.swift:1344
0x55e0ea4f436e, NIO.ChannelPipeline.fireChannelRead0(NIO.NIOAny) -> () at .build/checkouts/swift-nio/Sources/NIO/ChannelPipeline.swift:834
0x55e0ea4f436e, NIO.BaseStreamSocketChannel.readFromSocket() throws -> NIO.BaseSocketChannel<A>.ReadResult at /build/.build/checkouts/swift-nio/Sources/NIO/BaseStreamSocketChannel.swift:107
0x55e0ea5807d3
0x55e0ea5897b3, generic specialization <NIO.Socket> of NIO.BaseSocketChannel.readable() -> () at .build/checkouts/swift-nio/Sources/NIO/BaseSocketChannel.swift:1044
0x55e0ea5897b3, inlined generic function <NIO.Socket> of protocol witness for NIO.SelectableChannel.readable() -> () in conformance NIO.BaseSocketChannel<A> : NIO.SelectableChannel in NIO at /build/<compiler-generated>:1041
0x55e0ea5897b3, function signature specialization <Arg[2] = Dead> of generic specialization <NIO.SocketChannel> of NIO.SelectableEventLoop.handleEvent<A where A: NIO.SelectableChannel>(_: NIO.SelectorEventSet, channel: A) -> () at /build/.build/checkouts/swift-nio/Sources/NIO/SelectableEventLoop.swift:328
0x55e0ea588973, reabstraction thunk helper from @callee_guaranteed (@guaranteed NIO.SelectorEvent<NIO.NIORegistration>) -> (@error @owned Swift.Error) to @escaping @callee_guaranteed (@in_guaranteed NIO.SelectorEvent<NIO.NIORegistration>) -> (@error @owned Swift.Error) at /build/<compiler-generated>:0
0x55e0ea588973, partial apply forwarder for reabstraction thunk helper from @callee_guaranteed (@guaranteed NIO.SelectorEvent<NIO.NIORegistration>) -> (@error @owned Swift.Error) to @escaping @callee_guaranteed (@in_guaranteed NIO.SelectorEvent<NIO.NIORegistration>) -> (@error @owned Swift.Error) at /build/<compiler-generated>:0
0x55e0ea58c51c, NIO.Selector.whenReady(strategy: NIO.SelectorStrategy, _: (NIO.SelectorEvent<A>) throws -> ()) throws -> () at /build/.build/checkouts/swift-nio/Sources/NIO/Selector.swift:637
0x55e0ea5850d9, closure #2 () throws -> () in NIO.SelectableEventLoop.run() throws -> () at /build/.build/checkouts/swift-nio/Sources/NIO/SelectableEventLoop.swift:400
0x55e0ea5850d9, reabstraction thunk helper from @callee_guaranteed () -> (@error @owned Swift.Error) to @escaping @callee_guaranteed () -> (@out (), @error @owned Swift.Error) at /build/<compiler-generated>:0
0x55e0ea5850d9, generic specialization <()> of NIO.withAutoReleasePool<A>(() throws -> A) throws -> A at /build/.build/checkouts/swift-nio/Sources/NIO/SelectableEventLoop.swift:26
0x55e0ea5850d9, NIO.SelectableEventLoop.run() throws -> () at /build/.build/checkouts/swift-nio/Sources/NIO/SelectableEventLoop.swift:399
0x55e0ea53d69b, static NIO.MultiThreadedEventLoopGroup.(runTheLoop in _D5D78C61B22284700B9BD1ACFBC25157)(thread: NIO.NIOThread, canEventLoopBeShutdownIndividually: Swift.Bool, selectorFactory: () throws -> NIO.Selector<NIO.NIORegistration>, initializer: (NIO.NIOThread) -> (), _: (NIO.SelectableEventLoop) -> ()) -> () at /build/.build/checkouts/swift-nio/Sources/NIO/EventLoop.swift:838
0x55e0ea53d69b, closure #1 (NIO.NIOThread) -> () in static NIO.MultiThreadedEventLoopGroup.(setupThreadAndEventLoop in _D5D78C61B22284700B9BD1ACFBC25157)(name: Swift.String, selectorFactory: () throws -> NIO.Selector<NIO.NIORegistration>, initializer: (NIO.NIOThread) -> ()) -> NIO.SelectableEventLoop at /build/.build/checkouts/swift-nio/Sources/NIO/EventLoop.swift:858
0x55e0ea542029, partial apply forwarder for closure #1 (NIO.NIOThread) -> () in static NIO.MultiThreadedEventLoopGroup.(setupThreadAndEventLoop in _D5D78C61B22284700B9BD1ACFBC25157)(name: Swift.String, selectorFactory: () throws -> NIO.Selector<NIO.NIORegistration>, initializer: (NIO.NIOThread) -> ()) -> NIO.SelectableEventLoop at /build/<compiler-generated>:0
0x55e0ea5a3d9e, reabstraction thunk helper from @escaping @callee_guaranteed (@guaranteed NIO.NIOThread) -> () to @escaping @callee_guaranteed (@in_guaranteed NIO.NIOThread) -> (@out ()) at /build/<compiler-generated>:0
0x55e0ea542040, partial apply forwarder for reabstraction thunk helper from @escaping @callee_guaranteed (@guaranteed NIO.NIOThread) -> () to @escaping @callee_guaranteed (@in_guaranteed NIO.NIOThread) -> (@out ()) at /build/<compiler-generated>:0
0x55e0ea5a4d56, closure #1 (Swift.Optional<Swift.UnsafeMutableRawPointer>) -> Swift.Optional<Swift.UnsafeMutableRawPointer> in static NIO.ThreadOpsPosix.run(handle: inout Swift.Optional<Swift.UInt>, args: NIO.Box<(body: (NIO.NIOThread) -> (), name: Swift.Optional<Swift.String>)>, detachThread: Swift.Bool) -> () at /build/.build/checkouts/swift-nio/Sources/NIO/ThreadPosix.swift:105
0x7f9b824846da
0x7f9b8068f71e
0xffffffffffffffff

[FranzBusch]

This might have been caused by not having enough memory. We bumped our instance size up and are monitoring if any more crashes occur.

[ecamacho]

Thank you for keeping us updated. We'll try to replicate it as well.

[FranzBusch]

Some updates here:

Initially we ran XCMetrics including the Postgres DB + Redis on a T2.Micro (1 vCPU, 1 GiB memory). We saw crashes happening very frequently (every 10 minutes). We also saw out of memory errors in the logs and decided to upgrade

We resized the instance to a T2.Medium now (2 vCPU, 4 GiB memory) and see a much stabler performance. The instance crashed once in a 24h hour window. The crash is now a different one which does not look like a memory crash but rather a crash inside the XCLogParser.

0x7f4c95e7697f
0x7f4c96c59270
0x7f4c96c592b1
0x7f4c96bfe124
0x7f4c9699876a
0x7f4c96c04bd5
0x7f4c969f8200
0x7f4c9654d9d4
0x7f4c964b5cfe
0x7f4c964ad4d4
0x7f4c964b8840
0x562cf022a488, function signature specialization <Arg[1] = Dead> of XCLogParser.LogLoader.loadFromURL(Foundation.URL) throws -> Swift.String at /build/.build/checkouts/xclogparser/Sources/XCLogParser/loglocation/LogLoader.swift:29
0x562cf020922c, XCLogParser.LogLoader.loadFromURL(Foundation.URL) throws -> Swift.String at /build/<compiler-generated>:0
0x562cf020922c, XCLogParser.ActivityParser.parseActivityLogInURL(_: Foundation.URL, redacted: Swift.Bool, withoutBuildSpecificInformation: Swift.Bool) throws -> XCLogParser.IDEActivityLog at /build/.build/checkouts/xclogparser/Sources/XCLogParser/activityparser/ActivityParser.swift:50
0x562cf02d0804, function signature specialization <Arg[6] = Dead> of static XCMetricsBackendLib.LogParser.parseFromURL(_: Foundation.URL, machineName: Swift.String, projectName: Swift.String, userId: Swift.String, userIdSHA256: Swift.String, isCI: Swift.Bool) throws -> XCMetricsBackendLib.BuildMetrics at /build/Sources/XCMetricsBackendLib/UploadMetrics/LogProcessing/LogParser.swift:52
0x562cf02d13a5, static XCMetricsBackendLib.LogParser.parseFromURL(_: Foundation.URL, machineName: Swift.String, projectName: Swift.String, userId: Swift.String, userIdSHA256: Swift.String, isCI: Swift.Bool) throws -> XCMetricsBackendLib.BuildMetrics at /build/<compiler-generated>:0
0x562cf02d13a5, function signature specialization <Arg[3] = Dead> of static XCMetricsBackendLib.MetricsProcessor.process(metricsRequest: XCMetricsBackendLib.UploadMetricsRequest, logURL: Foundation.URL, redactUserData: Swift.Bool) throws -> XCMetricsBackendLib.BuildMetrics at /build/Sources/XCMetricsBackendLib/UploadMetrics/LogProcessing/MetricsProcessor.swift:13
0x562cf02c0ef2, static XCMetricsBackendLib.MetricsProcessor.process(metricsRequest: XCMetricsBackendLib.UploadMetricsRequest, logURL: Foundation.URL, redactUserData: Swift.Bool) throws -> XCMetricsBackendLib.BuildMetrics at /build/<compiler-generated>:0
0x562cf02c0ef2, closure #1 () -> () in XCMetricsBackendLib.ProcessMetricsJob.dequeue(Queues.QueueContext, XCMetricsBackendLib.UploadMetricsRequest) -> NIO.EventLoopFuture<()> at /build/Sources/XCMetricsBackendLib/UploadMetrics/Jobs/ProcessMetricsJob.swift:68
0x562cef66ad0d, reabstraction thunk helper from @escaping @callee_guaranteed () -> () to @escaping @callee_unowned @convention(block) () -> () at /build/<compiler-generated>:0
0x7f4c95481406
0x7f4c9548cc28
0x7f4c9548d8cd
0x7f4c95495521
0x7f4c95e6b6da
0x7f4c9407671e
0xffffffffffffffff

Since it only happened once I don't think it is critical right now but we will keep monitoring it.

@ecamacho We have a lot of jobs now that are in the DB marked as running during which XCMetrics crashed. Is there an easy way to re-queue them?

[ecamacho]

Interesting, not currently. We'll take a look, we may add one

[afterxleep]

We started to see this crash quite often as increased the number of people reporting, so It may be resource-related.

[ecamacho]

How big can be your .xcactivitylog file? Also, do you know if your project has a lot of Warnings or Notes? In #30 they are having memory-related issues due to a big number of notes in their logs

[afterxleep]

It probably about 6Mb. As for warnings, about 120+ Warnings. Well bump the specs a bit and keep you posted.

[iRILLLL]

Hi there, I have similar issues with 2mb in size. What's the recommended specs?

I tried with smaller project and it works.