EarlyOutOfMemoryException when backup is trying to run Server Wide backup

[petarvu]

We configured a server wide backup-task, but it fails with EarlyOutOfMemoryException:

Destination: Azure

RavenDB version: 5.4.1

System.AggregateException: One or more errors occurred. (The amount of available memory to commit on the system is low. Commit charge: 39.875 GBytes / 39.993 GBytes, Memory: 31.905 GBytes / 31.999 GBytes, Available memory for processing: 96.06 MBytes, Dirty memory: 0 Bytes, Managed memory: 12.632 GBytes, Unmanaged allocations: 20.037 GBytes, Top unmanaged allocations: name: .NET ThreadPool Worker, allocations: 408.86 MBytes, name: .NET ThreadPool Worker, allocations: 372.54 MBytes, name: .NET ThreadPool Worker, allocations: 261.68 MBytes, name: .NET ThreadPool Worker, allocations: 224.23 MBytes, name: .NET ThreadPool Worker, allocations: 179.38 MBytes)
 ---> Sparrow.LowMemory.EarlyOutOfMemoryException: The amount of available memory to commit on the system is low. Commit charge: 39.875 GBytes / 39.993 GBytes, Memory: 31.905 GBytes / 31.999 GBytes, Available memory for processing: 96.06 MBytes, Dirty memory: 0 Bytes, Managed memory: 12.632 GBytes, Unmanaged allocations: 20.037 GBytes, Top unmanaged allocations: name: .NET ThreadPool Worker, allocations: 408.86 MBytes, name: .NET ThreadPool Worker, allocations: 372.54 MBytes, name: .NET ThreadPool Worker, allocations: 261.68 MBytes, name: .NET ThreadPool Worker, allocations: 224.23 MBytes, name: .NET ThreadPool Worker, allocations: 179.38 MBytes
   at Sparrow.LowMemory.MemoryInformation.ThrowInsufficientMemory(MemoryInfoResult memInfo) in C:\Builds\RavenDB-Stable-5.4\54001\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 195
   at Sparrow.LowMemory.MemoryInformation.AssertNotAboutToRunOutOfMemory() in C:\Builds\RavenDB-Stable-5.4\54001\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 135
   at Sparrow.Utils.NativeMemory.AllocateMemory(Int64 size, ThreadStats& thread) in C:\Builds\RavenDB-Stable-5.4\54001\src\Sparrow\Utils\NativeMemory.cs:line 153
   at Sparrow.Server.UnmanagedGlobalSegment..ctor(Int32 size) in C:\Builds\RavenDB-Stable-5.4\54001\src\Sparrow.Server\ByteString.cs:line 466
   at Sparrow.Server.ByteStringMemoryCache.Allocate(Int32 size, Action allocationFailure) in C:\Builds\RavenDB-Stable-5.4\54001\src\Sparrow.Server\ByteString.cs:line 605
   at Sparrow.Server.ByteStringContext`1.AllocateSegment(Int32 size) in C:\Builds\RavenDB-Stable-5.4\54001\src\Sparrow.Server\ByteString.cs:line 1269
   at Sparrow.Server.ByteStringContext`1.AllocateInternalUnlikely(Int32 length, Int32 allocationUnit, ByteStringType type) in C:\Builds\RavenDB-Stable-5.4\54001\src\Sparrow.Server\ByteString.cs:line 1045
   at Sparrow.Server.ByteStringContext`1.AllocateInternal(Int32 length, ByteStringType type) in C:\Builds\RavenDB-Stable-5.4\54001\src\Sparrow.Server\ByteString.cs:line 960
   at Voron.Data.Tables.Table.DecompressValue(Transaction tx, Byte* ptr, Int32 size, ByteString& buffer) in C:\Builds\RavenDB-Stable-5.4\54001\src\Voron\Data\Tables\Table.cs:line 277
   at Voron.Data.Tables.Table.DirectReadDecompress(Int64 id, Byte* directRead, Int32& size) in C:\Builds\RavenDB-Stable-5.4\54001\src\Voron\Data\Tables\Table.cs:line 260
   at Voron.Data.Tables.Table.DirectRead(Int64 id, Int32& size) in C:\Builds\RavenDB-Stable-5.4\54001\src\Voron\Data\Tables\Table.cs:line 233
   at Voron.Data.Tables.Table.GetTableValueReader(IFixedSizeIterator it, TableValueReader& reader) in C:\Builds\RavenDB-Stable-5.4\54001\src\Voron\Data\Tables\Table.cs:line 1651
   at Voron.Data.Tables.Table.SeekForwardFrom(FixedSizeSchemaIndexDef index, Int64 key, Int64 skip)+MoveNext() in C:\Builds\RavenDB-Stable-5.4\54001\src\Voron\Data\Tables\Table.cs:line 1565
   at Raven.Server.Documents.DocumentsStorage.GetDocumentsFrom(DocumentsOperationContext context, Int64 etag, Int64 start, Int64 take, DocumentFields fields)+MoveNext() in C:\Builds\RavenDB-Stable-5.4\54001\src\Raven.Server\Documents\DocumentsStorage.cs:line 913
   at Raven.Server.Utils.Enumerators.PulsedTransactionEnumerator`2.MoveNext() in C:\Builds\RavenDB-Stable-5.4\54001\src\Raven.Server\Utils\Enumerators\PulsedTransactionEnumerator.cs:line 57
   at Raven.Server.Smuggler.Documents.DatabaseSource.GetDocumentsAsync(List`1 collectionsToExport, INewDocumentActions actions)+MoveNext() in C:\Builds\RavenDB-Stable-5.4\54001\src\Raven.Server\Smuggler\Documents\DatabaseSource.cs:line 182
   at Raven.Server.Smuggler.Documents.DatabaseSource.GetDocumentsAsync(List`1 collectionsToExport, INewDocumentActions actions)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at Raven.Server.Smuggler.Documents.DatabaseSmuggler.ProcessDocumentsAsync(SmugglerResult result, BuildVersionType buildType) in C:\Builds\RavenDB-Stable-5.4\54001\src\Raven.Server\Smuggler\Documents\DatabaseSmuggler.cs:line 680
   at Raven.Server.Smuggler.Documents.DatabaseSmuggler.ProcessDocumentsAsync(SmugglerResult result, BuildVersionType buildType) in C:\Builds\RavenDB-Stable-5.4\54001\src\Raven.Server\Smuggler\Documents\DatabaseSmuggler.cs:line 680
   at Raven.Server.Smuggler.Documents.DatabaseSmuggler.ProcessDocumentsAsync(SmugglerResult result, BuildVersionType buildType) in C:\Builds\RavenDB-Stable-5.4\54001\src\Raven.Server\Smuggler\Documents\DatabaseSmuggler.cs:line 755
   at Raven.Server.Smuggler.Documents.DatabaseSmuggler.ProcessTypeAsync(DatabaseItemType type, SmugglerResult result, BuildVersionType buildType, Boolean ensureStepsProcessed) in C:\Builds\RavenDB-Stable-5.4\54001\src\Raven.Server\Smuggler\Documents\DatabaseSmuggler.cs:line 287
   at Raven.Server.Smuggler.Documents.DatabaseSmuggler.ExecuteAsync(Boolean ensureStepsProcessed, Boolean isLastFile) in C:\Builds\RavenDB-Stable-5.4\54001\src\Raven.Server\Smuggler\Documents\DatabaseSmuggler.cs:line 94
   at Raven.Server.Smuggler.Documents.DatabaseSmuggler.ExecuteAsync(Boolean ensureStepsProcessed, Boolean isLastFile) in C:\Builds\RavenDB-Stable-5.4\54001\src\Raven.Server\Smuggler\Documents\DatabaseSmuggler.cs:line 104
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Raven.Server.Documents.PeriodicBackup.BackupTask.CreateBackup(DatabaseSmugglerOptionsServerSide options, String backupFilePath, Nullable`1 startDocumentEtag, Nullable`1 startRaftIndex) in C:\Builds\RavenDB-Stable-5.4\54001\src\Raven.Server\Documents\PeriodicBackup\BackupTask.cs:line 784
   at Raven.Server.Documents.PeriodicBackup.BackupTask.CreateLocalBackupOrSnapshot(PeriodicBackupStatus status, String backupFilePath, Nullable`1 startEtag, Nullable`1 startRaftIndex) in C:\Builds\RavenDB-Stable-5.4\54001\src\Raven.Server\Documents\PeriodicBackup\BackupTask.cs:line 653
   at Raven.Server.Documents.PeriodicBackup.BackupTask.RunPeriodicBackup(Action`1 onProgress, PeriodicBackupStatus& runningBackupStatus) in C:\Builds\RavenDB-Stable-5.4\54001\src\Raven.Server\Documents\PeriodicBackup\BackupTask.cs:line 269

[ayende]

You are using a pretty old build. First thing to try it to upgrade to 5.4.111 and see what the status of the operation is

[petarvu]

Thanks for the follow up. Upgraded to 5.4.111, and got following:

[19:48:47 INFO] Read 21,000 counters.
[19:53:38 INFO] Saving the backup status in the cluster
The amount of available memory to commit on the system is low. Commit charge: 39.869 GBytes / 39.986 GBytes, Memory: 31.798 GBytes / 31.999 GBytes, Available memory for processing: 206.22 MBytes, Dirty memory: 0 Bytes, Managed memory: 17.834 GBytes, Unmanaged allocations: 12.556 GBytes, Lucene managed: 140 MBytes, Lucene unmanaged: 12.02 MBytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 1.212 GBytes, managed thread id: 18, [#2] name: 'Integrera' Transaction Merging Thread, allocations: 110.74 MBytes, managed thread id: 41, [#3] name: .NET ThreadPool Worker, allocations: 72.21 MBytes, managed thread id: 80, [#4] name: .NET ThreadPool Worker, allocations: 16.42 MBytes, managed thread id: 33, [#5] name: .NET ThreadPool Worker, allocations: 11.96 MBytes, managed thread id: 77
Sparrow.LowMemory.EarlyOutOfMemoryException: The amount of available memory to commit on the system is low. Commit charge: 39.869 GBytes / 39.986 GBytes, Memory: 31.798 GBytes / 31.999 GBytes, Available memory for processing: 206.22 MBytes, Dirty memory: 0 Bytes, Managed memory: 17.834 GBytes, Unmanaged allocations: 12.556 GBytes, Lucene managed: 140 MBytes, Lucene unmanaged: 12.02 MBytes, Top unmanaged allocations: [#1] name: .NET ThreadPool Worker, allocations: 1.212 GBytes, managed thread id: 18, [#2] name: 'Integrera' Transaction Merging Thread, allocations: 110.74 MBytes, managed thread id: 41, [#3] name: .NET ThreadPool Worker, allocations: 72.21 MBytes, managed thread id: 80, [#4] name: .NET ThreadPool Worker, allocations: 16.42 MBytes, managed thread id: 33, [#5] name: .NET ThreadPool Worker, allocations: 11.96 MBytes, managed thread id: 77
   at Sparrow.LowMemory.MemoryInformation.ThrowInsufficientMemory(MemoryInfoResult memInfo) in C:\Builds\RavenDB-Stable-5.4\54083\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 192
   at Sparrow.LowMemory.MemoryInformation.AssertNotAboutToRunOutOfMemory() in C:\Builds\RavenDB-Stable-5.4\54083\src\Sparrow.Server\LowMemory\MemoryInformation.cs:line 132
   at Sparrow.Utils.NativeMemory.AllocateMemory(Int64 size, ThreadStats& thread) in C:\Builds\RavenDB-Stable-5.4\54083\src\Sparrow\Utils\NativeMemory.cs:line 156
   at Sparrow.Json.ArenaMemoryAllocator.GrowArena(Int32 requestedSize) in C:\Builds\RavenDB-Stable-5.4\54083\src\Sparrow\Json\ArenaMemoryAllocator.cs:line 222
   at Sparrow.Json.JsonOperationContext.GetLazyString(StringSegment field, Boolean longLived) in C:\Builds\RavenDB-Stable-5.4\54083\src\Sparrow\Json\JsonOperationContext.cs:line 484
   at Sparrow.Json.JsonOperationContext.GetLazyStringForFieldWithCachingUnlikely(StringSegment key) in C:\Builds\RavenDB-Stable-5.4\54083\src\Sparrow\Json\JsonOperationContext.cs:line 460
   at Sparrow.Json.CachedProperties.UnlikelyGetProperty(LazyStringValue propName) in C:\Builds\RavenDB-Stable-5.4\54083\src\Sparrow\Json\CachedProperties.cs:line 208
   at Sparrow.Json.BlittableJsonDocumentBuilder.ReadInternal[TWriteStrategy]() in C:\Builds\RavenDB-Stable-5.4\54083\src\Sparrow\Json\BlittableJsonDocumentBuilder.cs:line 118
   at Sparrow.Json.JsonOperationContext.ReadObjectInternal(Object builder, String documentId, UsageMode mode, IBlittableDocumentModifier modifier) in C:\Builds\RavenDB-Stable-5.4\54083\src\Sparrow\Json\JsonOperationContext.cs:line 577
   at Raven.Server.Documents.CountersStorage.ConvertFromBlobToNumbers(JsonOperationContext context, CounterGroupDetail counterGroupDetail) in C:\Builds\RavenDB-Stable-5.4\54083\src\Raven.Server\Documents\CountersStorage.cs:line 2301
   at Raven.Server.Smuggler.Documents.StreamDestination.StreamCounterActions.WriteCounterAsync(CounterGroupDetail counterDetail) in C:\Builds\RavenDB-Stable-5.4\54083\src\Raven.Server\Smuggler\Documents\StreamDestination.cs:line 1044
   at Raven.Server.Smuggler.Documents.DatabaseSmuggler.ProcessCountersAsync(SmugglerResult result) in C:\Builds\RavenDB-Stable-5.4\54083\src\Raven.Server\Smuggler\Documents\DatabaseSmuggler.cs:line 939
   at Raven.Server.Smuggler.Documents.DatabaseSmuggler.ProcessCountersAsync(SmugglerResult result) in C:\Builds\RavenDB-Stable-5.4\54083\src\Raven.Server\Smuggler\Documents\DatabaseSmuggler.cs:line 939
   at Raven.Server.Smuggler.Documents.DatabaseSmuggler.ProcessCountersAsync(SmugglerResult result) in C:\Builds\RavenDB-Stable-5.4\54083\src\Raven.Server\Smuggler\Documents\DatabaseSmuggler.cs:line 939
   at Raven.Server.Smuggler.Documents.DatabaseSmuggler.ProcessTypeAsync(DatabaseItemType type, SmugglerResult result, BuildVersionType buildType, Boolean ensureStepsProcessed) in C:\Builds\RavenDB-Stable-5.4\54083\src\Raven.Server\Smuggler\Documents\DatabaseSmuggler.cs:line 246
   at Raven.Server.Smuggler.Documents.DatabaseSmuggler.ExecuteAsync(Boolean ensureStepsProcessed, Boolean isLastFile) in C:\Builds\RavenDB-Stable-5.4\54083\src\Raven.Server\Smuggler\Documents\DatabaseSmuggler.cs:line 99
   at Raven.Server.Smuggler.Documents.DatabaseSmuggler.ExecuteAsync(Boolean ensureStepsProcessed, Boolean isLastFile) in C:\Builds\RavenDB-Stable-5.4\54083\src\Raven.Server\Smuggler\Documents\DatabaseSmuggler.cs:line 109

[ayende]

Can you send the debug package for this database?
What is the total size in GB? What is the number of documents?
Do you have a lot of counters?

If you try to do an export, does this work? If you try to do an export without counters, does this work?

This does looks like an issue on our end, but need more data

[petarvu]

Tried to manually export without counters and it failed again with the same issue, this time on Revisions. It stopped on 86,000 revisions. It was even worse this time when doing it manually, memory went to top and required restart of the ravendb service to function.

Total size: 443GB

Documents: 16,035,000
Counters: 4,360,975
Revisions: 114,000

Where can I send the debug package?

[arekpalinski]

Please send it to support@ravendb.net

[petarvu]

Please send it to support@ravendb.net

Done :)

[petarvu]

Any news regarding this issue?

[karmeli87]

Hi,

We've opened an issue to investigate this, you can track it here https://issues.hibernatingrhinos.com/issue/RavenDB-21671

[aviv86]

Hi @petarvu
We weren't able to figure this out yet.
Can you send us some more information in order to further investigate this issue?
I left you a comment here with more details:
https://issues.hibernatingrhinos.com/issue/RavenDB-21671