Out of memory on Export? User error?

[DAWA48]

Great tool! Sorry if this is a noob question or something wrong with my system but not sure how to navigate this.

I was able to successfully dump a few channels but having issues running the workplace export itself.

I used this command ./slackdump -export Workspace_export.zip -export-type standard -download

and it seems to be successful but after about 15 minutes each time I get this error:

2023/01/28 23:20:34 messages fetch complete, total: 500365
runtime: out of memory: cannot allocate 4194304-byte block (1727692800 in use)
fatal error: out of memory

goroutine 1 [running]:
runtime.throw({0x1272cb1, 0xd})
/usr/local/go/src/runtime/panic.go:992 +0x64 fp=0x142b0930 sp=0x142b091c pc=0xde56b4
runtime.(*mcache).allocLarge(0x1b40c18, 0x18000, 0x0)
/usr/local/go/src/runtime/mcache.go:215 +0x1d9 fp=0x142b0958 sp=0x142b0930 pc=0xdc42e9
runtime.mallocgc(0x18000, 0x1243500, 0x1)
/usr/local/go/src/runtime/malloc.go:1096 +0x446 fp=0x142b09a0 sp=0x142b0958 pc=0xdbb716
runtime.growslice(0x1243500, {0x7de000, 0x84, 0x84}, 0x85)
/usr/local/go/src/runtime/slice.go:278 +0x413 fp=0x142b09cc sp=0x142b09a0 pc=0xdf95e3
github.com/rusq/slackdump/v2/export.populateMsgs(0x469d4ca0, {0x4ef80000, 0x7a28d, 0x7c633}, 0x142b16c4)
/github/workspace/export/conversation.go:68 +0x201 fp=0x142b0e8c sp=0x142b09cc pc=0x11795a1
github.com/rusq/slackdump/v2/export.Export.byDate({{0x132af14, 0x12496000}, 0x12510600, {0x132bde0, 0x12404480}, {0x132b6f8, 0x12496018}, {{0x0, 0x0, 0x0}, ...}}, ...)
/github/workspace/export/conversation.go:22 +0x51 fp=0x142b0ef4 sp=0x142b0e8c pc=0x11790e1
github.com/rusq/slackdump/v2/export.(*Export).exportConversation(0x12452000, {0x132b918, 0x12496120}, 0x142b16c4, {{{{0x131d7bb0, 0x9}, 0x54da8046, 0x0, {0x0, 0x0}, ...}, ...}, ...})
/github/workspace/export/export.go:201 +0x235 fp=0x142b118c sp=0x142b0ef4 pc=0x117b9d5
github.com/rusq/slackdump/v2/export.(*Export).exclusiveExport.func1({{{{0x131d7bb0, 0x9}, 0x54da8046, 0x0, {0x0, 0x0}, 0x0, 0x0, 0x0, 0x0, ...}, ...}, ...})
/github/workspace/export/export.go:131 +0x22f fp=0x142b1274 sp=0x142b118c pc=0x117ac5f
github.com/rusq/slackdump/v2.(*Session).StreamChannels.func1({0x1342e000, 0x63, 0x8d})
/github/workspace/channels.go:36 +0xa7 fp=0x142b13e4 sp=0x142b1274 pc=0x116fc67
github.com/rusq/slackdump/v2.(*Session).getChannels(0x12510600, {0x132b918, 0x12496138}, {0x15c1100, 0x4, 0x4}, 0x142b152c)
/github/workspace/channels.go:77 +0x320 fp=0x142b1508 sp=0x142b13e4 pc=0x116ffe0
github.com/rusq/slackdump/v2.(*Session).StreamChannels(0x12510600, {0x132b918, 0x12496120}, {0x15c1100, 0x4, 0x4}, 0x1243d5bc)
/github/workspace/channels.go:34 +0x73 fp=0x142b1538 sp=0x142b1508 pc=0x116fb93
github.com/rusq/slackdump/v2/export.(*Export).exclusiveExport(0x12452000, {0x132b918, 0x12496090}, 0x1243d6c4, 0x1247e930)
/github/workspace/export/export.go:125 +0x2c9 fp=0x142b1634 sp=0x142b1538 pc=0x117a849
github.com/rusq/slackdump/v2/export.(*Export).exportChannels(0x12452000, {0x132b918, 0x12496090}, 0x1243d6c4)
/github/workspace/export/export.go:111 +0x9b fp=0x142b1660 sp=0x142b1634 pc=0x117a53b
github.com/rusq/slackdump/v2/export.(*Export).messages(0x12452000, {0x132b918, 0x12496030}, {0x13200000, 0xd80, 0x1094}) /github/workspace/export/export.go:88 +0x27b fp=0x142b1798 sp=0x142b1660 pc=0x117a08b
github.com/rusq/slackdump/v2/export.(*Export).Run(0x12452000, {0x132b918, 0x1247eab0})
/github/workspace/export/export.go:65 +0x1aa fp=0x142b17ec sp=0x142b1798 pc=0x1179cba
github.com/rusq/slackdump/v2/internal/app.Export({0x132b918, 0x1247ea98}, {{0x0, 0x0}, {0x1247e930}, {{0x126ae9d, 0x1}, {0x0, 0x0}, {0x0, ...}}, ...}, ...)
/github/workspace/internal/app/export.go:52 +0x5ed fp=0x142b1a00 sp=0x142b17ec pc=0x119ab5d
github.com/rusq/slackdump/v2/internal/app.Run({0x132ba58, 0x12423800}, {{0x0, 0x0}, {0x1247e930}, {{0x126ae9d, 0x1}, {0x0, 0x0}, {0x0, ...}}, ...}, ...)
/github/workspace/internal/app/app.go:25 +0x21a fp=0x142b1b18 sp=0x142b1a00 pc=0x119781a
main.run({0x132b8d8, 0x1241a11c}, {{{0x0, 0x0}, {0x1247e930}, {{0x126ae9d, 0x1}, {0x0, 0x0}, {0x0, ...}}, ...}, ...})
/github/workspace/cmd/slackdump/main.go:147 +0x47d fp=0x142b1d98 sp=0x142b1b18 pc=0x119d15d
main.main()
/github/workspace/cmd/slackdump/main.go:103 +0x547 fp=0x142b1fc4 sp=0x142b1d98 pc=0x119cb87
runtime.main()
/usr/local/go/src/runtime/proc.go:250 +0x231 fp=0x142b1ff0 sp=0x142b1fc4 pc=0xde8051
runtime.goexit()
/usr/local/go/src/runtime/asm_386.s:1326 +0x1 fp=0x142b1ff4 sp=0x142b1ff0 pc=0xe13011

goroutine 6 [syscall, 14 minutes]:
os/signal.signal_recv()
/usr/local/go/src/runtime/sigqueue.go:151 +0x3b
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x1a
created by os/signal.Notify.func1.1
/usr/local/go/src/os/signal/signal.go:151 +0x27

goroutine 7 [select, 14 minutes]:
os/signal.NotifyContext.func1()
/usr/local/go/src/os/signal/signal.go:288 +0x95
created by os/signal.NotifyContext
/usr/local/go/src/os/signal/signal.go:287 +0x187

goroutine 51 [chan receive, 14 minutes]:
github.com/rusq/slackdump/v2/downloader.(*Client).fltSeen.func1()
/github/workspace/downloader/filters.go:15 +0xce
created by github.com/rusq/slackdump/v2/downloader.(*Client).fltSeen
/github/workspace/downloader/filters.go:7 +0xa2

goroutine 15 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x1237cd0c, 0x72)
/usr/local/go/src/runtime/netpoll.go:302 +0x52
internal/poll.(*pollDesc).wait(0x12467174, 0x72, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x37
internal/poll.execIO(0x12467094, 0x1295ed8)
/usr/local/go/src/internal/poll/fd_windows.go:175 +0xfc
internal/poll.(*FD).Read(0x12467080, {0x14724000, 0x9f5a, 0x9f5a})
/usr/local/go/src/internal/poll/fd_windows.go:441 +0x13b
net.(*netFD).Read(0x12467080, {0x14724000, 0x9f5a, 0x9f5a})
/usr/local/go/src/net/fd_posix.go:55 +0x3f
net.(*conn).Read(0x12406f00, {0x14724000, 0x9f5a, 0x9f5a})
/usr/local/go/src/net/net.go:183 +0x4f
crypto/tls.(*atLeastReader).Read(0x46f2b530, {0x14724000, 0x9f5a, 0x9f5a})
/usr/local/go/src/crypto/tls/conn.go:784 +0x63
bytes.(*Buffer).ReadFrom(0x1256016c, {0x1329a84, 0x46f2b530})
/usr/local/go/src/bytes/buffer.go:204 +0x95
crypto/tls.(*Conn).readFromUntil(0x12560000, {0x1237cda8, 0x12406f00}, 0x5)
/usr/local/go/src/crypto/tls/conn.go:806 +0xdd
crypto/tls.(*Conn).readRecordOrCCS(0x12560000, 0x0)
/usr/local/go/src/crypto/tls/conn.go:613 +0x122
crypto/tls.(*Conn).readRecord(...)
/usr/local/go/src/crypto/tls/conn.go:581
crypto/tls.(*Conn).Read(0x12560000, {0x1257d000, 0x1000, 0x1000})
/usr/local/go/src/crypto/tls/conn.go:1284 +0x147
bufio.(*Reader).Read(0x12578150, {0x125105a4, 0x9, 0x9})
/usr/local/go/src/bufio/bufio.go:236 +0x1ce
io.ReadAtLeast({0x13299a8, 0x12578150}, {0x125105a4, 0x9, 0x9}, 0x9)
/usr/local/go/src/io/io.go:331 +0x97
io.ReadFull(...)
/usr/local/go/src/io/io.go:350
net/http.http2readFrameHeader({0x125105a4, 0x9, 0x9}, {0x13299a8, 0x12578150})
/usr/local/go/src/net/http/h2_bundle.go:1566 +0x6e
net/http.(*http2Framer).ReadFrame(0x12510580)
/usr/local/go/src/net/http/h2_bundle.go:1830 +0x89
net/http.(*http2clientConnReadLoop).run(0x2cf03fd0)
/usr/local/go/src/net/http/h2_bundle.go:8815 +0x15d
net/http.(*http2ClientConn).readLoop(0x124012c0)
/usr/local/go/src/net/http/h2_bundle.go:8711 +0x6e
created by net/http.(*http2Transport).newClientConn
/usr/local/go/src/net/http/h2_bundle.go:7439 +0xaa1

goroutine 20 [select, 14 minutes]:
github.com/rusq/slackdump/v2/downloader.(*Client).worker(0x125821c0, {0x132b918, 0x12496090}, 0x1257b100)
/github/workspace/downloader/downloader.go:186 +0xd5
github.com/rusq/slackdump/v2/downloader.(*Client).startWorkers.func1(0x0)
/github/workspace/downloader/downloader.go:174 +0x68
created by github.com/rusq/slackdump/v2/downloader.(*Client).startWorkers
/github/workspace/downloader/downloader.go:173 +0x54

goroutine 21 [select, 14 minutes]:
github.com/rusq/slackdump/v2/downloader.(*Client).worker(0x125821c0, {0x132b918, 0x12496090}, 0x1257ab80)
/github/workspace/downloader/downloader.go:186 +0xd5
github.com/rusq/slackdump/v2/downloader.(*Client).startWorkers.func1(0x1)
/github/workspace/downloader/downloader.go:174 +0x68
created by github.com/rusq/slackdump/v2/downloader.(*Client).startWorkers
/github/workspace/downloader/downloader.go:173 +0x54

goroutine 22 [select, 14 minutes]:
github.com/rusq/slackdump/v2/downloader.(*Client).worker(0x125821c0, {0x132b918, 0x12496090}, 0x1257b500)
/github/workspace/downloader/downloader.go:186 +0xd5
github.com/rusq/slackdump/v2/downloader.(*Client).startWorkers.func1(0x2)
/github/workspace/downloader/downloader.go:174 +0x68
created by github.com/rusq/slackdump/v2/downloader.(*Client).startWorkers
/github/workspace/downloader/downloader.go:173 +0x54

goroutine 23 [select, 14 minutes]:
github.com/rusq/slackdump/v2/downloader.(*Client).worker(0x125821c0, {0x132b918, 0x12496090}, 0x12416200)
/github/workspace/downloader/downloader.go:186 +0xd5
github.com/rusq/slackdump/v2/downloader.(*Client).startWorkers.func1(0x3)
/github/workspace/downloader/downloader.go:174 +0x68
created by github.com/rusq/slackdump/v2/downloader.(*Client).startWorkers
/github/workspace/downloader/downloader.go:173 +0x54

goroutine 52 [chan receive, 14 minutes]:
github.com/rusq/slackdump/v2/downloader.(*Client).fltSeen.func1()
/github/workspace/downloader/filters.go:15 +0xce
created by github.com/rusq/slackdump/v2/downloader.(*Client).fltSeen
/github/workspace/downloader/filters.go:7 +0xa2

goroutine 53 [chan receive, 14 minutes]:
github.com/rusq/slackdump/v2/downloader.(*Client).fltSeen.func1()
/github/workspace/downloader/filters.go:15 +0xce
created by github.com/rusq/slackdump/v2/downloader.(*Client).fltSeen
/github/workspace/downloader/filters.go:7 +0xa2

goroutine 39 [chan receive, 14 minutes]:
github.com/rusq/slackdump/v2/downloader.(*Client).fltSeen.func1()
/github/workspace/downloader/filters.go:15 +0xce
created by github.com/rusq/slackdump/v2/downloader.(*Client).fltSeen
/github/workspace/downloader/filters.go:7 +0xa2

[rusq]

Hey @DAJA48 thanks for raising this, this is not a user error — it's the programming assumption error :) The reason it runs out of memory is because when fetching channel messages — it caches them in memory before writing to disk, and one of the channels must be too big to fit in the memory available to the program.

It might take some time to implement the solution. The possible options to workaround this would be the following:

1. If you're running it in the VM, allocate more memory to the VM
2. If you can control the swap space, add some swap space.
3. Find a beefier machine for this export.

I'll create an issue based on this discussion, and will update when this is resolved. Sorry for the inconvenience!

Thank you for posting the full callstack - that's very useful!

[DAWA48]

Hey @DAJA48 thanks for raising this, this is not a user error — it's the programming assumption error :) The reason it runs out of memory is because when fetching channel messages — it caches them in memory before writing to disk, and one of the channels must be too big to fit in the memory available to the program.

It might take some time to implement the solution. The possible options to workaround this would be the following:

1. If you're running it in the VM, allocate more memory to the VM
2. If you can control the swap space, add some swap space.
3. Find a beefier machine for this export.

I'll create an issue based on this discussion, and will update when this is resolved. Sorry for the inconvenience!

Thank you for posting the full callstack - that's very useful!

@rusq Thanks for the quick reply!

Just for reference in case it's useful the system I was using has 16GBs ram with 43% utilization at the point of script execution. The system also has an i7-8650U processor.

[rusq]

Issue: #184

[rusq]

Hi @DAJA48,

I think I have fixed the issue (see #185), now slackdump should use at least 19 times less memory for that particular operation that was failing. You can read the details in the pull request, but so that you'd have an understanding — for a (synthetic, generated) message history of 1 mln messages it was using an estimate about 5.7 GB memory for sorting. After optimisation, it uses only 170 MB.

I have prepared a test build, but I'm not sure if you're running on Linux or macOS, so I built both.

Could you please try and rerun the export you were running again, but with this build, and see if this works for you?

slackdump-darwin.zip
slackdump-linux.zip

If this works, I'll release 2.2.9.

[rusq]

@DAJA48 How is it looking? Has it finished successfully?

I'm currently increasing the test coverage for export module, and from what I see, there should not be any problems. On a long export the only thing that can prevent it from completing is when Slack server returns 500 Internal Server Error, which will cause Slackdump to terminate (see #174 ). Otherwise, my hopes are high that everything went well for you.

[DAWA48]

@rusq Unfortunately it did terminate but because of something on my end not the script/application. It appears nothing actually saved to the zip file (guessing the final write/commit operation is at the end) so I've decided to just do batches instead of one big export given the size of the workspace and the time for completion. I presume the optimisation was across all functions? (channel/user dump etc?)

[rusq]

@DAJA48 sorry to hear! The optimisation only affects the transformation of the conversation during the export (input: API raw data, output: Slack Export compatible data). Was the ZIP file completely empty (i.e. size of 0 or 22 bytes)? When Slack Export is being executed in full mode (no specific channels), it does the following:

1. Request a chunk of channels
2. If there are channels in chunk, DO for each channel in the chunk:
   1. Get conversation
   2. Split by date (that's what we optimised)
   3. Write to ZIP or directory
3. Write an index of channels, DMs, groups and users into the output ZIP or Directory
4. Close the ZIP file (this finalises the ZIP file).

If the size was > 22 bytes, then it means that it did not get to steps 3 and 4 of the main. If the size was <= 22 bytes, then it's something new.

Did you know that you can specify the folder name instead of a ZIP file (if that helps). Just run ./slackdump -export some_name (without ZIP extension), and it will create a directory instead of a ZIP file.

[rusq]

Hey @DAJA48 I got good news, the retry on 500-599 was implemented, hopefully it will help you? The full description of the change is in #187.

[DAWA48]

@rusq sorry just seeing your response now and once again I appreciate your engagement.
Yes the zip file was empty after running 23 hours but I feel that didn't happen when I attempted to run 2.27 but I am not sure. I was not aware that I can just use a regular file directory so I will try that next time. Thanks again for all of the assistance - I will let you know if I experience any other issues.