DataLoader workers deadlocked

[esube]

I have this same issue (#804) with two workers and an AWS P2 instance with shared memory of 241G (less than 1% usage). Has anyone run into this issue without docker?

[aa88kk]

I also encountered the same problem. this issue maybe related to #1579, #1355.

[zym1010]

I'm the author of #1355. Now I tried setting pin_memory to False, and seems that problem solved.

I came up with this idea by reading pytorch/examples#56

[fmassa]

I think I have a reproduction:

import torch
import torch.utils.data

class DS(object):
    def __getitem__(self, idx):
        return torch.rand(3, 640, 640)
    def __len__(self):
        return 2000

ds = DS()
it = torch.utils.data.DataLoader(ds, batch_size=500, num_workers=1)

for i, data in enumerate(it):
    print(i)

In my machine, it got stuck at recv_bytes.
Note that the total size of a batch is greater than 2GB in this case, and there might be some limitation of queue to be < 2^31, or that pickle can't handle objects larger than 2^31.

@apaszke do you see a solution for this problem?

[apaszke]

Not really. This pickle limitation shouldn't cause problems here - the data is copied into shared memory and only a very small handle is sent to another process.

[apaszke]

But there might be some more overflows happening inside Python's multiprocessing library 😕 maybe it includes a segment size along with the fd?

[fmassa]

Ok, I just tried running the snippet I sent you into our clusters and it didn't crash as I was expecting. (I came up with that example in my local machine).
This is the stack trace that I got in the server

Traceback (most recent call last):
  File "/home/fmassa/sandbox/conda/envs/devenv/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/home/fmassa/sandbox/conda/envs/devenv/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/fmassa/sandbox/conda/envs/devenv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 38, in _worker_loop
    data_queue.put((idx, samples))
  File "/home/fmassa/sandbox/conda/envs/devenv/lib/python3.6/multiprocessing/queues.py", line 355, in put
    self._writer.send_bytes(obj)
  File "/home/fmassa/sandbox/conda/envs/devenv/lib/python3.6/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/fmassa/sandbox/conda/envs/devenv/lib/python3.6/multiprocessing/connection.py", line 393, in _send_bytes
    header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647

I'll try to come up with a better MWE.

[apaszke]

I also can't reproduce the error on my machine using your snippet. But the stack trace really seems to show that we're trying to send some large blob. That's surprising. Can you try messing around with multiprocessing and see what's going on? (You can modify /home/fmassa/sandbox/conda/envs/devenv/lib/python3.6/multiprocessing/connection.py by adding prints etc.)

[fmassa]

Ok, I now have a repro with the issue I was facing in the cluster.
The problem I was seeing was that I was indeed passing large blobs via numpy arrays (by circumventing the collate_fn to return a np.array). So a MWE that was causing my error message was the following:

import torch.utils.data
import numpy as np

class DS(object):
    def __getitem__(self, idx):
        return np.zeros((3, 640, 480))
    def __len__(self):
        return 8000

ds = DS()
it = torch.utils.data.DataLoader(ds, batch_size=500, num_workers=1, 
                                 collate_fn=lambda x: x)

for i, data in enumerate(it):
    print(i)

This is expected, as numpy arrays are pickled entirely, while torch tensors use shared memory .

And the issue I was having in my local machine with the script I sent earlier was probably insufficient shared memory (I had 4GB, one batch was > 2GB).

Maybe we should write a summary with gotchas of the dataloader? I'll start

• always return a tensor from your collate_fn, and not an np.array.
• If you are facing deadlocks, try increasing the amount of shared memory

Any others? :)

[esube]

The issue on my side most likely is not due to shared memory as I described above and I am using tensors just like the other dataset examples. I still have the deadlock and it happens at random times.

[zym1010]

@esube have you tried turning off pin_memory?

[esube]

@zym1010 actually, by default the pin_memory is turned off as constructor arg. Also, from the discussion you mentioned: pytorch/examples#56 it seems turning it on is useful when you use more than one GPU.

Mind you my dataset is SVHN 32x32 with batch size of 128. I don't think, this is a large batch for the setup: AWS P2 instance with 8 K80s and >400GB mem (241G shared) (physical memory not V) and 32 cores CPU.

[zym1010]

@esube alright. From my experience, pinning or not doesn't make much difference, even for multiple GPUs. But basically, not using pinning completely solves my problem.

[esube]

@zym1010 Thanks for your suggestion. Unfortunately, with or w/out pinning, I have this same hangup issue. The worker hangs up querying the index_queue and it is essentially a deadlock after that as the worker's loop line: r = index_queue.get() is a blocking call.

[zym1010]

@esube Yeah me too for getting stuck at index_queue.get(). There should be some race condition going on.

[apaszke]

I'm going to close it, because it's a duplicate of #1355. Please comment in the original issue.