DataLoader hangs with num_workers > 0

[greaber]

The following script reliably causes a deadlock (or perhaps hanging for some other reason) on my machine. Not sure if it is a pytorch bug or a librosa bug.

import sys
from urllib.request import urlopen
from urllib.error import URLError
from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader
import librosa

# I am using the ImageFolder dataset to load audio data.
# To do this, I rename all my WAV files to end in .jpg
# so ImageFolder will think they are images and then
# load them with librosa.  Note: it is important that there
# are no actual image files under ./data!

# The program hangs after printing 
#
#    trying to load ./data/foo/file.jpg
#
# However, if bug_condition is set to False, or if either
# fixes_bug1 or fixes_bug2 is set to True, then it does
# not hang.

bug_condition = True
fixes_bug1 = False
fixes_bug2 = False

if bug_condition:
    try:
        urlopen('http://localhost:8097')
    except URLError:
        pass
    else:
        sys.exit('please retry without a server running on port 8097')

if fixes_bug1:
    # Note: ./data/foo/file.jpg must be a valid WAV file.
    librosa.load('./data/foo/file.jpg')

def audio_loader(filename):
    print('trying to load', filename)
    audio = librosa.load(filename, sr=16000)[0]
    print('loaded', filename)
    return audio

data = ImageFolder(root='./data', loader=audio_loader)
loader = DataLoader(data, num_workers=(0 if fixes_bug2 else 1))

for x, y in loader:
    print('OK')

[greaber]

This code can be made a little smaller by opening an available url:

if bug_condition:
    urlopen('http://www.stackoverflow.com')

[colesbury]

You have a sys.exit call in the top-level of your module

[greaber]

The sys.exit never gets called. The point is that doing this urlopen causes the DataLoader to hang. I encountered it first in a case where the urlopen failed and so I had to catch the exception. But actually it doesn't matter if the urlopen fails or succeeds.

[apaszke]

I ran your script, but it worked correctly. I'm closing this, because it's a duplicate of #1355.

[abdulsalam-bande]

I had the same problem, I downgraded to Python 3.6 and It worked.