RuntimeError: cuda runtime error (10) : invalid device ordinal at torch/csrc/cuda/Module.cpp:88

[Joge-jiaye]

Traceback (most recent call last):
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connection.py", line 170, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/util/connection.py", line 96, in create_connection
raise err
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/util/connection.py", line 86, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 706, in urlopen
chunked=chunked,
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/http/client.py", line 1107, in request
self._send_request(method, url, body, headers)
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/http/client.py", line 1152, in _send_request
self.endheaders(body)
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/http/client.py", line 1103, in endheaders
self._send_output(message_body)
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/http/client.py", line 934, in _send_output
self.send(msg)
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/http/client.py", line 877, in send
self.connect()
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connection.py", line 200, in connect
conn = self._new_conn()
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connection.py", line 182, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f688e044b70>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/util/retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8081): Max retries exceeded with url: /env/main (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f688e044b70>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/visdom/init.py", line 711, in _send
data=json.dumps(msg),
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/visdom/init.py", line 677, in _handle_post
r = self.session.post(url, data=data)
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/requests/sessions.py", line 590, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8081): Max retries exceeded with url: /env/main (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f688e044b70>: Failed to establish a new connection: [Errno 111] Connection refused',))
[Errno 111] Connection refused
create web directory ./checkpoints/enlightening/web...
THCudaCheck FAIL file=torch/csrc/cuda/Module.cpp line=88 error=10 : invalid device ordinal
Traceback (most recent call last):
File "train.py", line 31, in
model.optimize_parameters(epoch)
File "/home/fjy/work/EnlightenGAN/models/single_model.py", line 394, in optimize_parameters
self.forward()
File "/home/fjy/work/EnlightenGAN/models/single_model.py", line 244, in forward
self.fake_B, self.latent_real_A = self.netG_A.forward(self.real_img, self.real_A_gray)
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 69, in forward
inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 80, in scatter
return scatter_kwargs(inputs, kwargs, device_ids, dim=self.dim)
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/scatter_gather.py", line 38, in scatter_kwargs
inputs = scatter(inputs, target_gpus, dim) if inputs else []
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/scatter_gather.py", line 31, in scatter
return scatter_map(inputs)
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/scatter_gather.py", line 18, in scatter_map
return list(zip(*map(scatter_map, obj)))
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/scatter_gather.py", line 15, in scatter_map
return Scatter.apply(target_gpus, None, dim, obj)
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/_functions.py", line 74, in forward
outputs = comm.scatter(input, ctx.target_gpus, ctx.chunk_sizes, ctx.dim, streams)
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/cuda/comm.py", line 188, in scatter
with torch.cuda.device(device), torch.cuda.stream(stream):
File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/cuda/init.py", line 209, in enter
torch._C._cuda_setDevice(self.idx)
RuntimeError: cuda runtime error (10) : invalid device ordinal at torch/csrc/cuda/Module.cpp:88