App not working on GPU RTX

[aniketmaurya]

Trace:

[root.serve_work_0] 2022-09-26T06:11:03.859Z INFO: loading model...
[root.serve_work_0] 2022-09-26T06:11:03.899Z INFO: Downloading weights...
[root.serve_work_0] 2022-09-26T06:11:49.966Z {'trained_betas'} was not found in config. Values will be initialized to default values.
[root.serve_work_0] 2022-09-26T06:12:07.920Z /home/zeus/.local/lib/python3.8/site-packages/torch/cuda/__init__.py:146: UserWarning: 
[root.serve_work_0] 2022-09-26T06:12:07.920Z NVIDIA A10G with CUDA capability sm_86 is not compatible with the current PyTorch installation.
[root.serve_work_0] 2022-09-26T06:12:07.920Z The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
[root.serve_work_0] 2022-09-26T06:12:07.920Z If you want to use the NVIDIA A10G GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
[root.serve_work_0] 2022-09-26T06:12:07.920Z   warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
[root.serve_work_0] 2022-09-26T06:12:07.920Z INFO: model loaded
[root.serve_work_0] 2022-09-26T06:12:07.946Z INFO:     Started server process [6]
[root.serve_work_0] 2022-09-26T06:12:07.946Z INFO:     Waiting for application startup.
[root.serve_work_0] 2022-09-26T06:12:07.946Z INFO:     Application startup complete.
[root.serve_work_0] 2022-09-26T06:12:07.947Z INFO:     Uvicorn running on http://0.0.0.0:49029/ (Press CTRL+C to quit)
[root.serve_work_0] 2022-09-26T06:20:23.929Z INFO: request: batch=[Data(dream='Cats in hats', high_quality=False)]
[root.serve_work_0] 2022-09-26T06:20:23.939Z INFO: Traceback (most recent call last):
[root.serve_work_0] 2022-09-26T06:20:23.939Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/uvicorn/protocols/http/httptools_impl.py", line 404, in run_asgi
[root.serve_work_0] 2022-09-26T06:20:23.939Z     result = await app(  # type: ignore[func-returns-value]
[root.serve_work_0] 2022-09-26T06:20:23.939Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
[root.serve_work_0] 2022-09-26T06:20:23.939Z     return await self.app(scope, receive, send)
[root.serve_work_0] 2022-09-26T06:20:23.939Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/fastapi/applications.py", line 270, in __call__
[root.serve_work_0] 2022-09-26T06:20:23.939Z     await super().__call__(scope, receive, send)
[root.serve_work_0] 2022-09-26T06:20:23.939Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/starlette/applications.py", line 124, in __call__
[root.serve_work_0] 2022-09-26T06:20:23.939Z     await self.middleware_stack(scope, receive, send)
[root.serve_work_0] 2022-09-26T06:20:23.939Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 184, in __call__
[root.serve_work_0] 2022-09-26T06:20:23.939Z     raise exc
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 162, in __call__
[root.serve_work_0] 2022-09-26T06:20:23.940Z     await self.app(scope, receive, _send)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/starlette/middleware/cors.py", line 84, in __call__
[root.serve_work_0] 2022-09-26T06:20:23.940Z     await self.app(scope, receive, send)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 75, in __call__
[root.serve_work_0] 2022-09-26T06:20:23.940Z     raise exc
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 64, in __call__
[root.serve_work_0] 2022-09-26T06:20:23.940Z     await self.app(scope, receive, sender)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
[root.serve_work_0] 2022-09-26T06:20:23.940Z     raise e
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
[root.serve_work_0] 2022-09-26T06:20:23.940Z     await self.app(scope, receive, send)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/starlette/routing.py", line 680, in __call__
[root.serve_work_0] 2022-09-26T06:20:23.940Z     await route.handle(scope, receive, send)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/starlette/routing.py", line 275, in handle
[root.serve_work_0] 2022-09-26T06:20:23.940Z     await self.app(scope, receive, send)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/starlette/routing.py", line 65, in app
[root.serve_work_0] 2022-09-26T06:20:23.940Z     response = await func(request)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/fastapi/routing.py", line 231, in app
[root.serve_work_0] 2022-09-26T06:20:23.940Z     raw_response = await run_endpoint_function(
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/fastapi/routing.py", line 162, in run_endpoint_function
[root.serve_work_0] 2022-09-26T06:20:23.940Z     return await run_in_threadpool(dependant.call, **values)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
[root.serve_work_0] 2022-09-26T06:20:23.940Z     return await anyio.to_thread.run_sync(func, *args)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
[root.serve_work_0] 2022-09-26T06:20:23.940Z     return await get_asynclib().run_sync_in_worker_thread(
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
[root.serve_work_0] 2022-09-26T06:20:23.940Z     return await future
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
[root.serve_work_0] 2022-09-26T06:20:23.940Z     result = context.run(func, *args)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/dream/components/stable_diffusion_serve.py", line 161, in predict_api
[root.serve_work_0] 2022-09-26T06:20:23.940Z     result = app.POOL.submit(
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 444, in result
[root.serve_work_0] 2022-09-26T06:20:23.940Z     return self.__get_result()
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
[root.serve_work_0] 2022-09-26T06:20:23.940Z     raise self._exception
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
[root.serve_work_0] 2022-09-26T06:20:23.940Z     result = self.fn(*self.args, **self.kwargs)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
[root.serve_work_0] 2022-09-26T06:20:23.940Z     return func(*args, **kwargs)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/dream/components/stable_diffusion_serve.py", line 93, in predict
[root.serve_work_0] 2022-09-26T06:20:23.940Z     preds = self._model(
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
[root.serve_work_0] 2022-09-26T06:20:23.940Z     return func(*args, **kwargs)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 182, in __call__
[root.serve_work_0] 2022-09-26T06:20:23.940Z     text_embeddings = self.text_encoder(text_input.input_ids.to(self.device))[0]
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
[root.serve_work_0] 2022-09-26T06:20:23.940Z     return forward_call(*input, **kwargs)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py", line 728, in forward
[root.serve_work_0] 2022-09-26T06:20:23.940Z     return self.text_model(
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
[root.serve_work_0] 2022-09-26T06:20:23.940Z     return forward_call(*input, **kwargs)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py", line 636, in forward
[root.serve_work_0] 2022-09-26T06:20:23.940Z     hidden_states = self.embeddings(input_ids=input_ids, position_ids=position_ids)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
[root.serve_work_0] 2022-09-26T06:20:23.940Z     return forward_call(*input, **kwargs)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/transformers/models/clip/modeling_clip.py", line 165, in forward
[root.serve_work_0] 2022-09-26T06:20:23.940Z     inputs_embeds = self.token_embedding(input_ids)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
[root.serve_work_0] 2022-09-26T06:20:23.940Z     return forward_call(*input, **kwargs)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 158, in forward
[root.serve_work_0] 2022-09-26T06:20:23.940Z     return F.embedding(
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO:   File "/home/zeus/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 2199, in embedding
[root.serve_work_0] 2022-09-26T06:20:23.940Z     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
[root.serve_work_0] 2022-09-26T06:20:23.940Z INFO: RuntimeError: CUDA error: no kernel image is available for execution on the device
[root.serve_work_0] 2022-09-26T06:20:23.941Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
[root.serve_work_0] 2022-09-26T06:20:23.941Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.