Triton installation not found. #125093
Labels
module: binaries
Anything related to official binaries that we release to users
module: build
Build system issues
oncall: pt2
triaged
This issue has been looked at a team member, and triaged and prioritized into an appropriate module
馃悰 Describe the bug
Using torch.compile with a GNN model on the
20240426
nightly build installed via pip raises Triton installation not found error.Error logs
Traceback (most recent call last):
File "/home/mfbalin/dgl-1/examples/sampling/graphbolt/lightning/../pyg/node_classification_advanced.py", line 446, in
main()
File "/home/mfbalin/dgl-1/examples/sampling/graphbolt/lightning/../pyg/node_classification_advanced.py", line 428, in main
train(train_dataloader, valid_dataloader, num_classes, model, args.device)
File "/home/mfbalin/dgl-1/examples/sampling/graphbolt/lightning/../pyg/node_classification_advanced.py", line 236, in train
out = model(minibatch.sampled_subgraphs, node_features)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 403, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 977, in catch_errors
return callback(frame, cache_entry, hooks, frame_state, skip=1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 411, in _convert_frame_assert
return _compile(
^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_utils_internal.py", line 70, in wrapper_function
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 700, in _compile
guarded_code = compile_inner(code, one_graph, hooks, transform)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/utils.py", line 268, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 568, in compile_inner
out_code = transform_code_object(code, transform)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/bytecode_transformation.py", line 1116, in transform_code_object
transformations(instructions, code_options)
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 173, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 515, in transform
tracer.run()
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 2241, in run
super().run()
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 875, in run
while self.step():
^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 790, in step
self.dispatch_table[inst.opcode](self, inst)
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 2398, in RETURN_VALUE
self._return(inst)
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 2383, in _return
self.output.compile_subgraph(
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1068, in compile_subgraph
self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1285, in compile_and_call_fx_graph
compiled_fn = self.call_user_compiler(gm)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/utils.py", line 268, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1376, in call_user_compiler
raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1357, in call_user_compiler
compiled_fn = compiler_fn(gm, self.example_inputs())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/dynamo/repro/after_dynamo.py", line 127, in debug_wrapper
compiled_gm = compiler_fn(gm, example_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/dynamo/repro/after_dynamo.py", line 127, in debug_wrapper
compiled_gm = compiler_fn(gm, example_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/init.py", line 1742, in call
return compile_fx(model, inputs, config_patches=self.config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1450, in compile_fx
return aot_autograd(
^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/backends/common.py", line 65, in compiler_fn
cg = aot_module_simplified(gm, example_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 958, in aot_module_simplified
compiled_fn = create_aot_dispatcher_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/utils.py", line 268, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 685, in create_aot_dispatcher_function
compiled_fn = compiler_fn(
^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 470, in aot_wrapper_dedupe
return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 672, in aot_wrapper_synthetic_base
return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 434, in aot_dispatch_autograd
compiled_fw_func = aot_config.fw_compiler(fw_module, adjusted_flat_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/utils.py", line 268, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1354, in fw_compiler_base
return inner_compile(
^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/repro/after_aot.py", line 83, in debug_wrapper
inner_compiled_fn = compiler_fn(gm, example_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/debug.py", line 304, in inner
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/utils.py", line 268, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 482, in compile_fx_inner
compiled_graph = fx_codegen_and_compile(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 775, in fx_codegen_and_compile
compiled_fn = graph.compile_to_fn()
^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/graph.py", line 1622, in compile_to_fn
return self.compile_to_module().call
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/utils.py", line 268, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/graph.py", line 1565, in compile_to_module
self.codegen_with_cpp_wrapper() if self.cpp_wrapper else self.codegen()
^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/graph.py", line 1517, in codegen
self.scheduler = Scheduler(self.buffers)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_dynamo/utils.py", line 268, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/scheduler.py", line 1279, in init
self.nodes = [self.create_scheduler_node(n) for n in nodes]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/scheduler.py", line 1371, in create_scheduler_node
return SchedulerNode(self, node)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/scheduler.py", line 700, in init
self._compute_attrs()
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/scheduler.py", line 711, in _compute_attrs
group_fn = self.scheduler.get_backend(self.node.get_device()).group_fn
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/scheduler.py", line 2436, in get_backend
self.backends[device] = self.create_backend(device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mfbalin/.venvs/venv/lib/python3.12/site-packages/torch/_inductor/scheduler.py", line 2428, in create_backend
raise RuntimeError(
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
RuntimeError: Cannot find a working triton installation. More information on installing Triton can be found at https://github.com/openai/triton
Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
You can suppress this exception and fall back to eager by setting:
import torch._dynamo
torch._dynamo.config.suppress_errors = True
Minified repro
python ../pyg/node_classification_advanced.py --torch-compile
Build DGL from source
https://github.com/dmlc/dgl/blob/master/examples/sampling/graphbolt/pyg/node_classification_advanced.py
Versions
Collecting environment information...
PyTorch version: 2.4.0.dev20240426+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Ubuntu 24.04 LTS (x86_64)
GCC version: (Ubuntu 13.2.0-23ubuntu4) 13.2.0
Clang version: Could not collect
CMake version: version 3.28.3
Libc version: glibc-2.39
Python version: 3.12.3 (main, Apr 10 2024, 05:33:47) [GCC 13.2.0] (64-bit runtime)
Python platform: Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.39
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 4090
Nvidia driver version: 552.22
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 9 5950X 16-Core Processor
CPU family: 25
Model: 33
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 1
Stepping: 2
BogoMIPS: 6800.06
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr arat npt nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload umip vaes vpclmulqdq rdpid fsrm
Virtualization: AMD-V
Hypervisor vendor: Microsoft
Virtualization type: full
L1d cache: 512 KiB (16 instances)
L1i cache: 512 KiB (16 instances)
L2 cache: 8 MiB (16 instances)
L3 cache: 32 MiB (1 instance)
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET, no microcode
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] torch==2.4.0.dev20240426+cu121
[pip3] torch_geometric==2.5.3
[pip3] torchaudio==2.2.0.dev20240426+cu121
[pip3] torchdata==0.7.1
[pip3] torchmetrics==1.3.2
[pip3] torchvision==0.19.0.dev20240426+cu121
[conda] Could not collect
cc @seemethere @malfet @osalpekar @atalman @ezyang @msaroufim @bdhirsh @anijain2305 @chauhang
The text was updated successfully, but these errors were encountered: