[BUG] GPU training not working with non-default tabular presets

[Newtoniano]

Bug Report Checklist

• I provided code that demonstrates a minimal reproducible example.
• I confirmed bug exists on the latest mainline of AutoGluon via source install.
• I confirmed bug exists on the latest stable version of AutoGluon.

Describe the bug
Fitting the dataset using num_gpus=1 only works when presets='medium_quality'.
This might have been introduced by the recent ray version bump #3774

In the ray dashboard I see these events, and the tasks are stuck:
Warning: The following resource request cannot be scheduled right now: {'CPU': 12.0, 'GPU': 0.5}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster.

I only have 1 gpu, setting num_gpus to some value <1 seems to at least let the ray tasks run, albeit this doesn’t really work since num_gpus>=1 is expected downstream

From what I can tell this issue is present on google colab too, so it should be easily reproducible.

Expected behavior
GPU training should work even for more complex tabular model presets

To Reproduce
Run the tabular quickstart example notebook with predictor = TabularPredictor(label=label).fit(train_data, num_gpus=1, presets='good_quality') or better presets

Installed Versions
autogluon==1.1.0

[dadlobster]

Same issue for me, ran on WSL2, got one output model and that's it, no CPU/GPU usage after that but did eat up a chunk of ram

[Innixma]

I've reproduced the issue. The problem results in a resource deadlock. I am working on a fix in #4210. The issue has to do with an inconsistency in how ray handles GPU resources compared to CPU resources for nested tasks.