[DONT MERGE - JUST RFC] Use hierarchical compilation

[anijain2305]

Just to show how to use hierarchical compilation. On my machine, I see compilation finishing very quickly. Would love for someone from TorchTune to do two things

• Measure the compilation time using the same infra as before to compare against full model compilation
• Compare perf numbers with full model compilation

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/963

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 13 New Failures

As of commit a044ee7 with merge base 30c75d4 ():

NEW FAILURES - The following jobs have failed:

• Lint / lint (3.10) (gh)
  Process completed with exit code 1.
• Multi-GPU Recipe Tests / recipe_test_multi_gpu (3.10, stable) (gh)
  tests/recipes/test_lora_finetune_single_device.py::TestLoRAFinetuneSingleDeviceRecipe::test_loss[True]
• Multi-GPU Recipe Tests / recipe_test_multi_gpu (3.11, stable) (gh)
• Multi-GPU Recipe Tests / recipe_test_multi_gpu (3.8, stable) (gh)
• Multi-GPU Recipe Tests / recipe_test_multi_gpu (3.9, stable) (gh)
  ##[error]The operation was canceled.
• Recipe Tests / recipe_test (3.10) (gh)
  tests/recipes/test_lora_finetune_single_device.py::TestLoRAFinetuneSingleDeviceRecipe::test_loss[True]
• Recipe Tests / recipe_test (3.11) (gh)
  tests/recipes/test_lora_finetune_single_device.py::TestLoRAFinetuneSingleDeviceRecipe::test_loss[True]
• Recipe Tests / recipe_test (3.8) (gh)
  ##[error]The operation was canceled.
• Recipe Tests / recipe_test (3.9) (gh)
  ##[error]The operation was canceled.
• Unit Test / unit_tests (3.10) (gh)
  ##[error]The operation was canceled.
• Unit Test / unit_tests (3.11) (gh)
  tests/torchtune/utils/test_wrap_compile.py::TestWrapCompile::test_wrap_compile
• Unit Test / unit_tests (3.8) (gh)
  ##[error]The operation was canceled.
• Unit Test / unit_tests (3.9) (gh)
  tests/torchtune/utils/test_wrap_compile.py::TestWrapCompile::test_wrap_compile

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[kartikayk]

Thanks for the PR @anijain2305! @ebsmothers and I can help with this. We'll take a look early next week on this