Nightly build flaky pytorch/vision / conda-py3_11-cpu builds

[atalman]

🐛 Describe the bug

Flaky issue on pytorch/vision / conda-py3_11-cpu builds. Has been happening for a while now.
Most likely due to corrupt worker environment:

+ __conda_exe run -p /Users/ec2-user/runner/_work/_temp/pytorch_pkg_helpers_8521283920_smoke python3 pytorch/vision/test/smoke_test.py
+ /opt/homebrew/Caskroom/miniconda/base/bin/conda run -p /Users/ec2-user/runner/_work/_temp/pytorch_pkg_helpers_8521283920_smoke python3 pytorch/vision/test/smoke_test.py
/Users/ec2-user/.local/lib/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: ''If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
  warn(
torchvision: 0.19.0a0+480eec2
Traceback (most recent call last):
  File "/Users/ec2-user/runner/_work/vision/vision/pytorch/vision/test/smoke_test.py", line 103, in <module>
    main()
  File "/Users/ec2-user/runner/_work/vision/vision/pytorch/vision/test/smoke_test.py", line 83, in main
    print(f"{torch.ops.image._jpeg_version() = }")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ec2-user/runner/_work/_temp/pytorch_pkg_helpers_8521283920_smoke/lib/python3.11/site-packages/torch/_ops.py", line 927, in __getattr__
    raise AttributeError(
AttributeError: '_OpNamespace' 'image' object has no attribute '_jpeg_version'

ERROR conda.cli.main_run:execute(124): `conda run python3 pytorch/vision/test/smoke_test.py` failed. (See above for error)
torch.cuda.is_available: False

Rerun is ususally successful

Versions

0.19.0

[atalman]

cc @huydhn @NicolasHug