DO NOT MERGE YET Replacing nightly CUDA11.0 builds with 11.1 #47938

janeyx99 · 2020-11-13T22:01:34Z

Based on #43366

Testing CUDA 11.1 build with split torch_cuda (#49050), previously, linking failed due to big binary size.

Using builder PR #627 to integrate the changes.

.circleci/scripts/setup_ci_environment.sh

facebook-github-bot

@janeyx99 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

dr-ci · 2020-11-13T22:50:12Z

💊 CI failures summary and remediations

As of commit bb1c18e (more details on the Dr. CI page):

9/10 failures introduced in this PR
1/10 tentatively recognized as flaky ❄️
- Click here to rerun these jobs

🕵️ 4 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

binary_linux_libtorch_3_7m_cu111_gcc5_4_cxx11-abi_nightly_static-with-deps_build (1/4)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

CMake Error at cmake/public/cuda.cmake:279 (target_link_libraries):


-- Adding OpenMP CXX_FLAGS: -fopenmp
-- Will link against OpenMP libraries: /usr/lib/gcc/x86_64-linux-gnu/5/libgomp.so;/usr/lib/x86_64-linux-gnu/libpthread.so
-- Found CUDA: /usr/local/cuda (found version "11.1") 
-- Caffe2: CUDA detected: 11.1
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 11.1
-- Found CUDNN: /usr/local/cuda/lib64/libcudnn_static.a  
-- Found cuDNN: v8.0.5  (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn_static.a)
CMake Error at cmake/public/cuda.cmake:279 (target_link_libraries):
  Cannot specify link libraries for target "caffe2::cudnn" which is not built
  by this project.
Call Stack (most recent call first):
  cmake/Dependencies.cmake:1148 (include)
  CMakeLists.txt:556 (include)


-- Configuring incomplete, errors occurred!
See also "/pytorch/build/CMakeFiles/CMakeOutput.log".
See also "/pytorch/build/CMakeFiles/CMakeError.log".

binary_linux_libtorch_3_7m_cu111_gcc5_4_cxx11-abi_nightly_static-without-deps_build (2/4)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

CMake Error at cmake/public/cuda.cmake:279 (target_link_libraries):


-- Adding OpenMP CXX_FLAGS: -fopenmp
-- Will link against OpenMP libraries: /usr/lib/gcc/x86_64-linux-gnu/5/libgomp.so;/usr/lib/x86_64-linux-gnu/libpthread.so
-- Found CUDA: /usr/local/cuda (found version "11.1") 
-- Caffe2: CUDA detected: 11.1
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 11.1
-- Found CUDNN: /usr/local/cuda/lib64/libcudnn_static.a  
-- Found cuDNN: v8.0.5  (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn_static.a)
CMake Error at cmake/public/cuda.cmake:279 (target_link_libraries):
  Cannot specify link libraries for target "caffe2::cudnn" which is not built
  by this project.
Call Stack (most recent call first):
  cmake/Dependencies.cmake:1148 (include)
  CMakeLists.txt:556 (include)


-- Configuring incomplete, errors occurred!
See also "/pytorch/build/CMakeFiles/CMakeOutput.log".
See also "/pytorch/build/CMakeFiles/CMakeError.log".

binary_linux_libtorch_3_7m_cu111_gcc5_4_cxx11-abi_nightly_shared-with-deps_build (3/4)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

CMake Error at cmake/public/cuda.cmake:279 (target_link_libraries):


-- Adding OpenMP CXX_FLAGS: -fopenmp
-- Will link against OpenMP libraries: /usr/lib/gcc/x86_64-linux-gnu/5/libgomp.so;/usr/lib/x86_64-linux-gnu/libpthread.so
-- Found CUDA: /usr/local/cuda (found version "11.1") 
-- Caffe2: CUDA detected: 11.1
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 11.1
-- Found CUDNN: /usr/local/cuda/lib64/libcudnn_static.a  
-- Found cuDNN: v8.0.5  (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn_static.a)
CMake Error at cmake/public/cuda.cmake:279 (target_link_libraries):
  Cannot specify link libraries for target "caffe2::cudnn" which is not built
  by this project.
Call Stack (most recent call first):
  cmake/Dependencies.cmake:1148 (include)
  CMakeLists.txt:556 (include)


-- Configuring incomplete, errors occurred!
See also "/pytorch/build/CMakeFiles/CMakeOutput.log".
See also "/pytorch/build/CMakeFiles/CMakeError.log".

binary_linux_libtorch_3_7m_cu111_gcc5_4_cxx11-abi_nightly_shared-without-deps_build (4/4)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

CMake Error at cmake/public/cuda.cmake:279 (target_link_libraries):


-- Adding OpenMP CXX_FLAGS: -fopenmp
-- Will link against OpenMP libraries: /usr/lib/gcc/x86_64-linux-gnu/5/libgomp.so;/usr/lib/x86_64-linux-gnu/libpthread.so
-- Found CUDA: /usr/local/cuda (found version "11.1") 
-- Caffe2: CUDA detected: 11.1
-- Caffe2: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- Caffe2: CUDA toolkit directory: /usr/local/cuda
-- Caffe2: Header version is: 11.1
-- Found CUDNN: /usr/local/cuda/lib64/libcudnn_static.a  
-- Found cuDNN: v8.0.5  (include: /usr/local/cuda/include, library: /usr/local/cuda/lib64/libcudnn_static.a)
CMake Error at cmake/public/cuda.cmake:279 (target_link_libraries):
  Cannot specify link libraries for target "caffe2::cudnn" which is not built
  by this project.
Call Stack (most recent call first):
  cmake/Dependencies.cmake:1148 (include)
  CMakeLists.txt:556 (include)


-- Configuring incomplete, errors occurred!
See also "/pytorch/build/CMakeFiles/CMakeOutput.log".
See also "/pytorch/build/CMakeFiles/CMakeError.log".

5 failures not recognized by patterns:

Job	Step	Action
^{binary_windows_libtorch_3_7_cu111_release_nightly_test}	^Test	🔁 rerun
^{binary_linux_conda_3_9_cu111_devtoolset7_nightly_test}	^{Run in docker}	🔁 rerun
^{binary_linux_conda_3_7_cu111_devtoolset7_nightly_test}	^{Run in docker}	🔁 rerun
^{binary_linux_conda_3_8_cu111_devtoolset7_nightly_test}	^{Run in docker}	🔁 rerun
^{binary_linux_conda_3_6_cu111_devtoolset7_nightly_test}	^{Run in docker}	🔁 rerun

❄️ 1 failure tentatively classified as flaky

but reruns have not yet been triggered to confirm:

binary_windows_libtorch_3_7_cu111_debug_nightly_build (1/1)

Step: "Persisting to workspace" (full log | diagnosis details | 🔁 rerun) ❄️

Error archiving workspace files: Error archiving files to tarball C:\Users\circleci\AppData\Local\Temp\workspace-layer-365ff348-af85-4679-b593-f8e215809d88531392954 : stdout: No space left on device

gzip: /c/Program Files/Git/usr/bin/tar: C:\Users\circleci\AppData\Local\Temp\workspace-layer-365ff348-af85-4679-b593-f8e215809d88531392954: Cannot write: Broken pipe /c/Program Files/Git/usr/bin/tar: Child returned status 1 /c/Program Files/Git/usr/bin/tar: Error is not recoverable: exiting now : exit status 2

Creating workspace archive...


Error archiving workspace files: Error archiving files to tarball C:\Users\circleci\AppData\Local\Temp\workspace-layer-365ff348-af85-4679-b593-f8e215809d88531392954 : stdout: No space left on device
 
 gzip: /c/Program Files/Git/usr/bin/tar: C\:\\Users\\circleci\\AppData\\Local\\Temp\\workspace-layer-365ff348-af85-4679-b593-f8e215809d88531392954: Cannot write: Broken pipe /c/Program Files/Git/usr/bin/tar: Child returned status 1 /c/Program Files/Git/usr/bin/tar: Error is not recoverable: exiting now : exit status 2

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

facebook-github-bot

@janeyx99 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

.circleci/cimodel/data/binary_build_data.py

walterddr

please convert it to a draft if it is intended for testing purposes?

CMakeLists.txt

…cordingly

updating driver link testing only 11.1 binaries prepping for torch_cuda split

peterjc123 · 2021-01-28T02:28:58Z

As for the Windows jobs, the problems are listed below.

"No space left on device". We will need to remove some large directories. Will do later.
"CUDA driver initialization failed, you might not have a CUDA gpu". driver_update.bat in pytorch/builder needs to be updated.
For conda jobs, you'll need to update bld.bat in pytorch/builder.

janeyx99 · 2021-01-28T16:02:10Z

Hey @peterjc123 thanks so much for looking into the Windows side!

"No space left on device". We will need to remove some large directories. Will do later.

Just curious; how and where is this normally done?

"CUDA driver initialization failed, you might not have a CUDA gpu". driver_update.bat in pytorch/builder needs to be updated.

I thought #574 in builder (pytorch/builder#574) updated the driver here...do we need an even newer version?

For conda jobs, you'll need to update bld.bat in pytorch/builder.

I just checked conda/pytorch-nightly/bld.bat and noticed it's been (partially?) updated for CUDA 11.1. I will look into this more but if you have any ideas on what needs to be updated, let me know!

peterjc123 · 2021-01-29T07:04:35Z

"CUDA driver initialization failed, you might not have a CUDA gpu". driver_update.bat in pytorch/builder needs to be updated.

I thought #574 in builder (pytorch/builder#574) updated the driver here...do we need an even newer version?

It should be smoke_test.bat. See pytorch/builder#629 and pytorch/builder#631

"No space left on device". We will need to remove some large directories. Will do later.

Just curious; how and where is this normally done?

We may log on to the CircleCI machine and use sth like Space Sniffer to get those directories.

peterjc123 · 2021-01-30T10:36:35Z

"No space left on device". We will need to remove some large directories. Will do later.

Will be addressed in #51405.

janeyx99 · 2021-02-09T23:11:32Z

This will not be used as we have 11.2

janeyx99 requested a review from seemethere November 13, 2020 22:01

facebook-github-bot added the cla signed label Nov 13, 2020

janeyx99 requested review from malfet, zasdfgbnm and a team November 13, 2020 22:01

janeyx99 commented Nov 13, 2020

View reviewed changes

.circleci/scripts/setup_ci_environment.sh Show resolved Hide resolved

facebook-github-bot reviewed Nov 13, 2020

View reviewed changes

janeyx99 added the ci/all label Nov 13, 2020

facebook-github-bot reviewed Nov 16, 2020

View reviewed changes

janeyx99 force-pushed the nightly-11.1 branch from 017d06c to 782f631 Compare November 16, 2020 16:59

janeyx99 added ci/all and removed ci/all labels Nov 16, 2020

janeyx99 mentioned this pull request Nov 16, 2020

add cuda11.1 to conda/build_pytorch.sh pytorch/builder#582

Merged

janeyx99 commented Nov 16, 2020

View reviewed changes

.circleci/cimodel/data/binary_build_data.py Outdated Show resolved Hide resolved

janeyx99 added ci/all and removed ci/all labels Nov 16, 2020

This was referenced Nov 16, 2020

CUDA 11.1 usues libcudart11.0 for backwards compat pytorch/builder#584

Merged

removing sm_75 from TORCH_ARCH_LIST for 11.1 to enable CI to pass pytorch/builder#589

Merged

janeyx99 force-pushed the nightly-11.1 branch from 8e966ff to 929e976 Compare November 19, 2020 18:56

walterddr reviewed Dec 1, 2020

View reviewed changes

CMakeLists.txt Outdated Show resolved Hide resolved

janeyx99 marked this pull request as draft December 1, 2020 16:11

janeyx99 force-pushed the nightly-11.1 branch 3 times, most recently from a7b7962 to 8ab61b0 Compare December 8, 2020 19:29

janeyx99 added 3 commits January 25, 2021 08:45

Split torch_cuda into torch_cuda_cu and torch_cuda_cpp; update API ac…

437833c

…cordingly

making the split optional

ead24de

reshifting split

a055d3d

janeyx99 added 2 commits January 26, 2021 21:34

moving BatchLinearAlgebra*.cu to cpp since it depends on cusolver

f9f6728

Replacing nightly CUDA11.0 builds with 11.1

9a70348

updating driver link testing only 11.1 binaries prepping for torch_cuda split

janeyx99 force-pushed the nightly-11.1 branch from 8ab61b0 to 9a70348 Compare January 27, 2021 20:22

changing ubuntu image to test tag as latest does not have cuda 11.1

d2acea9

janeyx99 force-pushed the nightly-11.1 branch from f0b4f82 to d2acea9 Compare January 27, 2021 21:16

janeyx99 mentioned this pull request Jan 28, 2021

Add build option to split torch_cuda library into torch_cuda_cu and torch_cuda_cpp #49050

Closed

janeyx99 force-pushed the nightly-11.1 branch from 91a37f0 to 099c6ee Compare January 28, 2021 16:39

Moving BUILD_SPLIT_CUDA=ON from populate_env to binary_*_build.sh

bb1c18e

janeyx99 force-pushed the nightly-11.1 branch from 099c6ee to bb1c18e Compare January 28, 2021 16:56

janeyx99 mentioned this pull request Feb 3, 2021

Enable binaries to be built with split torch_cuda pytorch/builder#641

Merged

janeyx99 closed this Feb 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DO NOT MERGE YET Replacing nightly CUDA11.0 builds with 11.1 #47938

DO NOT MERGE YET Replacing nightly CUDA11.0 builds with 11.1 #47938

janeyx99 commented Nov 13, 2020 •

edited

facebook-github-bot left a comment

dr-ci bot commented Nov 13, 2020 •

edited by facebook-github-bot

facebook-github-bot left a comment

walterddr left a comment

peterjc123 commented Jan 28, 2021 •

edited

janeyx99 commented Jan 28, 2021 •

edited

peterjc123 commented Jan 29, 2021 •

edited

peterjc123 commented Jan 30, 2021

janeyx99 commented Feb 9, 2021

DO NOT MERGE YET Replacing nightly CUDA11.0 builds with 11.1 #47938

DO NOT MERGE YET Replacing nightly CUDA11.0 builds with 11.1 #47938

Conversation

janeyx99 commented Nov 13, 2020 • edited

facebook-github-bot left a comment

Choose a reason for hiding this comment

dr-ci bot commented Nov 13, 2020 • edited by facebook-github-bot

💊 CI failures summary and remediations

🕵️ 4 new failures recognized by patterns

binary_linux_libtorch_3_7m_cu111_gcc5_4_cxx11-abi_nightly_static-with-deps_build (1/4)

binary_linux_libtorch_3_7m_cu111_gcc5_4_cxx11-abi_nightly_static-without-deps_build (2/4)

binary_linux_libtorch_3_7m_cu111_gcc5_4_cxx11-abi_nightly_shared-with-deps_build (3/4)

binary_linux_libtorch_3_7m_cu111_gcc5_4_cxx11-abi_nightly_shared-without-deps_build (4/4)

5 failures not recognized by patterns:

❄️ 1 failure tentatively classified as flaky

binary_windows_libtorch_3_7_cu111_debug_nightly_build (1/1)

facebook-github-bot left a comment

Choose a reason for hiding this comment

walterddr left a comment

Choose a reason for hiding this comment

peterjc123 commented Jan 28, 2021 • edited

janeyx99 commented Jan 28, 2021 • edited

peterjc123 commented Jan 29, 2021 • edited

peterjc123 commented Jan 30, 2021

janeyx99 commented Feb 9, 2021

janeyx99 commented Nov 13, 2020 •

edited

dr-ci bot commented Nov 13, 2020 •

edited by facebook-github-bot

peterjc123 commented Jan 28, 2021 •

edited

janeyx99 commented Jan 28, 2021 •

edited

peterjc123 commented Jan 29, 2021 •

edited