Add PatchMatchNet module for MVS and calculation of normals from depth #1129
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…stereo fusion - Normal maps are optional during stereo fusion. If the map is not found a map is estimated from the depth map using the cross-product method. - Confidence maps are now also used in stereo fusion along with a threshold specified in the options. If the confidence probability is below threshold the specific depth is ignored. If the confidence map does not exist, then a default map with probability 1.0 is used. - Introduced alternative calculation of overlapping images in `Model` class based on triangulation score instead of using the median triangulation angle and sorting by number of common points. This calculation can be enabled for fusion with the new `use_triangulation_scoring` option. The new method is what MVSNet and variants are using when processing a ColMap workspace. - Added flag to allow use of calculated normals (from depth maps) in stereo fusion, instead of using the ones estimated from patch-match. - Added flag to control whether or not the normal maps should be renormalized when rescaled. This is completing earlier work that was supposed to avoid normalization of normals during fusion - Corollary to this, we now have to explicitly normalize the normal vectors that are used to calculate the angular difference when filtering points during fusion. - Added utility method `SetSlice` to `Mat` class to allow setting entire entry of normal map more conveniently - Minor cleanup to fix compilation warnings Merged PR 3157: Include confidence maps in the MVS setup - Confidence maps are now part of the MVS setup and participate as dependencies in undistortion and batching - Calculated normal and confidence maps are written out when using a cached workspace to avoid redoing the calculations when a map get evicted from the cache - Minor change in normal map calculation to avoid using pixels with invalid (<=0) depth Merged PR 3356: Robust estimation of normals using planes - Improved calculation of normals from depth map using plane estimation with configurable window around the pixel of interest - Fix bug in normal calculation that was using pixel coordinates instead of local frame coordinates (needed multiplication with the inverse matrix of camera intrinsics) Cherry picked from !3327 Remove option to re-normalize normals
Initial implementation of PatchmatchNet evaluation modules. This is currently a standalone "library" not connected to any other parts of ColMap yet. Merged PR 3299: Add PatchmatchNet processing through TorchScript module - Added PatchMatchNet implementation as alternative to standard patch-match - New functionality controlled by the new option to load a TorchScript module from file `--PatchMatchStereo.mvs_module_path` - Created inheritance structure for `PatchMatch` (base class) and `PatchMatchCuda` and `PatchMatchNet` (derived) to facilitate the choice of processing method Remove LibTorch components Merged PR 3450: Update PatchMatchNet module and interface Fixes two issues from the previous module - Sizes now are handled internally to ensure each dimension is a multiple of 8 - Images are a vector of tensors to allow different sizes between reference and source images
This reverts commit 57113e3.
anmatako
commented
Feb 22, 2021
| #ifndef CUDA_ENABLED | ||
| std::cerr << "ERROR: Dense stereo reconstruction requires CUDA, which is not " | ||
| "available on your system." | ||
| #if !defined(CUDA_ENABLED) && !defined(TORCH_ENABLED) |
There was a problem hiding this comment.
Logic here is changed such that now we fail immediately only if both CUDA and Torch are missing. If either is present we can do patch-match through the existing method or PMNet
Merge main repo to fork
|
I've been trying to compile this but I get the following error: Which libtorch/cuda version are you using? |
@Dawars It seems that LibTorch requires MKL as a dependency even though it already contains the headers and binaries in the LibTorch package itself. See if installing MKL on your system would resolve your issue. On my end I made some modifications in the CMake configurations of LibTorch itself to make things work. I'll see if I can make changes in colmap CMake instead and have things work with vanilla LibTorch. For reference here's a diff between my modified LibTorch and the vanilla one (LibTorch 1.7.1 for CUDA 10.1 with CUDNN 7.6.0) |
|
@Dawars @ahojnnes I update colmap's cmake to set the MKL flags without needing the full dependency for LibTorch to build. Also, I removed an NVCC flag set by LibTorch that was causing issues with Eigen/Core. However I'm not sure what to do with this part of the diff: I'm not sure if the issue with |
|
I'll try it out on Ubuntu and let you know. One problem for me was that
cmake found the mkl.cmake from cgal which was installed on my machine,
maybe we need to supply a custom version with colmap.
…On 2021. Mar 1., Mon at 20:07, Antonios Matakos ***@***.***> wrote:
@Dawars <https://github.com/Dawars> @ahojnnes
<https://github.com/ahojnnes> I update colmap's cmake to set the MKL
flags without needed the full dependency and also removed an NVCC flag set
by LibTorch that was causing issues with Eigen/Core so now it should build
without MKL being present.
However I'm not sure what to do with this part of the diff:
diff --git "a/c:\\Users\\anmatako\\Downloads\\libtorch/include/ATen/Parallel.h" "b/lib\\libtorch/include/ATen/Parallel.h"
index 9e2f9be..cc652f2 100644
--- "a/c:\\Users\\anmatako\\Downloads\\libtorch/include/ATen/Parallel.h"
+++ "b/lib\\libtorch/include/ATen/Parallel.h"
@@ -38,7 +38,7 @@ namespace internal {
// Initialise num_threads lazily at first parallel call
inline CAFFE2_API void lazy_init_num_threads() {
- thread_local bool init = false;
+ static thread_local bool init = false;
if (C10_UNLIKELY(!init)) {
at::init_num_threads();
init = true;
diff --git "a/c:\\Users\\anmatako\\Downloads\\libtorch/include/c10/util/StringUtil.h" "b/lib\\libtorch/include/c10/util/StringUtil.h"
index d2744f1..79da0ae 100644
--- "a/c:\\Users\\anmatako\\Downloads\\libtorch/include/c10/util/StringUtil.h"
+++ "b/lib\\libtorch/include/c10/util/StringUtil.h"
@@ -74,7 +74,7 @@ struct _str_wrapper<const char*> final {
template<>
struct _str_wrapper<> final {
static const std::string& call() {
- thread_local const std::string empty_string_literal;
+ static thread_local const std::string empty_string_literal;
return empty_string_literal;
}
};
I'm not sure if the issue with thread_local having to be static is
specific to MSVC (windows) or if it happens on other platforms as well,
since I have no good way to test this cross-platform.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1129 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAI35ZCP6JH4HYRP37YFET3TBPQYJANCNFSM4YBNICEQ>
.
|
|
Now it compiles and runs fine, no additional cmake config needed for mkl. However the model file seems to be corrupted. I get the following error at: I checked it Python and Netron as well: |
|
I can load the module just fine in C++ and Python 3.8.5 on Windows using Wondering if there's some issue with committing the binary as part of the repo or an issue with Python version. See if it will run with a different python version. Also I can send you the module file directly so we can see if it's an issue caused when the file gets commited.
|
|
@Dawars one more thing you can try in case it's an issue with encodings between windows and Linux would be to pull PatchMatchNet from the tip of my branch here https://github.com/anmatako/PatchmatchNet Then uncomment these 3 lines here: https://github.com/anmatako/PatchmatchNet/blob/e21992b1c2d028536403632eb1bf4bfb1aa8f176/eval.py#L97-L99 and you can run from within the root folder of PatchmatchNet as follows: This will create a new TorchScript module named patchmatchnet-module.pt in your specified output folder. If you can load that module then it must be some conversion issue between OSes. |
|
With PyTorch 1.7.1 I can read the model file properly. I tried explicitly setting the file mode via: std::ifstream model_file(options_.mvs_module_path, std::ios::in | std::ios::binary);
model_[thread_index_] = torch::jit::load(model_file, kDevIn);but I still get the same result. Probably I'll have to compile a debug version of libtorch for linux to get more info. I have little experience with it but I'll try. Here is the stack trace: This is the error I got with PyTorch 1.6 might be related: |
|
Being able to load the module with Pytorch 1.7.1 at least means that the module does not seem to be corrupted. The Pytorch 1.6 issue you see looks like a simple incompatibility with older versions. As for the error you get when you try to load with LibTorch, I'm quite confused as well, as it should not need any special configs in fstream and should load without issues. Can you try with LibTorch 1.7.1 for CUDA 10.1 and cudnn 7.6.0? That's the same package I'm using and I was wondering if there's something in these dependencies that makes the loading incompatible when doing it from colmap.
|
|
I compiled the debug version and it works, which is not very useful. Now
I'm compiling in release mode.
I tried setting up Cuda 10.1 but on Ubuntu cublas is missing for this
version, therefore I used 11.0.
Not sure what to do from here. Also Pytorch 1.8 has just been released.
…On 2021. Mar 2., Tue at 21:20, Antonios Matakos ***@***.***> wrote:
Being able to load the module with Pytorch 1.7.1 at least means that the
module does not seem to be corrupted. The Pytorch 1.6 issue you see looks
like a simple incompatibility with older versions.
As for the error you get when you try to load with LibTorch, I'm quite
confused as well, as it should not need any special configs in fstream and
should load without issues. Can you try with LibTorch 1.7.1 for CUDA 10.1
and cudnn 7.6.0? That's the same package I'm using and I was wondering if
there's something in these dependencies that makes the loading incompatible
when doing it from colmap.
With PyTorch 1.7.1 I can read the model file properly.
I think the problem is that libtorch tries to open the file as a text
file, not binary, that was one of my problems with Python.
I tried explicitly setting the file mode via:
std::ifstream model_file(options_.mvs_module_path, std::ios::in | std::ios::binary);
model_[thread_index_] = torch::jit::load(model_file, kDevIn);
but I still get the same result.
Probably I'll have to compile a debug version of libtorch for linux to get
more info. I have little experience with it but I'll try.
Here is the stack trace:
First definition of patch-match module for thread index: 0
Signal: SIGSEGV (signal SIGSEGV: invalid address (fault address: 0x0))
*** Aborted at 1614714901 (unix time) try "date -d @1614714901" if you are using GNU date ***
PC: @ 0x7f2b751ee986 std::__detail::_Executor<>::_M_dfs()
*** SIGSEGV ***@***.***) received by PID 17575 (TID 0x7f2b22fc4700) from PID 17568; stack trace: ***
@ 0x7f2b84b3a631 (unknown)
@ 0x7f2b8305f3c0 (unknown)
@ 0x7f2b751ee986 std::__detail::_Executor<>::_M_dfs()
@ 0x7f2b751eeb53 std::__detail::_Executor<>::_M_dfs()
@ 0x7f2b751eec6c std::__detail::_Executor<>::_M_dfs()
@ 0x7f2b751ef412 std::__detail::__regex_algo_impl<>()
@ 0x7f2b319995fe c10::Device::Device()
@ 0x7f2b7544963d torch::jit::Unpickler::readInstruction()
@ 0x7f2b7544b540 torch::jit::Unpickler::run()
@ 0x7f2b7544baf1 torch::jit::Unpickler::parse_ivalue()
@ 0x7f2b753ef9c2 torch::jit::readArchiveAndTensors()
@ 0x7f2b753efcdd torch::jit::(anonymous namespace)::ScriptModuleDeserializer::readArchive()
@ 0x7f2b753f2605 torch::jit::(anonymous namespace)::ScriptModuleDeserializer::deserialize()
@ 0x7f2b753f2bd9 torch::jit::load()
@ 0x7f2b753f5455 torch::jit::load()
@ 0x55620f2f4c46 colmap::mvs::PatchMatchNet::InitModule()
@ 0x55620f2f43d6 colmap::mvs::PatchMatchNet::PatchMatchNet()
@ 0x55620ec7c9b0 colmap::mvs::PatchMatchController::ProcessProblem()
@ 0x55620ec8fb63 std::__invoke_impl<>()
@ 0x55620ec8fa50 std::__invoke<>()
@ 0x55620ec8f851 _ZNSt5_BindIFMN6colmap3mvs20PatchMatchControllerEFvRKNS1_17PatchMatchOptionsEmEPS2_S3_mEE6__callIvJEJLm0ELm1ELm2EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
@ 0x55620ec8f367 std::_Bind<>::operator()<>()
@ 0x55620ec8efdd std::__invoke_impl<>()
@ 0x55620ec8ed55 std::__invoke<>()
@ 0x55620ec8ea7d _ZZNSt13__future_base11_Task_stateISt5_BindIFMN6colmap3mvs20PatchMatchControllerEFvRKNS3_17PatchMatchOptionsEmEPS4_S5_mEESaIiEFvvEE6_M_runEvENKUlvE_clEv
@ 0x55620ec8f436 _ZNKSt13__future_base12_Task_setterISt10unique_ptrINS_7_ResultIvEENS_12_Result_base8_DeleterEEZNS_11_Task_stateISt5_BindIFMN6colmap3mvs20PatchMatchControllerEFvRKNSA_17PatchMatchOptionsEmEPSB_SC_mEESaIiEFvvEE6_M_runEvEUlvE_vEclEv
@ 0x55620ec8f08c _ZNSt17_Function_handlerIFSt10unique_ptrINSt13__future_base12_Result_baseENS2_8_DeleterEEvENS1_12_Task_setterIS0_INS1_7_ResultIvEES3_EZNS1_11_Task_stateISt5_BindIFMN6colmap3mvs20PatchMatchControllerEFvRKNSD_17PatchMatchOptionsEmEPSE_SF_mEESaIiEFvvEE6_M_runEvEUlvE_vEEE9_M_invokeERKSt9_Any_data
@ 0x55620eacd258 std::function<>::operator()()
@ 0x55620eacc75e std::__future_base::_State_baseV2::_M_do_set()
@ 0x55620ead4019 std::__invoke_impl<>()
@ 0x55620ead1136 std::__invoke<>()
@ 0x55620eacce3e _ZZSt9call_onceIMNSt13__future_base13_State_baseV2EFvPSt8functionIFSt10unique_ptrINS0_12_Result_baseENS4_8_DeleterEEvEEPbEJPS1_S9_SA_EEvRSt9once_flagOT_DpOT0_ENKUlvE_clEv
Signal: SIGSEGV (unknown crash reason)
Process finished with exit code 11
This is the error I got with PyTorch 1.6 might be related:
with open('/home/dawars/projects/colmap_torch/mvs-modules/patchmatchnet-module_windows.pt', 'br') as f:
model = torch.jit.load(f)
Traceback (most recent call last):
File "/home/dawars/miniconda3/envs/historic/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3417, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-4-d6e3587a7e88>", line 2, in <module>
model = torch.jit.load(f)
File "/home/dawars/miniconda3/envs/historic/lib/python3.7/site-packages/torch/jit/__init__.py", line 277, in load
cpp_module = torch._C.import_ir_module_from_buffer(cu, f.read(), map_location, _extra_files)
RuntimeError:
Arguments for call are not valid.
The following variants are available:
aten::upsample_nearest1d.out(Tensor self, int[1] output_size, float? scales=None, *, Tensor(a!) out) -> (Tensor(a!)):
Expected a value of type 'List[int]' for argument 'output_size' but instead found type 'Optional[List[int]]'.
aten::upsample_nearest1d(Tensor self, int[1] output_size, float? scales=None) -> (Tensor):
Expected a value of type 'List[int]' for argument 'output_size' but instead found type 'Optional[List[int]]'.
The original call is:
File "C:\Users\anmatako\AppData\Roaming\Python\Python38\site-packages\torch\nn\functional.py", line 3130
if input.dim() == 3 and mode == 'nearest':
return torch._C._nn.upsample_nearest1d(input, output_size, scale_factors)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
if input.dim() == 4 and mode == 'nearest':
return torch._C._nn.upsample_nearest2d(input, output_size, scale_factors)
Serialized File "code/__torch__/torch/nn/functional/___torch_mangle_46.py", line 155
_49 = False
if _49:
_51 = torch.upsample_nearest1d(input, output_size3, scale_factors6)
~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
_50 = _51
else:
'interpolate' is being compiled since it was called from 'FeatureNet.forward'
Serialized File "code/__torch__/models/net.py", line 139
def forward(self: __torch__.models.net.FeatureNet,
x: Tensor) -> List[Tensor]:
_35 = __torch__.torch.nn.functional.___torch_mangle_46.interpolate
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
_36 = torch.empty([1], dtype=None, layout=None, device=None, pin_memory=None, memory_format=None)
_37 = torch.empty([1], dtype=None, layout=None, device=None, pin_memory=None, memory_format=None)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1129 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAI35ZCJAMSZSBXYTXHTDHDTBVCAHANCNFSM4YBNICEQ>
.
|
Dawars
reviewed
Mar 5, 2021
Comment on lines
+80
to
+88
| if (model_.count(thread_index_) == 0) { | ||
| std::cout << "First definition of patch-match module for thread index: " | ||
| << options_.gpu_index << std::endl; | ||
| model_[thread_index_] = | ||
| torch::jit::load(options_.mvs_module_path, kDevIn); | ||
| } else { | ||
| std::cout << "Patch-match module already defined for thread index: " | ||
| << options_.gpu_index << std::endl; | ||
| } |
There was a problem hiding this comment.
Only the first run is successful, otherwise the thread terminates at model.forwards(...) (https://github.com/colmap/colmap/pull/1129/files#diff-ec9150c5522870ad0fd07f523905377b4d0670e7f42d92f2bbe11ceb42adb1beR59).
When I load the pytorch module every time this problem doesn't occur
There was a problem hiding this comment.
That's a very interesting failure mode. Failing at the forward() evaluation likely means that the module is not there for the subsequent runs. Are you executing on a single or multi GPU environment? If running on multi-GPU environment, try using just a single GPU index in the patch-match options and see if it fails the same way.
The reason I ended up with this setup instead of loading the module for each problem is to take advantage of the optimizations that LibTorch does in the JIT modules. My main finding was that loading the module every time and no optimization is about 2x slower compared to reusing the module and allow optimizations.
If you can share a dataset that causes this failure I can try to reproduce on my end and see if I can debug it effectively.
There was a problem hiding this comment.
I'm using a single GPU, not loading the model each time makes sense.
I'll send you the dataset.
|
Upvote on the integration of 3rd party learning-based MVS methods. With the recent popularity of colmap amongst the greater CV community, and the advancements in the learning-based SfM & MVS methods, it would be very beneficial for both sides to be able to incorporate methods such as PatchMatchNet, MVSNet, SuperPoint, SuperGlue, etc.. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This work mainly integrated PatchMatchNet functionality in colamp using a TorchScript pre-trained module. Additionally it introduces functionality to calculate normal maps from depth maps since PatchMatchNet evaluation does not create normal maps as part of its process. More details about the changes:
<colmap-root>/lib/thus creating alibtorchsubfolder. Then CMake should be able to find the dependency and set the correct compilation flags.patch_match_stereoby setting themvs_module_pathoption to a valid TorchScript module. One such module is included as part of this PR in<colmap-root>\mvs-modules\patchmatchnet-module.pt(images: List[Tensor], intrisics: Tensor, extrinsics: Tensor, depth_params: Tensor)with the output being a tuple of(depth: Tensor, confidence: Tensor). Thus any module that subscribes to that input/output format for forward evaluation can be used instead.--StereoFusion.calculate_normals.--StereoFusion.use_triangulation_scoring. This is included for parity with PatchMatchNet that has this method for finding related images instead of the colmap default. (useful for comparing results between colmap and Python)