Parallel processing - improved

[dxdc]

Summary

This is an improvement to #959.

Bug fix for faiss

• Minor bug fix: 0bed18f

Simplified/optimized use of omp parallel

• Eliminates the OOM faults from the existing version, and is generally a lot simpler. It seems to be very slightly faster as well.

Unordered maps

• Switch to unordered maps. Since we are running this in parallel, we can achieve faster lookups, and there is no advantage to use ordered maps since we are sorting the final output anyway.

Optimized for loops

• Rather than iterating through the entire loop twice, and checking whether or not it's a degenerate pair within the inner loop, this optimization allows for just a single run through to produce all of the unique pairs.

    for (unsigned int i = 0; i < filenames.size() - 1; i++) {
      for (unsigned int j = i + 1; j < filenames.size(); j++) {
        // comparison here
      }
    }

• The iteration through the TMK arrays was modified to accommodate a more facile parallelization. Essentially, we build a list of filenames like this first. This also allows us to prefill the adjacencyMatrix diagonal and avoid an unneeded extra computation.

  std::vector<std::string> filenames;
  filenames.reserve(metadataToFeatures.size());

  for (const auto &s : metadataToFeatures) {
    // prefill adjacencyMatrix diagonal
    adjacencyMatrix[s.first].insert(s.first);
    filenames.push_back(s.first);
  }

Documentation

• Updated README to add notes about how to build the parallel versions

Test Plan

[Dcallies]

Hmm, this is definitely an argument for more unit testing in the longer term. I'm worried about our team's continued knowledge gap and the ability to verify the outputs.

[Dcallies]

Can you create issues for these?

[dxdc]

Sure. I combined them into a single Windows-based issue: #971

[dxdc]

Hmm, this is definitely an argument for more unit testing in the longer term. I'm worried about our team's continued knowledge gap and the ability to verify the outputs.

Yeah. I have been manually verifying the outputs myself against the standard set of TMK files.