Explicit flush

[balmukundblr]

Description

Please note- This is not a new PR- Original PR (apache/lucene-solr#2349) was raised on old apache/lucene-solr github repository. This is just a copy in new repo.

Longer completion time for Close Index call.

Once AddDoc task completes, Benchmark algo calls ForceMerge/CloseIndex task, which eventually allows all pending flushes to be completed. Since flushes during CloseIndex call are sequential, it takes longer time to complete and delays the overall Index completion time. While indexing 1 million documents with reuters21578 (plain text Document derived from reuters21578 corpus), we observed CloseIndex call takes around 35% of total time.

Solution

Developed a new FlushIndexTask, which uses flushNextBuffer() Lucene API, to flush document at Index thread level, while not impacting any other Index threads. Adding this task in the algo file, immediately after AddDoc task, would ensure flushing all docs before calling ForceMerge/CloseIndex task.
With this solution in place, CloseIndex task time was reduced significantly and it also improved total time for Indexing.

Tests

Since, we are using existing Lucene API - flushNextBuffer(), hence it already has test cases.
-Passed existing tests

Checklist

Please review the following and check all that apply:

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.
• I have given Lucene maintainers access to contribute to my PR branch. (optional but recommended)
• I have developed this patch against the main branch.
• I have run ./gradlew check.
• I have added tests for my changes.

[mikemccand]

Thanks @balmukundblr -- looks great -- I'll try to push today.

[mikemccand]

Thank you @balmukundblr!

[balmukundblr]

Thank you @balmukundblr!

Thank you very much Mike for your great support.