Lucene search backend #8963
Draft
Lucene search backend #8963
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
A user might have deleted a link to a fulltext-file in the source tex file and we need to reflect that change in the index as well.
btut
commented
Jul 12, 2022
Comment on lines
92
to
97
| HashMap<String, Float> boosts = new HashMap<String, Float>(); | ||
| SearchFieldConstants.searchableBibFields.stream().forEach(field -> boosts.put(field, Float.valueOf(4))); | ||
|
|
||
| if (query.getSearchFlags().contains(SearchRules.SearchFlags.FULLTEXT)) { | ||
| Arrays.stream(SearchFieldConstants.PDF_FIELDS).forEach(field -> boosts.put(field, Float.valueOf(1))); | ||
| } |
There was a problem hiding this comment.
Easy, but ugly. Maybe define a boost property for the standard Fields and get that by Field-name?
|
Very nice to see this implemented! A few comments from my side:
|
Would definitely reduce coding effort and migration code.
Thought about that too, maybe with the option for power-users to show floats instead. I am very bad at UI design though, did not get good feedback whenever I implemented something, so would need some help with that.
Thats true, but filtering kind of defeats the purpose of a fulltext/fuzzy search IMO. Where do you draw the line for the filtering predicate? Score==0? In large libraries that will result in a table with lots of "bad" search results with scores slightly above 0 and people could need to scroll to find much better matches.
In any case, we would need to have a very nice way to display the score if we do not sort on it, so users can still tell better fits apart. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
2 tasks
# Conflicts: # CHANGELOG.md # src/jmh/java/org/jabref/benchmarks/Benchmarks.java # src/main/java/org/jabref/gui/JabRefFrame.java # src/main/java/org/jabref/gui/LibraryTab.java # src/main/java/org/jabref/gui/entryeditor/EntryEditor.java # src/main/java/org/jabref/gui/entryeditor/fileannotationtab/FulltextSearchResultsTab.java # src/main/java/org/jabref/gui/externalfiles/ExternalFilesEntryLinker.java # src/main/java/org/jabref/gui/externalfiles/ImportHandler.java # src/main/java/org/jabref/gui/groups/GroupDialogView.java # src/main/java/org/jabref/gui/groups/GroupsPreferences.java # src/main/java/org/jabref/gui/maintable/MainTable.java # src/main/java/org/jabref/gui/maintable/MainTableColumnFactory.java # src/main/java/org/jabref/gui/maintable/columns/FileColumn.java # src/main/java/org/jabref/gui/preview/PreviewPanel.java # src/main/java/org/jabref/gui/search/GlobalSearchBar.java # src/main/java/org/jabref/gui/search/RebuildFulltextSearchIndexAction.java # src/main/java/org/jabref/gui/search/SearchResultsTableDataModel.java # src/main/java/org/jabref/logic/pdf/search/indexing/IndexingTaskManager.java # src/main/java/org/jabref/model/database/BibDatabaseContext.java # src/main/java/org/jabref/model/pdf/search/SearchFieldConstants.java # src/main/java/org/jabref/model/search/rules/SearchRules.java # src/main/java/org/jabref/preferences/JabRefPreferences.java # src/main/java/org/jabref/preferences/SearchPreferences.java # src/test/java/org/jabref/gui/groups/GroupTreeViewModelTest.java
|
Should compile now, still heavy issues with the performance. Will have to investigate with jdk 21 and virtual threads. |
|
Your code currently does not meet JabRef's code guidelines. The tool reviewdog already placed comments on GitHub to indicate the places. See the tab "Files" in you PR. Please carefully follow the setup guide for the codestyle. Afterwards, please run checkstyle locally and fix the issues. More information on code quality in JabRef is available at https://devdocs.jabref.org/getting-into-the-code/development-strategy.html. |
|
Was not able to compile on Windows 10 with jdk 20. |
|
Used the build from the website and found:
|
|
gradlew clean? |
|
Gradlew clean fixed the compilation, but now I get the following errors whenever I try to open a library: and |
|
hm, interesting, can reproduce the bug on my windows machine, but not on my arch linux laptop |
# Conflicts: # CHANGELOG.md # src/main/java/org/jabref/model/search/rules/ContainsBasedSearchRule.java # src/main/java/org/jabref/model/search/rules/GrammarBasedSearchRule.java # src/main/java/org/jabref/model/search/rules/RegexBasedSearchRule.java
# Conflicts: # CHANGELOG.md # src/main/java/org/jabref/model/search/rules/ContainsBasedSearchRule.java # src/main/java/org/jabref/model/search/rules/GrammarBasedSearchRule.java # src/main/java/org/jabref/model/search/rules/RegexBasedSearchRule.java
|
The build of this PR is available at https://builds.jabref.org/pull/8963/merge. |
|
|
I will just mention this my old comment here as well, so that it is not forgotten: #4237 (comment) I have noted that for the Lucene search approach (#8963 (comment)), the following solution has been suggested to the 'floating mode' problem: I am not familiar with the technicalities of Lucene, but I reckon a score of 0 represents an item, that does not match the search criteria at all. That is not (!) the same as what the old floating mode in JabRef 3.8.2 offered - the items in gray did match the search criteria to a certain degree. The only difference between the items with white background and those with gray background was, that the white ones belonged to the group, that the user had selected at the moment. The gray ones did not (!) belong to the selected group. Hopefully, this old behaviour of JabRef can be restored with the new Lucene search. |
|
Some notes on the search syntax. Users might find it confusing if partial-word-search is not the default search. See laurent22/joplin#6349 and laurent22/joplin#6455 for discussions in a related project. Full search documentation of Joplin: https://joplinapp.org/help/apps/search |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
First draft of the switch to the Lucene Search Backend
Whats working
Todo
Follow-ups
Problems:
Fixes #8857
CHANGELOG.mddescribed in a way that is understandable for the average user (if applicable)