Solr should normalize ISBNs

[cdrini]

Currently, searches like isbn:"978-84-00-04725-2" yield no results because of the '-'. isbn is defined in the schema.xml file to be of type string, which means they are indexed and queried verbatim. They should probably have their own field type which handles these.

[LeadSongDog]

Duplicates #49 and related to #27 #142

[tfmorris]

There are ISBN and LCCN normalization filters for SOLR here:
https://github.com/mlibrary/umich_solr_library_filters

[LeadSongDog]

@salmman-bhai
This overlaps #49 but doesn't dupe it.

1. Yes we should correct the data type so hyphens won't matter, but
2. also make the isbn more useful in unifying publishers by making them block sequential/sortable
3. Add checks for existing uses of an isbn
   3a. before creating new editions and
   3b. while deduping existing ones
4. Vett that the proposed isbn has an extant external target eg via query at worldcat

[tfmorris]

@LeadSongDog You said both dupe and not dupe at different times. I agree that #49 is a separate issue (data updates vs search).

This issue is very specific - indexing and searching for ISBNs. Let's not extend it to include all things ISBN related. Fixing the original issue, as framed by @cdrini, would provide useful benefit for the users.

[brad2014]

Is the fix going to be to (a) create a proper isbn normalization function (that accommodates isbn's with text in them, etc, as noted in #1194), (b) on setting an isbn, first normalizes it before storing it, and (c) on search, first normalizes it before searching?

@cdrini - I'd like to change the title of this issue to "Properly normalize incoming isbn prior to update or search" and mark #49 as a duplicate. Acceptable?

[cdrini]

The root cause of #49 might be this, but I have no clue what LC Bot is/does, so not certain.

The fix would basically be to update our solr config and do a full reindex, but we don't have a flow for a full reindex at the moment (that's what #1843 is working on). I'll rename it to "Solr should normalize ISBNs".

[hornc]

@cdrini Is still an issue? I didn't explicitly try to fix this issue, but I think some of recent work and refactoring around ISBN lookups, and search improvements to apply isbn conversions consistently may have fixed this.

Now when I do a category ALL search for isbn:978-84-00-04725-2

https://openlibrary.org/search?q=isbn:978-84-00-04725-2

I am redirected straight to https://openlibrary.org/books/OL3849512M/Hombres_y_documentos_de_la_filosofi%CC%81a_espan%CC%83ola which shows that both hyphen stripping and isbn13<->isbn10 conversion is happening on search.

[cdrini]

Oh sweet! That's awesome! Hmmm; so when is the normalization process happening when you query for isbn:978-84-00-04725-2? in work_search?

[hornc]

@cdrini do you agree we can close this issue? I think the "fix" for this is what broke #2623 :)

[cdrini]

https://openlibrary.org/search?q=isbn%3A%22978-84-00-04725-2%22&mode=everything works! Looks done to me.

[cdrini]

(Note technically this should be done on solr directly (not in the python); Solr actually has processors for ISBNs that handle all this stuff. But we can deal with it later; I'm content closing this)