SortableIntField in Solr 5.0.0 is deprecated

[elishowk]

According to this page https://lucene.apache.org/solr/4_10_0/solr-core/org/apache/solr/schema/SortableIntField.html "org.apache.solr.schema.SortableIntField" should be replaced by "org.apache.solr.schema.TrieIntField"

But may be you already know....

• recreate the bug: install Solr 5.0.0, create a new core, and execute manage.py build_solr_schema, then move it to your new core
• go to the admin of your new core http://localhost:8983/solr/ and see the error message saying SortableIntField is deprecated.
• in fact, all the SortableField are deprecated in favor of their equivalent in TrieField.

Versions:

• Python 2.7
• Django 1.4
• Haystack 2.3.1
• Search engine used : Solr 5.0.0 with the latest pysolr

Quoting CHANGES.txt in the 5.0.0 distribution :

* The following legacy numeric and date field types, deprecated in Solr 4.8, are no
  longer supported: BCDIntField, BCDLongField, BCDStrField, IntField, LongField,
  FloatField, DoubleField, SortableIntField, SortableLongField, SortableFloatField,
  SortableDoubleField, and DateField.  Convert these types in your schema to the
  corresponding Trie-based field type and then re-index.  See SOLR-5936 for more
  information.

[acdha]

Looks like this is just a pull request to change the field types in two places:

https://github.com/django-haystack/django-haystack/search?utf8=%E2%9C%93&q=SortableIntField

[elishowk]

I fear this is not only a matter of replacing the field name... I'm unfortunately a beginner about the details of a Solr schema. Also, other problems appear with version 5.0.0, like order_by a Date field which seems broken...

[philippeluickx]

+1
Struggling with DateField not being sortable...

Reason: can not sort on multivalued field: registration_date

[elishowk]

@philippeluickx You should try the pull-request  #1148 that works fine with 5.0.0

[philippeluickx]

Yup, trying it now!

[philippeluickx]

Getting an error here...

Traceback (most recent call last):
File "orava_project/manage.py", line 17, in
execute_from_command_line(sys.argv)
File "/home/orava/.virtualenvs/orava/local/lib/python2.7/site-packages/django/core/management/init.py", line 385, in execute_from_command_line
utility.execute()
File "/home/orava/.virtualenvs/orava/local/lib/python2.7/site-packages/django/core/management/init.py", line 377, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/orava/.virtualenvs/orava/local/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(_args, *_options.dict)
File "/home/orava/.virtualenvs/orava/local/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
output = self.handle(_args, *_options)
File "/home/orava/.virtualenvs/orava/local/lib/python2.7/site-packages/haystack/management/commands/build_solr_schema.py", line 56, in handle
self.log(field, resp, backend)
File "/home/orava/.virtualenvs/orava/local/lib/python2.7/site-packages/haystack/management/commands/build_solr_schema.py", line 98, in log
raise Exception('unable to decode Solr API, are sure you started Solr and created the configured Core (%s) ?' % backend.conn.url)
Exception: unable to decode Solr API, are sure you started Solr and created the configured Core (http://127.0.0.1:8983/solr/orava_core) ?

Could it be because of the # sign? I can access the admin through:
http://127.0.0.1:8983/solr/#/orava_core

[elishowk]

This seems you haven't started or configured your Solr 5.0.0 Core named "orava_core".
The URL without # is good.
I have inadvertently revert changes in the docs that explain how to start Solr 5.0.0 and create a new core before executing manage.py build_solr_schema (no more arguments except --settings there)
Check the last version of docs : https://github.com/Stupeflix/django-haystack/blob/master/docs/installing_search_engines.rst

[philippeluickx]

I checked and double checked.
It is running as it should, core exists and solr service is up.
Not sure if this is a help, but the response I get back.

core missing in the URL? http://127.0.0.1:8983/solr/schema should be http://127.0.0.1:8983/solr/orava_core/schema ?

{'cookies': <[]>, '_content': '\n\n   <title>Status page</title>\n\n\n
Internal Server Error
\n
The server encountered an unexpected condition which prevented it from fulfilling the request
\n
You can get technical details here.
\nPlease continue your visit at our home page.\n
\n\n\n', 'headers': {'date': 'Tue, 03 Mar 2015 20:27:59 GMT', 'accept-ranges': 'bytes', 'content-type': 'text/html; charset=UTF-8', 'content-length': '486'}, 'url': u'http://127.0.0.1:8983/solr/schema', 'status_code': 500, '_content_consumed': True, 'encoding': 'UTF-8', 'request': , 'connection': , 'elapsed': datetime.timedelta(0, 0, 32916), 'raw': , 'reason': 'Server Error', 'history': []}

[elishowk]

Could you try to write a new test about the command build_solr_schema without arguments (= 5.0.0 compatible) in the test suite ? This would help me fix the problem, also this would help the pull request being merged... thanks

[philippeluickx]

Thing is, I am actually still trying to get it to work... By providing the --stdout option I got the xml schema generated, but even with that I am getting DateField being multivalued (which is not)

I also would have no clue on how to write a test for the schema generation, as solr 5 is totally new for me (and I am seriously considering downgrading just to get it work).. sorry!

[philippeluickx]

Another one not working as expected: [Reason: sort param could not be parsed as a query, and is not a field that exists in the index: geodist()]

Having the feeling that haystack is not ready yet for 5.0.0 and needs some more testing before going in production. I will gladly give the support I can, but downgrading for now...

[elishowk]

This expected that with Solr 5, if you copy a schema.xml into the core directory, this will be overwritten by the default managed-schema definition.

"managed-schema" is automatically generated by Solr 5 at the core's creation and you must not edit it, but better use the Schema REST API. This is why I wrote the new build_solr_schema.

read more : https://cwiki.apache.org/confluence/display/solr/Managed+Schema+Definition+in+SolrConfig

Sure, more testing is better, but I think we can work a little more on pull request #1148 and get it done !

[elishowk]

The reason your DateField still being multivalued is that some old schema config is still taken into account by the managed schema of your core. May be you should start over: delete the core, delete the core directory, create the core, and build_solr_schema again.

then check your Schema in the web admin of your core where you have the real configuration taken into account by Solr. Schema.xml is not taken into account by Solr 5 by default

[philippeluickx]

Through the schema browser, the DateField is being marked as Multivalued for Properties and Schema, but not for Index (no idea what that means).

I simply created a second core and I am having the same issues.

Solr is up and running
Found 1 Solr nodes:

Solr process 14332 running on port 8983
{
"solr_home":"/var/solr/data/",
"version":"5.0.0 1659987 - anshumgupta - 2015-02-15 12:26:10",
"startTime":"2015-03-04T07:40:49.99Z",
"uptime":"0 days, 4 hours, 7 minutes, 38 seconds",
"memory":"87 MB (%17.7) of 490.7 MB"}

[elishowk]

Please try this django app and its "manage.py solr" to test solr 5.0.0 deployment
https://github.com/Stupeflix/django-haystack-solr-commands

Before everything, please stop any other Solr instance to avoid conflicts.
Everything about install and configuration is in the README.md
This is what I personally at work now.
I hope it will work now... as I don't understand more than that your issue.

[elishowk]

We should move this discussion to pull-request #1148

[nazariyg]

Most of Solr 5+ issues can be solved by using a custom schema.xml template based on one of the Solr 5's default schemas https://github.com/nazariyg/Solr-5-for-django-haystack

[kinuax]

The steps given by @nazariyg are very useful, thanks.

In my case (django-haystack 2.4.0 and Sol 5.3.0) I built the schema (build_solr_schema command) with the default template and edited the output before pasting to conf directory:
-replace sort.Sortable<type>Field classes with sort.Trie<type>Field
-remove enablePositionIncrements="true" argument in filters with solr.StopFilterFactory class
-remove side="front" argument in filter with solr.EdgeNGramFilterFactory

These Solr classes and arguments are deprecated in previous versions.

[nazariyg]

@asier5 You would still probably need to have the django-haystack's edge_ngram and ngram field types defined in the schema if you ever decide to add auto-completion functionality.

[kinuax]

@nazariyg Thanks for the information. I see that the output of build_solr_schema already defines these two field types.