feat(mango): add allow_fallback for user-specified indexes on _find

[pgj]

This is an alternative to #4710 based on the results of the related discussion.

It is not always beneficial for the performance if the Mango query planner tries to assign an index to the selector. User-specified indexes may save the day, but since they are only hints for the planner, fallbacks may still happen.

Introduce the allow_fallback flag which can be used to tell if falling back to other indexes is acceptable when an index is explicitly specified by the user. When set to false, give up on planning and return an HTTP 400 response right away. This way the user has the chance to learn about the requested but missing index, optionally create it and try again.

By default, fallbacks are allowed to maintain backwards compatibility. It is possible to set allow_fallback to true but currently it coincides with the default behavior hence becomes a no-op in practice.

Fixes #4511

Testing recommendations

The changes could be verified by the following commands:

make eunit apps=mango
make mango-test

This is a non-breaking change, so there should be no observable difference in the behavior for the users. When allow_fallback is set to false and use_index is set to some index, it may return HTTP 400 depending on the suitability of the given index.

Checklist

• Code is written and works correctly
• Changes are covered by tests
• Documentation changes were made in the src/docs folder

[nickva]

+1

Nicely done, @pgj!

Perhaps we could get a few additional approvals from anyone involved in the original discussion #4710

[pgj]

Yes, I agree. @rnewson, @willholley, @ricellis please comment if you have the time.

[ricellis]

I was under the impression that allow_fallback was going to apply to all _find requests not just those with use_index. I know @willholley disagreed with that overloading, but I thought that was only related to the cluster toggles. I didn't think there was an objection to disallowing all_docs fallback as long as applied consistently.

[pgj]

I do not want to respond on behalf of @willholley but in my understanding, in his respective comment he opposed to extending the allow_fallback mechanism to _all_docs: "I'm less keen on overloading it to removing the use of the _all_docs index". He also included a historical reference to suggest that it would not give us much: "Mango did originally prevent the _all_docs fallback unless the query contained a clause that would require it. [..] lots of users just blindly added a { _id: { "$gt": 0 } } clause to [..] work around the restriction."

[rnewson]

if allow_fallback: false will allow fallback, then it will not be merged.

[pgj]

@rnewson do you mean that we shall extend this change to disabling the _all_docs fallback as well?

[rnewson]

Yes. Remember this an opt-in in the request body by the user. They are asking us to not fallback; they want an error instead of inefficient execution. I do not understand @willholley's objection to this.

[willholley]

I don't think this issue / PR has anything to do with query efficiency - only fixing the unclear semantics of use_index.

I'm open to a discussion about if we want to / how to prevent use of inefficient indexes or, but that's much broader problem space in which I don't see _all_docs as any different to user-defined indexes.