Mixing NativeQuery with CriteriaQuery and a Criteria with GeoLocation fails

[bernhardkern]

Mixing a NativeQuery and a CriteriaQuery fails, when using Criteria with geoLocation.

Version: Spring Data ElasticSearch 5.2.2
Spring Boot Version: 3.2.2
ElasticSearch Version: 8.12.0

Github, Service demonstrating the issue

Github, Test demonstrating the issue

 val nativeGeoDistance = GeoDistanceQuery.Builder().field("location").distance("10km")            
 .location(GeoLocation.Builder().latlon(LatLonGeoLocation.Builder().lat(pointWeAreLookingFor.lat).lon(pointWeAreLookingFor.lon).build()).build()).build()
        val nq = NativeQueryBuilder().withQuery { q -> q.geoDistance(nativeGeoDistance) }.build()

        return operations.search(nq, WithGeoLocation::class.java)

delivers the expected result -> only one document is found

val cq = CriteriaQuery(Criteria("location").within(pointWeAreLookingFor, "10km"))
        val nq = NativeQueryBuilder().withQuery(cq).build()

        return operations.search(nq, WithGeoLocation::class.java)

fails, both documents are found. It also fails with bounding box queries. Everything that is geo-related.

If I run the critieriaQuery directly, without nesting it in NativeQuery, it works. Sometimes this nesting is necessary, e.g. when using Aggregations, but in general we write the Queries in CriteriaQuery.

Issue: Using a CriteriaQuery as base for the NativeQueryBuilder fails when using a geo based query. Other critiera work (e.g. Term).

[sothawo]

Had a first look at the sample repo; alas there is no explicit Spring Data Elasticsearch configuration, so I cannot just plug in an intercepting proxy to see what's going over the line. I will need to set this up in a configurable way.

[sothawo]

BTW, If you are using Elasticsearch 8.12, be aware that Elasticsearch broke the communication at one place between the client and the server, the problem comes from the server side, so it will happen with a client lib of 8.11 as well, see #2836 and the issues linked there

[sothawo]

Ok, had some time to debug this now.

A CriteriaQuery can have criteria that are defining the query part of a search, and other ones used in the filter part of a search. A geo_location within criteria is one for the filter part. And this currently is only added to the search when using a CriteriaQuery directly, not when it's added to a NativeQuery. And so in your example the whole query is basically ignored resultig inreturning all documents from the index.

So we need to fix that for a CriteriaQuery in a NativeQuery we not only add the query but also the filter part.

[sothawo]

backported to 5.2.x and 5.1.x

[bernhardkern]

Thx for fixing!

[mateuszszewczyk]

So we need to fix that for a CriteriaQuery in a NativeQuery we not only add the query but also the filter part.

The fix indeed works, but only when aggregation is not used, as the filtering added by criteria, for example, criteria.and(new Criteria("addressLocation").within(geoPoint, "15m"));, is added as a post_filter rather than as a filter in the query. This leads to issues because the aggregation is returned for the entire index rather than for the search results.

var criteria = new Criteria("addressCountry").is(country);
        if (StringUtils.isNotEmpty(request.getLat()) && StringUtils.isNotEmpty(request.getLng())) {
            var geoPoint = new GeoPoint(Double.parseDouble(request.getLat()), Double.parseDouble(request.getLng()));
            criteria.and(new Criteria("addressLocation").within(geoPoint, "15m"));
        }

{
    "aggregations": {
        "test_aggregation": {
            "terms": {
                "field": "test",
                "size": 2147483647
            }
        }
    },
    "from": 0,
    "post_filter": {
        "geo_bounding_box": {
            "address_location": {
                "bottom_right": {
                    "lat": 52.1747983647779,
                    "lon": 21.0412053143817
                },
                "top_left": {
                    "lat": 52.2135742278123,
                    "lon": 20.9395817792254
                }
            }
        }
    },
    "query": {
        "bool": {
            "must": [
                {
                    "query_string": {
                        "default_operator": "and",
                        "fields": [
                            "address_country"
                        ],
                        "query": "EN"
                    }
                }
            ]
        }
    }

[sothawo]

I see, that only comes up when using the CriteriaQuery in the native query context. As far as I see filters can be either post filter or be set within a bool query. As a CriteriaQuery always is built as a bool query it might be possible to move the filter part inside the query. Have to check if this breaks anything (#2857)

[bernhardkern]

Hi @sothawo

The fix works for the SearchHits, but does not work for Aggregations. I tested the new code with spring-data-elasticsearch 5.3.0-M2.

I added another test as a showcase in the repository
Github, Service demonstrating the issue

Github, Test demonstrating the issue

it adds this to the NativeQuery:

.withAggregation(
                "exampleAggregation",
                Aggregation.of { it.terms(TermsAggregation.Builder().field("aggregationId").size(20).build()) },
            )

and this to the test:

result shouldHaveSize 3 // **works with spring-data-elasticsearch 5.3.0-M2**

val foundAggregationIds = (result.aggregations as ElasticsearchAggregations)
    .aggregationsAsMap()["exampleAggregation"]
        ?.aggregation()
        ?.aggregate?.lterms()
        ?.buckets()?.array()
        ?.map { it.key() }?.toSet() ?: emptySet()

I persist 4 documents and want to find 3 documents. After the upgrade to 5.3.0-M2, the result contains the expected 3 results, but the aggregation still contains all 4 documents.

I can also create a new ticket, if this would be easier, please tell me what you prefer.
Thx.

[sothawo]

That is what I wrote in my last comment: the geolocation filter is set as a post filter and so it does affect the query but not the aggregation. There is already #2857 to see if this can be changed, so there is no need for a new issue.

[bernhardkern]

Thx for the explanation!