[Bug]: jaeger-spark-dependency unable to connect to AWS Opensearch 2.5

[tyree731]

What happened?

My company recently deployed Jaeger, using the Elasticsearch storage type, within our Kubernetes cluster using v1.43.0 of jaeger-operator and it's been working great, short of one issue. Our "Elasticsearch" storage is an AWS Opensearch instance running Opensearch 2.5, and while Jaeger itself is more than happy to use this, the jaeger-spark-dependency cronjob is less happy. Attached is our scrubbed configuration file.

We're using the workarounds described here (#668) in order to connect, including both specifying the port, and including the elasticsearchNodesWanOnly configuration. However, jaeger-spark-dependencies consistently fails with the attached exception trace.

So yeah, it can't seem to evaluate this against our Opensearch instance. My best research into this error message landed on this Stackoverflow question (https://stackoverflow.com/questions/73391470/spark-fails-to-read-from-elasticsearch-opensearch-invalid-map-received-dynamic), the solution to which seemingly involved dealing with a misunderstanding from the elasticsearch connector used by the library, but I can't say for sure.

Thanks!

Steps to reproduce

1. Make an AWS Opensearch 2.5 instance, with HTTP access using username and password
2. Point jaeger-operator at that instance
3. Observe the jaeger-spark-dependency cronjob logs

Expected behavior

Jaeger dependencies would be computed and store within the Opensearch instance.

Relevant log output

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/app/jaeger-spark-dependencies-0.0.1-SNAPSHOT.jar) to method java.nio.Bits.unaligned()
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
23/03/31 15:46:47 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/03/31 15:46:48 INFO ElasticsearchDependenciesJob: Running Dependencies job for 2023-03-31T00:00Z, reading from jaeger-span-read index, result storing to jaeger-dependencies-write
Exception in thread "main" org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: invalid map received dynamic_templates=[{span_tags_map={path_match=tag.*, mapping={ignore_above=256, type=keyword}}}, {process_tags_map={path_match=process.tag.*, mapping={ignore_above=256, type=keyword}}}]
        at org.elasticsearch.hadoop.serialization.dto.mapping.FieldParser.parseField(FieldParser.java:165)
        at org.elasticsearch.hadoop.serialization.dto.mapping.FieldParser.parseMapping(FieldParser.java:107)
        at org.elasticsearch.hadoop.serialization.dto.mapping.FieldParser.parseIndexMappings(FieldParser.java:88)
        at org.elasticsearch.hadoop.serialization.dto.mapping.FieldParser.parseMappings(FieldParser.java:59)
        at org.elasticsearch.hadoop.rest.RestClient.getMappings(RestClient.java:321)
        at org.elasticsearch.hadoop.rest.RestClient.getMappings(RestClient.java:307)
        at org.elasticsearch.hadoop.rest.RestRepository.getMappings(RestRepository.java:293)
        at org.elasticsearch.hadoop.rest.RestService.findPartitions(RestService.java:252)
        at org.elasticsearch.spark.rdd.AbstractEsRDD.esPartitions$lzycompute(AbstractEsRDD.scala:79)
        at org.elasticsearch.spark.rdd.AbstractEsRDD.esPartitions(AbstractEsRDD.scala:78)
        at org.elasticsearch.spark.rdd.AbstractEsRDD.getPartitions(AbstractEsRDD.scala:48)
        at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:273)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:269)
        at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
        at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:273)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:269)
        at org.apache.spark.Partitioner$.$anonfun$defaultPartitioner$4(Partitioner.scala:78)
        at org.apache.spark.Partitioner$.$anonfun$defaultPartitioner$4$adapted(Partitioner.scala:78)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
        at scala.collection.immutable.List.foreach(List.scala:392)
        at scala.collection.TraversableLike.map(TraversableLike.scala:238)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
        at scala.collection.immutable.List.map(List.scala:298)
        at org.apache.spark.Partitioner$.defaultPartitioner(Partitioner.scala:78)
        at org.apache.spark.rdd.RDD.$anonfun$groupBy$1(RDD.scala:714)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:385)
        at org.apache.spark.rdd.RDD.groupBy(RDD.scala:714)
        at org.apache.spark.api.java.JavaRDDLike.groupBy(JavaRDDLike.scala:243)
        at org.apache.spark.api.java.JavaRDDLike.groupBy$(JavaRDDLike.scala:239)
        at org.apache.spark.api.java.AbstractJavaRDDLike.groupBy(JavaRDDLike.scala:45)
        at io.jaegertracing.spark.dependencies.elastic.ElasticsearchDependenciesJob.run(ElasticsearchDependenciesJob.java:273)
        at io.jaegertracing.spark.dependencies.elastic.ElasticsearchDependenciesJob.run(ElasticsearchDependenciesJob.java:249)
        at io.jaegertracing.spark.dependencies.DependenciesSparkJob.run(DependenciesSparkJob.java:54)
        at io.jaegertracing.spark.dependencies.DependenciesSparkJob.main(DependenciesSparkJob.java:40)

Screenshot

No response

Additional context

No response

Jaeger backend version

No response

SDK

No response

Pipeline

No response

Stogage backend

AWS Opensearch 2.5

Operating system

Linux

Deployment model

Kubernetes

Deployment configs

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: <internal name is here>
spec:
  strategy: production
  ingress:
    enabled: false
  storage:
    type: elasticsearch
    options:
      es:
        use-aliases: true
        server-urls: <URL is here>
    secretName: <elasticsearch secret goes here>
    esRollover:
      conditions: "{\"max_age\": \"2d\"}"          # conditions when to rollover to a new index
      readTTL: 168h                                # how long should be old data available for reading (7 days)
      schedule: "55 23 * * *"                      # cron expression for it to run
    dependencies:
      elasticsearchNodesWanOnly: true