[Bug]: Jaeger spark job failing to generate dependecy graph with opensearch 2.x as storage backend #129
Labels
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened?
jaeger-spark job pod for dependency graph generation, failing with CrashLoopBackOff error. Looks like the job using search_type as scan, which is deprecated now. We are using opensearch2.0 as storage backend. The valid values are query_then_fetch, dfs_query_then_fetch.
Steps to reproduce
Expected behavior
spark job should run to completion and we should be able to see the dependency graph on Jaeger UI.
Relevant log output
Screenshot
No response
Additional context
No response
Jaeger backend version
jaeger-collector:1.37.0
SDK
opentelemetry-javaagent-all
Pipeline
OTEL java SDK --> otel-collector --> jaeger-collector --> opensearch 2.x
Stogage backend
opensearch 2.x
Operating system
centos 7
Deployment model
kubernetes bare metal
Deployment configs
USER-SUPPLIED VALUES: USER-SUPPLIED VALUES: null agent: affinity: {} annotations: {} cmdlineParams: {} daemonset: updateStrategy: {} useHostPort: false dnsPolicy: ClusterFirst enabled: false extraConfigmapMounts: [] extraEnv: [] extraSecretMounts: [] image: jaegertracing/jaeger-agent imagePullSecrets: [] nodeSelector: {} podAnnotations: {} podLabels: {} podSecurityContext: {} priorityClassName: "" pullPolicy: IfNotPresent resources: {} securityContext: {} service: annotations: {} binaryPort: 6832 compactPort: 6831 loadBalancerSourceRanges: [] samplingPort: 5778 type: ClusterIP zipkinThriftPort: 5775 serviceAccount: annotations: {} automountServiceAccountToken: false create: false name: null serviceMonitor: additionalLabels: {} enabled: false tolerations: [] useHostNetwork: false cassandra: config: cluster_name: jaeger dc_name: dc1 endpoint_snitch: GossipingPropertyFileSnitch rack_name: rack1 seed_size: 1 persistence: enabled: false collector: affinity: {} annotations: {} autoscaling: enabled: false maxReplicas: 10 minReplicas: 2 cmdlineParams: {} dnsPolicy: ClusterFirst enabled: true extraConfigmapMounts: [] extraSecretMounts: [] image: jaegertracing/jaeger-collector imagePullSecrets: [] ingress: annotations: {} enabled: false nodeSelector: {} podAnnotations: {} podLabels: {} podSecurityContext: {} priorityClassName: "" pullPolicy: IfNotPresent replicaCount: 1 resources: {} securityContext: {} service: annotations: {} grpc: port: 14250 http: port: 14268 loadBalancerSourceRanges: [] type: ClusterIP zipkin: {} serviceAccount: annotations: {} automountServiceAccountToken: false create: true name: null serviceMonitor: additionalLabels: {} enabled: false tolerations: [] elasticsearch: {} esIndexCleaner: affinity: {} annotations: {} cmdlineParams: {} concurrencyPolicy: Forbid enabled: false extraConfigmapMounts: [] extraEnv: [] extraSecretMounts: [] failedJobsHistoryLimit: 3 image: jaegertracing/jaeger-es-index-cleaner imagePullSecrets: [] nodeSelector: {} numberOfDays: 7 podAnnotations: {} podLabels: {} podSecurityContext: runAsUser: 1000 pullPolicy: Always resources: {} schedule: 55 23 * * * securityContext: runAsUser: 1000 serviceAccount: automountServiceAccountToken: false create: true name: null successfulJobsHistoryLimit: 3 tag: latest tolerations: [] esLookback: affinity: {} annotations: {} cmdlineParams: {} concurrencyPolicy: Forbid enabled: false extraConfigmapMounts: [] extraEnv: - name: UNIT value: days - name: UNIT_COUNT value: "7" extraSecretMounts: [] failedJobsHistoryLimit: 3 image: jaegertracing/jaeger-es-rollover imagePullSecrets: [] nodeSelector: {} podAnnotations: {} podLabels: {} podSecurityContext: runAsUser: 1000 pullPolicy: Always resources: {} schedule: 5 0 * * * securityContext: {} serviceAccount: automountServiceAccountToken: false create: true name: null successfulJobsHistoryLimit: 3 tag: latest tolerations: [] esRollover: affinity: {} annotations: {} cmdlineParams: {} concurrencyPolicy: Forbid enabled: false extraConfigmapMounts: [] extraEnv: - name: CONDITIONS value: '{"max_age": "1d"}' extraSecretMounts: [] failedJobsHistoryLimit: 3 image: jaegertracing/jaeger-es-rollover imagePullSecrets: [] initHook: annotations: {} extraEnv: [] podAnnotations: {} podLabels: {} ttlSecondsAfterFinished: 120 nodeSelector: {} podAnnotations: {} podLabels: {} podSecurityContext: runAsUser: 1000 pullPolicy: Always resources: {} schedule: 10 0 * * * securityContext: {} serviceAccount: automountServiceAccountToken: false create: true name: null successfulJobsHistoryLimit: 3 tag: latest tolerations: [] extraObjects: [] fullnameOverride: "" hotrod: affinity: {} enabled: false image: pullPolicy: Always pullSecrets: [] repository: jaegertracing/example-hotrod ingress: annotations: {} enabled: false hosts: - chart-example.local tls: null nodeSelector: {} podSecurityContext: {} replicaCount: 1 resources: {} securityContext: {} service: annotations: {} loadBalancerSourceRanges: [] name: hotrod port: 80 type: ClusterIP serviceAccount: automountServiceAccountToken: false create: true name: null tolerations: [] tracing: host: null port: 6831 ingester: affinity: {} annotations: {} autoscaling: enabled: false maxReplicas: 10 minReplicas: 2 cmdlineParams: {} dnsPolicy: ClusterFirst enabled: false extraConfigmapMounts: [] extraSecretMounts: [] image: jaegertracing/jaeger-ingester imagePullSecrets: [] nodeSelector: {} podAnnotations: {} podLabels: {} podSecurityContext: {} pullPolicy: IfNotPresent replicaCount: 1 resources: {} securityContext: {} service: annotations: {} loadBalancerSourceRanges: [] type: ClusterIP serviceAccount: automountServiceAccountToken: false create: true name: null serviceMonitor: additionalLabels: {} enabled: false tolerations: [] kafka: autoCreateTopicsEnable: true replicaCount: 1 zookeeper: replicaCount: 1 serviceAccount: create: true nameOverride: "" provisionDataStore: cassandra: false elasticsearch: false kafka: false query: affinity: {} agentSidecar: enabled: true annotations: {} cmdlineParams: {} dnsPolicy: ClusterFirst enabled: true extraConfigmapMounts: [] extraEnv: [] extraVolumes: [] image: jaegertracing/jaeger-query imagePullSecrets: [] ingress: annotations: {} enabled: false health: exposed: false nodeSelector: {} oAuthSidecar: args: [] containerPort: 4180 enabled: false extraConfigmapMounts: [] extraEnv: [] extraSecretMounts: [] image: quay.io/oauth2-proxy/oauth2-proxy:v7.1.0 pullPolicy: IfNotPresent podAnnotations: {} podLabels: {} podSecurityContext: {} priorityClassName: "" pullPolicy: IfNotPresent replicaCount: 1 resources: {} securityContext: {} service: annotations: {} loadBalancerSourceRanges: [] port: 80 type: ClusterIP serviceAccount: annotations: {} automountServiceAccountToken: false create: true name: null serviceMonitor: additionalLabels: {} enabled: false sidecars: [] tolerations: [] schema: activeDeadlineSeconds: 300 annotations: {} extraEnv: [] image: jaegertracing/jaeger-cassandra-schema imagePullSecrets: [] podAnnotations: {} podLabels: {} podSecurityContext: {} pullPolicy: IfNotPresent resources: {} securityContext: {} serviceAccount: automountServiceAccountToken: true create: true name: null spark: affinity: {} annotations: {} cmdlineParams: {} concurrencyPolicy: Forbid enabled: true extraConfigmapMounts: - name: jaeger-tls mountPath: /tls subPath: "" configMap: jaeger-tls readOnly: true extraEnv: - name: ES_SSL_NO_VERIFY value: "true" - name: VERIFY_CERTS value: "false" - name: "JAVA_OPTS" value: "-Djavax.net.ssl.trustStore=/tls/trust.store -Djavax.net.ssl.trustStorePassword=xxxx" - name: "ES_TIME_RANGE" value: "now-1h" - name: "ES_VERSION" value: "7" extraSecretMounts: [] path: /etc/pki/java/cacerts hostPath: /cacert.pem failedJobsHistoryLimit: 5 image: jaegertracing/spark-dependencies imagePullSecrets: [] nodeSelector: {} podAnnotations: {} podLabels: {} pullPolicy: Always resources: {} schedule: 10,20,25,30,40,50,0 * * * * serviceAccount: automountServiceAccountToken: false create: true name: null successfulJobsHistoryLimit: 5 tag: latest tolerations: [] storage: cassandra: cmdlineParams: {} existingSecret: jaeger-elassandra-auth extraEnv: [] host: elassandra.elassandra.svc.cluster.local keyspace: jaeger password: null port: 9042 tls: enabled: false secretName: cassandra-tls-secret usePassword: false user: jaeger elasticsearch: cmdlineParams: {es.version=7, es.tls.enabled=true, es.tls.skip-host-verify=true, es.create-index-templates=false} extraEnv: [] host: xxxxx nodesWanOnly: true password: admin port: 9200 scheme: https usePassword: true user: admin kafka: authentication: none brokers: - kafka:9092 extraEnv: [] topic: jaeger-test type: elasticsearch tag: ""The text was updated successfully, but these errors were encountered: