Problem with running Delight on a standalone Spark cluster

[lior-wnwd]

Hi,

I found Datamechanics Delight tool online, and started using it locally on my Mac. It seems amazing!
However, while trying to run it from our cluster, it failed and wrote that it "Could not process the Spark Events log. Metrics are unavailable."

We're running Spark 2.4.7 on AWS in standalone mode. The history server is enabled, as well as spark.eventLog.enabled property.
The job does manage to send heartbeats to Delight, and there are no errors in the log (all logs are INFO).
At the end of the log, once the job is finished successfully, I have a message:

[spark-listener-group-shared] INFO  co.datamechanics.delight.DelightStreamingConnector- Application will be available in a few minutes on Delight at this url: https://delight.datamechanics.co/apps/some-id

When opening the link, from the overview screen, I am able to see the Spark UI of the history server, but the overview of Delight is empty (see image below).

This is the spark-submit command I'm using (there are some additional configs on the machine itself):

/root/spark/bin/spark-submit /
--master spark://my-master.com --name app-name /
--class com.example.MyApp --total-executor-cores 24 /
--conf spark.eventLog.dir=file:/tmp 
--conf spark.history.fs.logDirectory=file:/tmp /
--repositories https://oss.sonatype.org/content/repositories/snapshots /
--packages co.datamechanics:delight_2.11:latest-SNAPSHOT /
--conf spark.delight.accessToken.secret=my-token  /
--conf spark.extraListeners=co.datamechanics.delight.DelightListener /
 my-app.jar

How can I solve this issue, in order to use Delight from our cluster?

Thanks,
Lior