Connector install - OpenLineage does not Initialize

[yxu1183]

Describe the bug
When the DB compute cluster starts, OpenLineage doesn't initialize as expected, and no events are produced.

To Reproduce
Steps to reproduce the behavior:

1. Follow the Connector installation and post installation instructions.
2. Upload the OpenLineage Jar into DBFS.
3. Configured the all compute cluster (single user mode) with spark configuration.
4. Uploaded the open lineage init script into the user workspace directory.
5. Updated the init script path in the cluster with workspace as source and file path.
6. Start the compute cluster.

Expected behavior
By the instructions - https://github.com/OpenLineage/OpenLineage/tree/main/integration/spark/databricks#initialization-logs - we should see 3 log entries for intiatlization

"Registered listener io.openlineage.." - this one appears
"OpenLineageContext: Init OpenLineageContext:" - this one is missing
"AsyncEventQueue: Process of event SparkListenerApplicationStart" - this one appears

Logs

1. Please include any Spark code being ran that generates this error
   spark.openlineage.version v1
spark.openlineage.namespace <adb_workspace_id>#<cluster_id>
spark.openlineage.host https://<functionapp_name>.azurewebsites.net/
spark.openlineage.url.param.code {{secrets/<scope>/<function_defualt_key}}

2. Init Scripts Path updated in the cluster:
   Type: Workspace
   File path: /Users/user_name/init_scripts/open-lineage-init-script.sh

Open Lineage script's absolute path - /Users/user_name/init_scripts/open-lineage-init-script.sh

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Windows
• OpenLineage Version: 0.18.0
• Databricks Runtime Version: 11.3
• Cluster Type: All compute cluster
• Cluster Mode: Single User Access
• Using Credential Passthrough: N/A

Additional context
Add any other context about the problem here.

[wjohnson]

@yxu1183 did you attempt to run a notebook and it failed to produce lineage?

If you're seeing the AsyncEventQueue: Process of event SparkListenerApplicationStart you should be able to receive lineage events!