[FEATURE] Remove spark-docker directory and all contents from the project, OR identify clear use cases.

[peter-mcclonski]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?

As part of the migration to Kubeflow, we should take this opportunity to clean up extraneous or OBE elements of the repository. At this time, it seems that the spark-docker directory and its associated Dockerfile are unused.

Describe the solution you would like

If that is the case, we should either (A) remove the unused content, or (B) identify the needs previously served by the content, and evaluate whether those features both remain valuable, and if they are currently being fulfilled.

Describe alternatives you have considered

Additional context