Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spark DataSets using OSMEntity case classes #75

Open
angelcervera opened this issue Jan 19, 2021 · 0 comments
Open

Spark DataSets using OSMEntity case classes #75

angelcervera opened this issue Jan 19, 2021 · 0 comments
Labels
enhancement spark Spark connector

Comments

@angelcervera
Copy link
Member

angelcervera commented Jan 19, 2021

At the moment, the Spark connector is using DataFrames. It would be useful to allow direct interaction between OSMEntity types and the Spark Connector using DataSets.

Something like these cases should work.

      import spark.implicits._
      val dataset = Seq(
        NodeEntity(1, 11, 10, Map({ "nodeId" -> "1"})),
        NodeEntity(2, 12, 20, Map({ "nodeId" -> "2"})),
        NodeEntity(3, 13, 30, Map.empty),
        NodeEntity(4, 14, 40, Map.empty),
        NodeEntity(5, 15, 50, Map.empty),
        NodeEntity(6, 16, 60, Map.empty),
        WayEntity(7, Seq(1,2,3,4), Map({ "wayId" -> "7"})),
        WayEntity(8, Seq(4,5,6), Map({ "wayId" -> "8"})),
      ).toDS()

      dataset.show()

Or

      import spark.implicits._
      val monaco = spark.sqlContext.read
        .format("osm.pbf")
        .load("src/test/resources/monaco.osm.pbf")
        .persist()

      monaco.as[OSMEntity]
@angelcervera angelcervera added enhancement spark Spark connector labels Jan 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement spark Spark connector
Projects
None yet
Development

No branches or pull requests

1 participant