Unable to save the output while using the barefoot for offline map matching in scalable manner

[Rajjat]

I am using barefoot map matching algorithm for my Ph.D. project. While saving the output in the file, some matched traces are saved but after some time, it produces an error:-

19/09/25 16:50:02 INFO Filter: HMM break - no state transitions
19/09/25 16:50:02 ERROR Utils: Aborting task
java.util.NoSuchElementException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1447)

Scala code for map matching as per git repo:-

  // Run a map job on RDD that uses the matcher instance.
  val matches = traces.groupBy(x => x._1).map(x => {
    val trip = x._2.map({
      x => new MatcherSample(x._1, x._2, x._3)
    }).toList
    matcher.value.mmatch((trip.asJava))
  })
  val checkForempty = matches.filter(f => (!f.isEmpty()))
  val geoJson= checkForempty.map(f => f.toGeoJSON())
  geoJson.saveAsTextFile(output)

Please help me out! Thanks.

[smattheis]

Is the error related to the HMM break? (Are you aware of what the HMM break means?) Could you please provide the full log output? The problem may be in your code handling the HMM break in an appropriate manner ...

(Please note that we can not give support in general for the integration of Barefoot in other libraries like Spark because the scope is just much too wide but I try help where it's possible.)

[Rajjat]

Yes, the error is related to HMM error transitions. No state transmissions occurs when the result is empty but sequence and predecessors are not whereas no state emssions occurs when the result is empty. I am trying to run it on spark and mimic the code present on barefoot wiki. Here is the code for mapMatching.scala

      val matcher = spark.sparkContext.broadcast(new BroadcastMatcher(host, port, database, 
                              user, passwd,roadtype))
        //roadtype is road-types.json"

       // Load trace data as RDD from CSV file asset of tuples:
      // (object-id: String, time: Long, position: Point)
    val traces = spark.sparkContext.textFile(input).map(x => {
    val y = x.split(",")
    val k = y(0).replace(" (", "").replace(")", "").split(" ")
    (y(2), toepoch(y(1)), new Point(k(1).toDouble, k(0).toDouble))
  })
  // Run a map job on RDD that uses the matcher instance.
  val matches = traces.groupBy(x => x._1).map(x => {
    val trip = x._2.map({
      x => new MatcherSample(x._1, x._2, x._3)
    }).toList
    matcher.value.mmatch((trip.asJava))
  })

  
  val removeEmpty = matches.filter(f => (!f.isEmpty()))
  val geoJson = removeEmpty.map(f => f.toGeoJSON().toString())
  geoJson.saveAsTextFile(output)

//here is the BroadcastMatcher class

 object BroadcastMatcher {
   private var instance = null: Matcher

   private def initialize(host: String, port: Int, name: String, user: String, pass: String, config: String) {
 
if (instance != null) return
this.synchronized {
  if (instance == null) { // initialize map matcher once per Executor (JVM process/cluster node)

		val config1= Configuration.read(config)
             val reader = new PostGISReader(host, port, name, "bfmap_ways", user, pass, config1)
           val map = RoadMap.Load(reader)

          map.construct();

         val router = new Dijkstra[Road, RoadPoint]()
         val cost = new TimePriority()
          val spatial = new Geography()

         instance = new Matcher(map, router, cost, spatial)
       }
    }
  }
}

   class BroadcastMatcher(host: String, port: Int, name: String, user: String, pass: String, config: 
    String) extends Serializable {
    def mmatch(samples: List[MatcherSample]): MatcherKState = {
    mmatch(samples, 0, 0)
  }

   def mmatch(samples: List[MatcherSample], minDistance: Double, minInterval: Int): 
 MatcherKState = {
BroadcastMatcher.initialize(host, port, name, user, pass, config)
BroadcastMatcher.instance.mmatch(new ArrayList[MatcherSample](samples), minDistance, 
minInterval)
    }
 }

There are some map matched output in the file. The full log is as follows:-

19/09/26 14:25:08 INFO RoadMap: inserting roads ...
19/09/26 14:25:08 INFO PostGISReader: execute query
19/09/26 14:25:10 INFO RoadMap: inserted 100000 (188998) roads
19/09/26 14:25:12 INFO RoadMap: inserted 200000 (381896) roads
19/09/26 14:25:14 INFO RoadMap: inserted 300000 (576250) roads
19/09/26 14:25:16 INFO RoadMap: inserted 400000 (770361) roads
19/09/26 14:25:18 INFO RoadMap: inserted 500000 (966234) roads
19/09/26 14:25:20 INFO RoadMap: inserted 600000 (1161355) roads
19/09/26 14:25:22 INFO RoadMap: inserted 700000 (1356532) roads
19/09/26 14:25:24 INFO RoadMap: inserted 800000 (1549450) roads
19/09/26 14:25:27 INFO RoadMap: inserted 900000 (1738815) roads
19/09/26 14:25:29 INFO RoadMap: inserted 1000000 (1928086) roads
19/09/26 14:25:31 INFO RoadMap: inserted 1100000 (2117873) roads
19/09/26 14:25:33 INFO RoadMap: inserted 1200000 (2309286) roads
19/09/26 14:25:34 INFO RoadMap: inserted 1257094 (2418660) roads and finished
19/09/26 14:25:34 INFO PostGISReader: close reader
19/09/26 14:25:35 INFO RoadMap: ~424 megabytes used for road data (estimate)
19/09/26 14:25:36 INFO RoadMap: index and topology constructing ...
19/09/26 14:25:43 INFO RoadMap: index and topology constructed
19/09/26 14:25:44 INFO RoadMap: ~195 megabytes used for spatial index (estimate)
19/09/26 14:25:44 INFO Filter: HMM break - no state transitions
19/09/26 14:25:44 INFO Filter: HMM break - no state transitions
19/09/26 14:25:44 INFO Filter: HMM break - no state transitions
19/09/26 14:25:44 INFO Filter: HMM break - no state transitions
19/09/26 14:25:46 INFO Filter: HMM break - no state transitions
19/09/26 14:25:47 INFO Filter: HMM break - no state transitions
19/09/26 14:25:47 INFO Filter: HMM break - no state transitions
19/09/26 14:25:47 INFO Filter: HMM break - no state transitions
19/09/26 14:25:47 INFO Filter: HMM break - no state transitions
19/09/26 14:25:47 INFO Filter: HMM break - no state transitions
19/09/26 14:25:47 INFO Filter: HMM break - no state transitions
19/09/26 14:25:47 INFO Filter: HMM break - no state transitions
19/09/26 14:25:47 INFO Filter: HMM break - no state transitions
19/09/26 14:25:47 INFO Filter: HMM break - no state transitions
19/09/26 14:25:47 INFO Filter: HMM break - no state transitions
19/09/26 14:25:48 INFO Filter: HMM break - no state transitions
19/09/26 14:25:48 INFO Filter: HMM break - no state transitions
19/09/26 14:25:48 INFO Filter: HMM break - no state transitions
19/09/26 14:25:49 INFO Filter: HMM break - no state emissions
19/09/26 14:25:49 INFO Filter: HMM break - no state transitions
19/09/26 14:25:49 INFO Filter: HMM break - no state transitions
19/09/26 14:25:50 INFO Filter: HMM break - no state transitions
19/09/26 14:25:50 INFO Filter: HMM break - no state transitions
19/09/26 14:25:50 INFO Filter: HMM break - no state transitions
19/09/26 14:25:50 INFO Filter: HMM break - no state transitions
19/09/26 14:25:50 INFO Filter: HMM break - no state transitions
19/09/26 14:25:50 INFO Filter: HMM break - no state transitions
19/09/26 14:25:50 INFO Filter: HMM break - no state transitions
19/09/26 14:25:50 INFO Filter: HMM break - no state transitions
19/09/26 14:25:50 INFO Filter: HMM break - no state transitions
19/09/26 14:25:50 INFO Filter: HMM break - no state transitions
19/09/26 14:25:50 INFO Filter: HMM break - no state transitions
19/09/26 14:25:50 INFO Filter: HMM break - no state transitions
19/09/26 14:25:50 INFO Filter: HMM break - no state transitions
19/09/26 14:25:50 INFO Filter: HMM break - no state transitions
19/09/26 14:25:51 INFO Filter: HMM break - no state transitions
19/09/26 14:25:51 INFO Filter: HMM break - no state transitions
19/09/26 14:25:51 INFO Filter: HMM break - no state emissions
19/09/26 14:25:51 INFO Filter: HMM break - no state emissions
19/09/26 14:25:51 INFO Filter: HMM break - no state transitions
19/09/26 14:25:51 INFO Filter: HMM break - no state transitions
19/09/26 14:25:51 INFO Filter: HMM break - no state transitions
19/09/26 14:25:51 INFO Filter: HMM break - no state transitions
19/09/26 14:25:51 INFO Filter: HMM break - no state transitions
19/09/26 14:25:51 INFO Filter: HMM break - no state transitions
19/09/26 14:25:51 INFO Filter: HMM break - no state transitions
19/09/26 14:25:51 INFO Filter: HMM break - no state transitions
19/09/26 14:25:51 INFO Filter: HMM break - no state transitions
19/09/26 14:25:51 INFO Filter: HMM break - no state transitions
19/09/26 14:25:51 INFO Filter: HMM break - no state transitions
19/09/26 14:25:51 INFO Filter: HMM break - no state transitions
19/09/26 14:25:51 INFO Filter: HMM break - no state transitions
19/09/26 14:25:52 INFO Filter: HMM break - no state transitions
19/09/26 14:25:52 INFO Filter: HMM break - no state transitions
19/09/26 14:25:52 INFO Filter: HMM break - no state transitions
19/09/26 14:25:52 INFO Filter: HMM break - no state transitions
19/09/26 14:25:52 INFO Filter: HMM break - no state transitions
19/09/26 14:25:52 INFO Filter: HMM break - no state transitions
19/09/26 14:25:52 INFO Filter: HMM break - no state transitions
19/09/26 14:25:52 INFO Filter: HMM break - no state transitions
19/09/26 14:25:53 INFO Filter: HMM break - no state transitions
19/09/26 14:25:53 INFO Filter: HMM break - no state transitions
19/09/26 14:25:53 INFO Filter: HMM break - no state transitions
19/09/26 14:25:53 INFO Filter: HMM break - no state transitions
19/09/26 14:25:53 INFO Filter: HMM break - no state transitions
19/09/26 14:25:53 INFO Filter: HMM break - no state transitions
19/09/26 14:25:53 INFO Filter: HMM break - no state transitions
19/09/26 14:25:53 INFO Filter: HMM break - no state transitions
19/09/26 14:25:53 INFO Filter: HMM break - no state transitions
19/09/26 14:25:53 INFO Filter: HMM break - no state transitions
19/09/26 14:25:53 INFO Filter: HMM break - no state transitions
19/09/26 14:25:54 INFO Filter: HMM break - no state transitions
19/09/26 14:25:54 INFO Filter: HMM break - no state transitions
19/09/26 14:25:54 INFO Filter: HMM break - no state transitions
19/09/26 14:25:54 INFO Filter: HMM break - no state transitions
19/09/26 14:25:54 ERROR Utils: Aborting task
java.util.NoSuchElementException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1447)
at java.util.HashMap$KeyIterator.next(HashMap.java:1469)
at com.bmwcarit.barefoot.markov.KState.sequence(KState.java:300)
at com.bmwcarit.barefoot.matcher.MatcherKState.toGeoJSON(MatcherKState.java:72)
at net.l3s.spark.map.matching.mapMatching$$anonfun$10.apply(mapMatching.scala:96)
at net.l3s.spark.map.matching.mapMatching$$anonfun$10.apply(mapMatching.scala:96)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1138)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1137)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1137)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1375)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1145)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1125)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
19/09/26 14:25:54 ERROR Executor: Exception in task 1.0 in stage 4.0 (TID 8)
java.util.NoSuchElementException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1447)
at java.util.HashMap$KeyIterator.next(HashMap.java:1469)
at com.bmwcarit.barefoot.markov.KState.sequence(KState.java:300)
at com.bmwcarit.barefoot.matcher.MatcherKState.toGeoJSON(MatcherKState.java:72)
at net.l3s.spark.map.matching.mapMatching$$anonfun$10.apply(mapMatching.scala:96)
at net.l3s.spark.map.matching.mapMatching$$anonfun$10.apply(mapMatching.scala:96)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1138)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1137)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1137)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1375)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1145)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1125)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
19/09/26 14:25:54 ERROR Utils: Aborting task
java.util.NoSuchElementException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1447)
at java.util.HashMap$KeyIterator.next(HashMap.java:1469)
at com.bmwcarit.barefoot.markov.KState.sequence(KState.java:300)
at com.bmwcarit.barefoot.matcher.MatcherKState.toGeoJSON(MatcherKState.java:72)
at net.l3s.spark.map.matching.mapMatching$$anonfun$10.apply(mapMatching.scala:96)
at net.l3s.spark.map.matching.mapMatching$$anonfun$10.apply(mapMatching.scala:96)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1138)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1137)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1137)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1375)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1145)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1125)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
19/09/26 14:25:54 ERROR Executor: Exception in task 0.0 in stage 4.0 (TID 7)
java.util.NoSuchElementException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1447)
at java.util.HashMap$KeyIterator.next(HashMap.java:1469)
at com.bmwcarit.barefoot.markov.KState.sequence(KState.java:300)
at com.bmwcarit.barefoot.matcher.MatcherKState.toGeoJSON(MatcherKState.java:72)
at net.l3s.spark.map.matching.mapMatching$$anonfun$10.apply(mapMatching.scala:96)
at net.l3s.spark.map.matching.mapMatching$$anonfun$10.apply(mapMatching.scala:96)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1138)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1137)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1137)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1375)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1145)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1125)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
19/09/26 14:25:54 WARN TaskSetManager: Lost task 1.0 in stage 4.0 (TID 8, localhost, executor driver): java.util.NoSuchElementException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1447)
at java.util.HashMap$KeyIterator.next(HashMap.java:1469)
at com.bmwcarit.barefoot.markov.KState.sequence(KState.java:300)
at com.bmwcarit.barefoot.matcher.MatcherKState.toGeoJSON(MatcherKState.java:72)
at net.l3s.spark.map.matching.mapMatching$$anonfun$10.apply(mapMatching.scala:96)
at net.l3s.spark.map.matching.mapMatching$$anonfun$10.apply(mapMatching.scala:96)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1138)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1137)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1137)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1375)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1145)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1125)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

19/09/26 14:25:54 ERROR TaskSetManager: Task 1 in stage 4.0 failed 1 times; aborting job
19/09/26 14:25:54 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool
19/09/26 14:25:54 INFO TaskSetManager: Lost task 0.0 in stage 4.0 (TID 7) on localhost, executor driver: java.util.NoSuchElementException (null) [duplicate 1]
19/09/26 14:25:54 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool
19/09/26 14:25:54 INFO TaskSchedulerImpl: Cancelling stage 4
19/09/26 14:25:54 INFO DAGScheduler: ResultStage 4 (saveAsTextFile at mapMatching.scala:97) failed in 47.341 s due to Job aborted due to stage failure: Task 1 in stage 4.0 failed 1 times, most recent failure: Lost task 1.0 in stage 4.0 (TID 8, localhost, executor driver): java.util.NoSuchElementException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1447)
at java.util.HashMap$KeyIterator.next(HashMap.java:1469)
at com.bmwcarit.barefoot.markov.KState.sequence(KState.java:300)
at com.bmwcarit.barefoot.matcher.MatcherKState.toGeoJSON(MatcherKState.java:72)
at net.l3s.spark.map.matching.mapMatching$$anonfun$10.apply(mapMatching.scala:96)
at net.l3s.spark.map.matching.mapMatching$$anonfun$10.apply(mapMatching.scala:96)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1138)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1137)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1137)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1375)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1145)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1125)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
19/09/26 14:25:54 INFO DAGScheduler: Job 2 failed: saveAsTextFile at mapMatching.scala:97, took 48.076742 s
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 4.0 failed 1 times, most recent failure: Lost task 1.0 in stage 4.0 (TID 8, localhost, executor driver): java.util.NoSuchElementException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1447)
at java.util.HashMap$KeyIterator.next(HashMap.java:1469)
at com.bmwcarit.barefoot.markov.KState.sequence(KState.java:300)
at com.bmwcarit.barefoot.matcher.MatcherKState.toGeoJSON(MatcherKState.java:72)
at net.l3s.spark.map.matching.mapMatching$$anonfun$10.apply(mapMatching.scala:96)
at net.l3s.spark.map.matching.mapMatching$$anonfun$10.apply(mapMatching.scala:96)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1138)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1137)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1137)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1375)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1145)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1125)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1499)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1487)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1486)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1486)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:814)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1714)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1669)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1658)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:630)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2022)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2043)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2075)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1151)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply(PairRDDFunctions.scala:1096)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply(PairRDDFunctions.scala:1096)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:1096)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply$mcV$sp(PairRDDFunctions.scala:1070)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1035)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1035)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:1035)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply$mcV$sp(PairRDDFunctions.scala:961)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:961)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:961)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:960)
at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply$mcV$sp(RDD.scala:1489)
at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1468)
at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1468)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1468)
at net.l3s.spark.map.matching.mapMatching$.run(mapMatching.scala:97)
at net.l3s.spark.map.matching.mapMatching$.main(mapMatching.scala:37)
at net.l3s.spark.map.matching.mapMatching.main(mapMatching.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.util.NoSuchElementException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1447)
at java.util.HashMap$KeyIterator.next(HashMap.java:1469)
at com.bmwcarit.barefoot.markov.KState.sequence(KState.java:300)
at com.bmwcarit.barefoot.matcher.MatcherKState.toGeoJSON(MatcherKState.java:72)
at net.l3s.spark.map.matching.mapMatching$$anonfun$10.apply(mapMatching.scala:96)
at net.l3s.spark.map.matching.mapMatching$$anonfun$10.apply(mapMatching.scala:96)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1138)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1137)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1137)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1375)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1145)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1125)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

[smattheis]

Which version do you use? Have you maybe modified the source code of Barefoot because I cannot map the stack trace to the code of the most recent version of Barefoot, i.e., in line 72 of com.bmwcarit.barefoot.matcher.MatcherKState there is no call of the sequence method of class com.bmwcarit.barefoot.markov.KState:

barefoot/src/main/java/com/bmwcarit/barefoot/matcher/MatcherKState.java

Line 72 in 1faaea4

JSONObject json = new JSONObject();

and in line 300 of com.bmwcarit.barefoot.markov.KState there is no call of an iterator method but a return statement:

barefoot/src/main/java/com/bmwcarit/barefoot/markov/KState.java

Line 300 in 1faaea4

return ksequence;

[Rajjat]

val reader = new PostGISReader(host, port, name, "bfmap_ways", user, pass, Configuration.read(new JSONObject(config))

the com.bmwcarit.barefoot.road.Configuration class is missing in the new version. So I used 0.0.2 version. from which source, I should import configuration?

[smattheis]

Please use the most recent version of the library. I cannot give any support for older versions. The Configuration class has been replaced by the com.bmwcarit.barefoot.roadmap.Loader class. So you can use Loader.roadtypes(new JSONObject(config)) to do the same thing in newer versions. Please see the Javadocs: http://bmwcarit.github.io/barefoot/doc/0.1.0/com/bmwcarit/barefoot/roadmap/Loader.html
If the problem remains with most recent version, we can check if there is a bug in the library.

[Rajjat]

Thanks for the reply. It means barefoot wiki is outdated as Configuration class is still used there for reading road-types.json. Also in the mvn-repository, the latest pom is 0.1.1 (https://mvnrepository.com/artifact/com.bmw-carit/barefoot/0.1.1) whereas on gitHub, it is 0.1.5.

[smattheis]

Is the problem solved?

[Rajjat]

Yes, the problem is solved. I would suggest to modify the wiki page regarding "Loader.roadtypes(new JSONObject(config))". Thanks for your help!

[ZhiWeiCui]

@Rajjat
I'm using spark to deploy this service, I'm having some troubles, I want to know more about your project details.
Looking forward to your reply.

[yaya159456]

@ZhiWeiCui I'm also using spark to deploy this service, I'm having some troubles, I want to know more about your project details.
Looking forward to your reply.