You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm trying to run the sample code in scispark_zeppelin_notebooks on the docker-scispark-zeppelin, but it aborted with this exception. Any Solutions?
import org.dia.core.SciSparkContext
SciSc: org.dia.core.SciSparkContext = org.dia.core.SciSparkContext@41ae71d1
sRDD: org.apache.spark.rdd.RDD[org.dia.core.SciTensor] = MapPartitionsRDD[94] at map at SciSparkContext.scala:168
reshaped: org.apache.spark.rdd.RDD[org.dia.core.SciTensor] = MapPartitionsRDD[95] at map at <console>:104
sum: org.dia.core.SciTensor =
Variable in use = ch4
Set(ch4)
(SOURCE,hdfs://localhost:9000/workshop_s3/201/merg/merg_2006091107_4km-pixel.nc)
java.lang.IllegalArgumentException: NDArrayIndex is out of range. Beginning index:2000 must be less than its size:1238
at org.nd4j.linalg.indexing.NDArrayIndex.validate(NDArrayIndex.java:322)
at org.nd4j.linalg.indexing.NDArrayIndex.resolve(NDArrayIndex.java:306)
at org.nd4j.linalg.indexing.ShapeOffsetResolution.exec(ShapeOffsetResolution.java:187)
at org.nd4j.linalg.api.ndarray.BaseNDArray.get(BaseNDArray.java:3527)
at org.dia.tensors.Nd4jTensor.apply(Nd4jTensor.scala:157)
at org.dia.tensors.Nd4jTensor.apply(Nd4jTensor.scala:31)
The sample code:
import org.dia.core.SciSparkContext
val SciSc = new SciSparkContext(sc)
SciSc.setLocalProperty("array-lib", "nd4j")
val sRDD = SciSc.NetcdfDFSFile("hdfs://localhost:9000/workshop_s3/201/merg/", List("ch4"))
val reshaped = sRDD.map(sciT => sciT("ch4").reshape( Array(4125 * 1238), "reshaped_ch4"))
val sum = sRDD.reduce((sciTA, sciTB) => sciTA + sciTB)
print(sum.tensor(600->610,2000->2010))
The text was updated successfully, but these errors were encountered:
I'm trying to run the sample code in scispark_zeppelin_notebooks on the docker-scispark-zeppelin, but it aborted with this exception. Any Solutions?
The sample code:
The text was updated successfully, but these errors were encountered: