Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NDArrayIndex is out of range. Beginning index:2000 must be less than its size:1238 #221

Open
9kalikali opened this issue Mar 27, 2020 · 0 comments

Comments

@9kalikali
Copy link

I'm trying to run the sample code in scispark_zeppelin_notebooks on the docker-scispark-zeppelin, but it aborted with this exception. Any Solutions?

import org.dia.core.SciSparkContext
SciSc: org.dia.core.SciSparkContext = org.dia.core.SciSparkContext@41ae71d1
sRDD: org.apache.spark.rdd.RDD[org.dia.core.SciTensor] = MapPartitionsRDD[94] at map at SciSparkContext.scala:168
reshaped: org.apache.spark.rdd.RDD[org.dia.core.SciTensor] = MapPartitionsRDD[95] at map at <console>:104
sum: org.dia.core.SciTensor = 
Variable in use = ch4
Set(ch4)
(SOURCE,hdfs://localhost:9000/workshop_s3/201/merg/merg_2006091107_4km-pixel.nc)
java.lang.IllegalArgumentException: NDArrayIndex is out of range. Beginning index:2000 must be less than its size:1238
	at org.nd4j.linalg.indexing.NDArrayIndex.validate(NDArrayIndex.java:322)
	at org.nd4j.linalg.indexing.NDArrayIndex.resolve(NDArrayIndex.java:306)
	at org.nd4j.linalg.indexing.ShapeOffsetResolution.exec(ShapeOffsetResolution.java:187)
	at org.nd4j.linalg.api.ndarray.BaseNDArray.get(BaseNDArray.java:3527)
	at org.dia.tensors.Nd4jTensor.apply(Nd4jTensor.scala:157)
	at org.dia.tensors.Nd4jTensor.apply(Nd4jTensor.scala:31)

The sample code:

import org.dia.core.SciSparkContext

val SciSc = new SciSparkContext(sc)
SciSc.setLocalProperty("array-lib", "nd4j")
val sRDD = SciSc.NetcdfDFSFile("hdfs://localhost:9000/workshop_s3/201/merg/", List("ch4"))
val reshaped = sRDD.map(sciT => sciT("ch4").reshape( Array(4125 * 1238), "reshaped_ch4")) 
val sum = sRDD.reduce((sciTA, sciTB) => sciTA + sciTB)
print(sum.tensor(600->610,2000->2010)) 
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant