java.lang.ArrayIndexOutOfBoundsException: -2 at java.util.ArrayList.elementData(ArrayList.java:418) #280
Comments
|
This looks like an error with a closure. What is going on inside HBaseHelpers? See Externalizer in the stack? That is used to serialize closures it tries java serialization first then falls back to kryo. I'm not sure why you get this error, but understanding more about what is getting pulled into the closure can help. Generally seeing all the code is needed here. A common strategy to side step these issues is to make things in closures lazy where defined (not inside the closure). You might also ask on the scalding gitter channel to get more help. |
|
@johnynek Thanks for responding. This is package object HBaseHelpers {
def getCellValueAsString( row: Result, col: HBaseCol ): String =
Bytes.toString( row.getValue( Bytes.toBytes( col.family.name ), Bytes.toBytes( col.name.name ) ) )
}Does help with your hypothesis in any way? |
|
I don't recall ever using a package object like that, but it should work.
Debugging serialization issues is no fun. Generally it works, but when it
does not, you need to struggle a bit sometimes. You could try spores:
https://www.scala-lang.org/blog/2016/11/30/spores-release.html
It is attempting to solve this issue by making compile time errors if
closures can't be serialized. Should work with scalding.
…On Mon, Jan 23, 2017 at 07:54 Roey Izhaki ***@***.***> wrote:
@johnynek <https://github.com/johnynek> Thanks for responding.
This is HBaseHelpers (it is in a file on the classpath):
package object HBaseHelpers {
def getCellValueAsString( row: Result, col: HBaseCol ): String =
Bytes.toString( row.getValue( Bytes.toBytes( col.family.name ), Bytes.toBytes( col.name.name ) ) )
}
Does help with your hypothesis in any way?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#280 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAEJdgvUIcBjUWwjPAT3oNEvgGq9CQ5fks5rVOlEgaJpZM4Lq2Pz>
.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Upon their advice, re-posting EsotericSoftware/kryo#490 here.
So I've hit this error, and all my effort turned futile. It's odd, because it works but then adding all sorts of lines break in exactly the same way.
I have no idea if this has to do with the cluster setup or with dependencies (tracing the dependencies on Maven shows that kyro 2.21 is used).
Any ideas? Pointers? Things to try?
Code
inputPipe .read .mapTo( ( 'rowkey, 'row ) -> ( 'rowkey, 'text ) ) { kvRow: ( ImmutableBytesWritable, Result ) => { val (rowkey, row ) = kvRow // Next line works //val text = Bytes.toString( row.getValue( Bytes.toBytes( 'content.name ), Bytes.toBytes( 'text.name ) ) ) // Breaks (see error below) val text = HBaseHelpers.getCellValueAsString( row, textCol ) // - Doesn't work for some reason // Breaks (same error below) // val doc = proc.annotate( text ) // val ser = new DocumentSerializer // val out = ser.save( doc ) ( rowkey, text ) } } .write( output )Build
Relevant bits:
Error
The text was updated successfully, but these errors were encountered: