Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to add compression when serializing/deserializing java.util.zip.ZipException: incorrect header check #332

Open
wkoszycki opened this issue Oct 9, 2019 · 0 comments

Comments

@wkoszycki
Copy link

Hi I would like to add compression when serializing object and then deserialize object back. Since I haven't found any info related in either GH or scaladoc, I tried to manipulate exisintg KryoPool class to achieve that. However I am getting :
com.esotericsoftware.kryo.KryoException: Buffer too small: capacity: 0, required: 1

here is the class:

/*
Copyright 2013 Twitter, Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package com.twitter.chill;

import com.esotericsoftware.kryo.Kryo;
import com.esotericsoftware.kryo.io.Input;
import com.esotericsoftware.kryo.io.InputChunked;
import com.esotericsoftware.kryo.io.Output;
import org.constellation.consensus.SnapshotInfo;

import java.io.ByteArrayInputStream;
import java.io.OutputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.util.zip.*;

/** Simple ResourcePool to save on Kryo instances, which
 * are expensive to allocate
 */
abstract public class KryoPoolHacked extends ResourcePool<SerDeState > {

  protected KryoPoolHacked(int poolSize) { super(poolSize); }

  @Override
  public void release(SerDeState  st) {
    st.clear();
    super.release(st);
  }
  /** Output is created with new Output(outBufferMin, outBufferMax);
   */
  public static KryoPoolHacked withBuffer(int poolSize,
      final KryoInstantiator ki,
      final int outBufferMin,
      final int outBufferMax) {
    return new KryoPoolHacked(poolSize) {
      protected SerDeState  newInstance() {
        return new SerDeState (ki.newKryo(), new Input(), new Output(new DeflaterOutputStream(new ByteArrayOutputStream(),new Deflater(4, true))));
      }
    };
  }

  /** Output is created with new Output(new ByteArrayOutputStream())
   * This will automatically resize internally
   */
  public static KryoPoolHacked withByteArrayOutputStream(int poolSize,
      final KryoInstantiator ki) {
    return new KryoPoolHacked(poolSize) {
      protected SerDeState  newInstance() {
        return new SerDeState (ki.newKryo(), new Input(), new Output(new DeflaterOutputStream(new ByteArrayOutputStream(),new Deflater(4, true)))) {
          /*
           * We have to take extra care of the ByteArrayOutputStream
           */
          @Override
          public void clear() {
            super.clear();
            ByteArrayOutputStream byteStream = (ByteArrayOutputStream)output.getOutputStream();
            byteStream.reset();
          }
          @Override
          public byte[] outputToBytes() {
            output.flush();
            ByteArrayOutputStream byteStream = (ByteArrayOutputStream)output.getOutputStream();
            return byteStream.toByteArray();
          }
          @Override
          public void writeOutputTo(OutputStream os) throws IOException {
            output.flush();
            ByteArrayOutputStream byteStream = (ByteArrayOutputStream)output.getOutputStream();
            byteStream.writeTo(os);
          }
        };
      }
    };
  }

  public <T> T deepCopy(T obj) {
    return (T)fromBytes(toBytesWithoutClass(obj), obj.getClass());
  }

  public Object fromBytes(byte[] ary) {
    SerDeState  serde = borrow();
    try {
      serde.setInput(new InflaterInputStream(new ByteArrayInputStream(ary)));
      return serde.readClassAndObject();
    }
    finally {
      release(serde);
    }
  }

  public <T> T fromBytes(byte[] ary, Class<T> cls) {
    SerDeState  serde = borrow();
    System.out.println("wkoszycki2");
    try {
      serde.setInput(new InflaterInputStream(new ByteArrayInputStream(ary)));
      return serde.readObject(cls);
    }
    finally {
      release(serde);
    }
  }


  public byte[] toBytesWithClass(Object obj) {
    SerDeState  serde = borrow();
    try {
      serde.writeClassAndObject(obj);
      return serde.outputToBytes();
    }
    finally {
      release(serde);
    }
  }

  public byte[] toBytesWithoutClass(Object obj) {
    SerDeState  serde = borrow();
    try {
      serde.writeObject(obj);
      return serde.outputToBytes();
    }
    finally {
      release(serde);
    }
  }

  public boolean hasRegistration(Class obj) {
    SerDeState  serde = borrow();
    try {
      return serde.hasRegistration(obj);
    }
    finally {
      release(serde);
    }
  }
}

setting bytes[] as input first and then stream

  public Object fromBytes(byte[] ary) {
    SerDeState  serde = borrow();
    try {
      serde.setInput(ary);
      serde.setInput(new InflaterInputStream(new ByteArrayInputStream(ary)));
      return serde.readClassAndObject();
    }
    finally {
      release(serde);
    }
  }

ends with following:
java.util.zip.ZipException: incorrect header check

Please help

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant