[OptionalField] attribute not supported by slim serializer?

[urosberce]

Marking new fields in a serialized class as [OptionalField] does not work(like it does with Binary serializer), exception is thrown during deserialization.

Is there any way around this?

Thanks!

[itadapter]

in the next version, not current. [OptionalField] introduces way too much overhead and Slim is not purposed for version tolerance period. You can try using NFX Arow serializer which is faster and version tolerant but requires more rigid models and various declaratons (akin to protobuf)

described your use-case in more details

[urosberce]

Thank you for your answer. I understand perfectly. Although it's a pitty, as I find your serializer to be the best and most versatile. I have a relatively complex model to serialize and all the other serializers I tested failed or were lagging far behind in performance. So, keep up the good work!

Any idea if I could get around it by extending ISerializable and manipulating get/set object data somehow?

[itadapter]

Any idea if I could get around it by extending ISerializable and manipulating get/set object data somehow?

Absolutely! You can do just what you have described, which is basically implementing ISerializable and .ctor for deserialization (you can see unit tests for cases that cover that, for example here: https://github.com/agnicore/nfx/blob/master/src/testing/NFX.UTest/Serialization/Slim2.cs#L701-L725

Keep in mind, that using ISerializable degrades the ser/deser performance as it does boxing and copies via SerializationInfo

[urosberce]

So I gave protobuf-sharp a try. Took me a while to decorate all the members (I have over 900 in about 50 classes). And to my surprise - the produced binary was larger than with slim serializer, and ser/deser was slower! But a real deal breaker with protobuf for me was it's inability to denote null values in arrays/lists. I have some arrays where exact structure(including nulls wherever they appear) is essential, cant change that.

So I will try to implement ISerializable for slim to manage [OptionalFields]. Been playing for a bit with it now. I understand how to use SerializationInfo and enumerator.MoveNext() to modify specific types and members. But I have no idea how I could tackle [OptionalFields]. Any hint would be highly appreciated!

Thanks!

[urosberce]

Oh, dont mind my previous post, I got it working using ISerializable, works like a charm!

Thanks a lot!

[urosberce]

Hey, it's me again, playing with slim serializer some more :)

It happens quite often when objects extended with ISerializable are deserialized, in the constructor MyObject(SerializationInfo info, StreamingContext context), the SerializationInfo info is an invalid object, its MemberCount is 0, trying to get enumerator throws an exception, etc. I couldnt find any pattern, it really seems random. Sometimes it works fine, sometimes not. If I then comment out ISerializable extension, it always deserializes just fine.

Do you have any idea why this could be happening? I'll dig into it a bit more...

Thanks!

[itadapter]

need to see your code... no idea per description provided

[urosberce]

Hey, sorry for a late reply, but I had to go through with an update with "old" binary serializer. I would still love to replace it with slim, but first I need to solve this issue.

I cant really post the actual code, as the classes I serialize are quite large.

So here's whats going on:

[Serializable]
public class MyClass: ISerializable
{
        string id;
        //etc... (using only serializable types, strings, decimals, ints, and Lists and 1d arrays of those)

        public MyClass(SerializationInfo info, StreamingContext context)
        {
            //I do iteration through info entries at this point, BUT:
            //In some cases, info is an empty object, MemberCount = 0, and trying to iterate it with
            //info.GetEnumerator().MoveNext() throws an exception      
            //If I comment out custom serialization (//:ISerializable), the SAME object is deserialized without
            //any errors      
        }
        public void GetObjectData(SerializationInfo info, StreamingContext context)
        {            
                //...
        }
}        

I hope it is a bit more clear about what the problem is. As I said, I tested ser and deser for many objects, and with about 1/3 of objects this happens when I try to use custom serialization with ISerializable. And I really can not figure out a pattern. All objects are virtually the same.

Thanks!

[itadapter]

Well, it seems that the stream gets corrupted. That is why they say bin serializers are brittle in general. Are you able to d-serialize the NEXT object in stream after the first one fails, in other words, if you write to stream: a,b; lets say [a] can not be read, is the stream positioned correctly at the [b] first byte?

Then, are the types being ser/deser get rebuilt in between app runs? Looks like the object structure may have changed?

As far as custom serialization, can you send me the code for .ctor(SerInfo) GetObjectData(SerInfo)? This may be some trivial fluke. How are there 1/3 objects different than the other 2/3? Are you using batching mode type registry (which gives 20-60% performance boost)?

[urosberce]

Hey there! Sorry again for a late reply. Finally got some time again to work on this. So first to answer your questions:

• no, I can not read from the stream anymore after an object fails
• no, the structure does not change
• no, I am using PerCall mode

I noticed one thing; If I serialize my object without custom serialization (ISerializable commented out) I cannot deserialize it with ISerializable back on, and vice versa. It is always the case. It should be easy to reproduce this behaviour.

My custom ser and deser code is very basic for testing purposes:

public MyClass(SerializationInfo info, StreamingContext context)
        {
            Type thistype = GetType();
            var enumerator = info.GetEnumerator();
            while (enumerator.MoveNext())
            {
                var current = enumerator.Current;
                var f = thistype.GetField(current.Name);
                f.SetValue(this, current.Value);                
            }
        }
        public void GetObjectData(SerializationInfo info, StreamingContext context)
        {
            Type thistype = GetType();
            foreach (var f in thistype.GetFields())
            {
                if (!f.IsNotSerialized) info.AddValue(f.Name, f.GetValue(this));                              
            }
        }

I did some debugging, and error always seems to arise in reader.ReadVarIntStr() --> ReadString() ->int count = this.ReadInt(), with count being a negative value. For example:

System.ArgumentOutOfRangeException: Non-negative number required.
Parameter name: count
at System.Text.UTF8Encoding.GetString(Byte[] bytes, Int32 index, Int32 count)
at NFX.IO.SlimReader.ReadString()
at NFX.IO.SlimReader.ReadVarIntStr()
at NFX.Serialization.Slim.TypeDescriptor.deserializeInfo(SlimReader reader, TypeRegistry registry, RefPool refs, StreamingContext streamingContext)
at NFX.Serialization.Slim.TypeDescriptor.DeserializeInstance(SlimReader reader, TypeRegistry registry, RefPool refs, Object& instance, StreamingContext streamingContext)
at NFX.Serialization.Slim.TypeSchema.DeserializeRefTypeInstance(Object instance, SlimReader reader, TypeRegistry registry, RefPool refs, StreamingContext streamingContext)
at NFX.Serialization.Slim.SlimSerializer.deserialize(SlimReader reader, RefPool pool)
at NFX.Serialization.Slim.SlimSerializer.Deserialize(Stream stream)

System.ArgumentOutOfRangeException: Non-negative number required.
Parameter name: count
at System.Text.UTF8Encoding.GetString(Byte[] bytes, Int32 index, Int32 count)
at NFX.IO.SlimReader.ReadString()
at NFX.IO.SlimReader.ReadVarIntStr()
at NFX.Serialization.Slim.TypeSchema.DeserializeRootOrInner(SlimReader reader, TypeRegistry registry, RefPool refs, StreamingContext streamingContext, Boolean root, Type valueType)
at NFX.Serialization.Slim.TypeSchema.Deserialize(SlimReader reader, TypeRegistry registry, RefPool refs, StreamingContext streamingContext, Type valueType)
at NFX.Serialization.Slim.TypeDescriptor.deserializeInfo(SlimReader reader, TypeRegistry registry, RefPool refs, StreamingContext streamingContext)
at NFX.Serialization.Slim.TypeDescriptor.DeserializeInstance(SlimReader reader, TypeRegistry registry, RefPool refs, Object& instance, StreamingContext streamingContext)
at NFX.Serialization.Slim.TypeSchema.DeserializeRefTypeInstance(Object instance, SlimReader reader, TypeRegistry registry, RefPool refs, StreamingContext streamingContext)
at NFX.Serialization.Slim.SlimSerializer.deserialize(SlimReader reader, RefPool pool)
at NFX.Serialization.Slim.SlimSerializer.Deserialize(Stream stream)

And ideas?

Thanks a lot!

[itadapter]

I noticed one thing; If I serialize my object without custom serialization (ISerializable commented out) I cannot deserialize it with ISerializable back on, and vice versa. It is always the case. It should be easy to reproduce this behaviour.

This is a no-no - you can only deserialize what has beenserialized with THE SAME code, (you are saying you have commented something out), in other words.

Commenting ISerializable just for deseialization will not work, it produces completely different binary layout.

thistype.GetFields() <-- not sure what you are doing with this but it is strange. You need to access property/field names verbatim as in: info.AddValue("age", this.m_Age)...

[urosberce]

Hey, thanks for your answer. Ok I think I get it, I thought slim serializer was using field names as does BinaryFormatter, but it doesnt I guess, and that's why it fails when trying to deserialize it with ISerializable?

So suppose I have an instance of MyClass stored, which I serialized with slim (without ISerializable). And now I want to add a new field to MyClass class definition, but still be able to open(deser) instances that were serialized without this new field. With BinaryFormatter I would use [OptionalField] for that new field. What options do I have with slim?

Why do you think thistype.GetFields() is strange?? It simply gets all the fields in the object being serialized, so I can iterate them and save them using info.AddValue(...) like you suggest (so I dont have to manually save each field, but all of them in a loop instead).

Thanks!

[itadapter]

So suppose I have an instance of MyClass stored, which I serialized with slim (without ISerializable). And now I want to add a new field

Slim serializer purposely was never designed for that.
Use Arow serializer instead, why do you use binary serialization in the first place?