Serialization Strategy to serialize an Array

[Snalibe]

Hi,

I'm currently using the VariableDeltaSerializer to serialize the variables of my class. One of them however is an array that can change size. We can remove or insert elements to it.

On the serialization side I do something like this:

---

serializer.Serialize(arraySize)
for (uint64 = 0; i < arraySize; ++i)
{
serializer.Serialize(myArray[i]);
}

When I keep adding elements to the array, eventually I crash in Raknet (see callstack below).
I understand that Raknet keeps internally the data of my structure to verify if the data changed since last serialize. So if I remove an element of the array, it might end up with extra data that is now unused in it's memory since I serialize less.

However, my crash occurs when I keep adding to the array (eventually)

---

curPage seem to contain unitialized data and doesn't seem valid.

curPage->availableStack[curPage->availableStackSize++]=memoryWithPage;

runtime.network.dll!DataStructures::MemoryPoolRakNet::VariableDeltaSerializer::ChangedVariablesList::Release(RakNet::VariableDeltaSerializer::ChangedVariablesList * m, const char * file, unsigned int line) Line 194 C++
runtime.network.dll!RakNet::VariableDeltaSerializer::FreeVarsAssociatedWithReceipt(RakNet::RakNetGUID guid, unsigned int receiptId) Line 179 C++
runtime.network.dll!RakNet::VariableDeltaSerializer::OnMessageReceipt(RakNet::RakNetGUID guid, unsigned int receiptId, bool messageArrived) Line 29 C++
runtime.network.dll!ZRakNetReplica::OnNotifyReplicaOfMessageDeliveryStatus(RakNet::RakNetGUID guid, unsigned int receiptId, bool messageArrived) Line 114 C++
runtime.network.dll!ZReplicaManager::DoNotifyReplicaOfMessageDeliveryStatus::__l2::(ZRakNetReplica * replica) Line 234 C++
runtime.network.dll!ZDelegate<void __cdecl(ZRakNetReplica * __ptr64)>::operator()(ZRakNetReplica * <args_0>) Line 286 C++
runtime.network.dll!ZReplicaManager::CallOnReplicas(ZDelegate<void __cdecl(ZRakNetReplica *)> f) Line 253 C++
runtime.network.dll!ZReplicaManager::DoNotifyReplicaOfMessageDeliveryStatus(RakNet::RakNetGUID guid, unsigned int receiptId, bool messageArrived) Line 235 C++
runtime.network.dll!ZReplicaManager::NotifyReplicaOfMessageConsumptionStatus(RakNet::RakNetGUID guid, unsigned int receiptId, bool messageConsumed) Line 199 C++
runtime.network.dll!ZNetworkComponent::ProcessRaknetPackets() Line 419 C++
runtime.network.dll!ZNetworkComponent::Update(float fDeltaTime) Line 222 C++

---

I am wondering if I'm serializing my array the proper way. I haven't seen exemples using Raknet that serialize dynamic containers.

Thx!

[BigJoe01]

Use BitStream
Write array size and write array items, when read read first item as array size and read number of items

[Snalibe]

Thx BigJoe.

I was aware of the possibility of using bitstream, however I was attempting to use VariableDeltaSerializer to reduce the amount of data sent over the network.

Using a bit stream would mean that I need to send it each time I serialize and could be costly the bigger the structure gets.

[BigJoe01]

What mean the bigger structure?
Watch example in ReplicaManager3 main.cpp

[Snalibe]

Well let's say you have a structure that has 1000 elements. If the server adds one element to it, when the serialization function is gonna iterate on it, it's gonna call Serialize(element) 1001 times.

Using a bit stream, all the 1001 elements will get written. It theses are uint64 for exemple, it is a lot of data.

By using VariableDeltaSerializers, the idea is to call Serialize(element) the same way, but internally, Raknet compares the new value with the old. If it has not changed, it will only write one bit to indicate that nothing changed for that variable.

It will do that 1000 times and only on the 1001 th element, it will write the full uint64. This would result in far less data to be sent over the network.

[BigJoe01]

Delta serializer working with fixed size memory area

[Snalibe]

There is no way around that isn't?

So I guess that if I want to do what I said, I need to implement my own compression strategy using bitstream.

[Snalibe]

We have a system that recursively serialize classes where certain member variables are flagged as replicated.

I don't know in advance the type of member variable. It can be a class, a float, etc. It can also be a dynamic container. The programmer that is gonna use a replicated dynamic container would be responsible for it's size. I just want to support the generic case.

This is why I'm trying to figure out a way to keep using the VariableDeltaSerializer.