Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(THL): ByteBuffer implementation for THL Serialization #4933

Open
wants to merge 12 commits into
base: develop
Choose a base branch
from

Conversation

DarkWeird
Copy link
Contributor

@DarkWeird DarkWeird commented Oct 28, 2021

Contains

THL Serialization - ByteBuffer.
Minimal binary format integrated with THL.

Profits:

  1. Faster then protobuf. Reading zero-copy.(writing - not)
  2. Pure java
  3. Deep integration with THL
  4. Schema-less (we don't need it with TypeHandlers)

Cons:

  1. Can be failing edge cases
  2. You cannot determinate how many bytes eats serialization before you start serialization. (You cannot create file and buffers with eager calculated size)

It can replaces Protobuf at network and savings.
It can be extracted to gestalt and can be reused in DS.

How to test

  • Run tests at THL

Outstanding before merging

  • Docs.

@github-actions github-actions bot added the Type: Improvement Request for or addition/enhancement of a feature label Oct 28, 2021
@DarkWeird
Copy link
Contributor Author

@pollend
Copy link
Member

pollend commented Nov 24, 2021

@DarkWeird we might want to use this for encoding chunk data. https://en.wikipedia.org/wiki/LEB128 can we move this into gestalt. If its fast enough I would opt to also do this for chunk data and we can version the format to manage this :P.

@DarkWeird
Copy link
Contributor Author

DarkWeird commented Nov 24, 2021

@DarkWeird we might want to use this for encoding chunk data. https://en.wikipedia.org/wiki/LEB128 can we move this into gestalt.

Why no?

Simple variable length int. LEB128 like Protobuf's varint/varlong
another questions - How you will to calculate offsets for arrays without prefix length?

Int array = 4 * size. and 4 * index.
Long, Double,Float same.

@DarkWeird
Copy link
Contributor Author

@pollend what you did?

Comment on lines +206 to +209
return type == BBType.FLOAT
|| type == BBType.DOUBLE
|| type == BBType.INTEGER
|| type == BBType.LONG;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can define all this encoding information in BBType

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. but not directly.
I cannot use enum values in same enum 'constructors'

Another variant, i can provide something like capabilities.. but it is will mess NULL(0,false,false,false,false,false,true)

or switches... switches... switches in methods..

@pollend
Copy link
Member

pollend commented Nov 26, 2021

I rebaesed the changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Improvement Request for or addition/enhancement of a feature
Projects
Status: Backlog
Development

Successfully merging this pull request may close these issues.

None yet

2 participants