Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add BigInt support #211

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Conversation

jasonpaulos
Copy link

@jasonpaulos jasonpaulos commented May 5, 2022

This PR adds native support for JavaScript BigInts.

I didn't want to cause a breaking change in any way, so I've introduced a few different decoding options that essentially allow library users to opt into this feature. Additionally, since no BigInts are returned by this library by default, everything else should remain functional in environments where BigInt is not supported.

Encoding

Similar to Numbers, BigInts are encoded using the smallest possible MessagePack int type. The added benefit of encoding a BigInt is that integers larger than Number.MAX_SAFE_INTEGER can be encoded with the correct precision.

If you attempt to encode a BigInt larger than the maximum uint64 value or smaller than the minimum int64 value, an error will be thrown, since MessagePack does not support larger integer types.

Decoding

Decoding BigInts is more nuanced. I believe there is no single correct way to do this, since consumers of this library will want different behavior depending on their use cases. For example, some may never want to see a BigInt from this library, some may only want to see BigInts if the value being decoded cannot fit into a Number, and some may want to always receive BigInts for consistency.

Instead of choosing just a single way to decoding integers, I've offered the following possibilities, which can be specified with the new DecodeOptions.intMode option:

  • IntMode.UNSAFE_NUMBER: Always returns the value as a number. Be aware that there will be a loss of precision if the value is outside the range of Number.MIN_SAFE_INTEGER to Number.MAX_SAFE_INTEGER.
  • IntMode.SAFE_NUMBER: Always returns the value as a number, but throws an error if the value is outside of the range of Number.MIN_SAFE_INTEGER to Number.MAX_SAFE_INTEGER.
  • IntMode.MIXED: Returns all values inside the range of Number.MIN_SAFE_INTEGER to Number.MAX_SAFE_INTEGER as numbers and all values outside that range as bigints.
  • IntMode.BIGINT: Always returns the value as a bigint, even if it is small enough to safely fit in a number.

For backwards compatibility, IntMode.UNSAFE_NUMBER is the default value for decoding. As a result, there should be no change in behavior for users who do not specify this option.

At Algorand, we've had success adopting a similar approach for JSON integer decoding: algorand/js-algorand-sdk#260

Tests

New tests have been added to ensure the new functionality works as intended. Additionally, I've enabled the bignum tests from msgpack-test-js.

Feedback is appreciated, please let me know if there are any questions or suggested changes!

Closes #115

@jasonpaulos
Copy link
Author

Hi @gfx, is this PR something you would consider adding to the library?

@sagacity
Copy link

We'd also love it if this could be merged. The current ExtensionCodec example isn't really usable since it encodes large integers as strings, which would require special casing on the decoding side (which, in our case, isn't in JS).

@jasonpaulos
Copy link
Author

Hi @gfx, could you please approve the workflow run for this PR so github actions will run?

@gfx
Copy link
Member

gfx commented Jan 11, 2023

Hi. Thank you for the pull request, and sorry for my delayed response.

I'd like to include the BigInt support in the next major release (v3.0). Please wait for my review & decision.

@CHOYSEN
Copy link

CHOYSEN commented Mar 1, 2023

Any update?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BigInt should be handled by native codec instead of extension codec
4 participants