Skip to content

Library (codec generator) for packing integers to JavaScript UTF-16 strings

License

Notifications You must be signed in to change notification settings

screepers/utf15

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

utf15 Build Status

Library (codec generator) for packing integers to JavaScript UTF-16 strings

The main use cases of the library:

  • Converting integer arrays to UTF-16 string: in case you know the exact bit depth ("maximum value") of numbers, you can save them with reduced bit depth to a few UTF-16 code units instead of much larger string by JSON.stringify. Example:

    [1337,1337,1337] + depth:15 => "թթթ"

  • Converting array with variadic bit depths to UTF-16 string: values can be effectively packed even if they have different bit depths. Example:

    [0xFF, 0xAA, 0xFFFF, 0xDEAD] + depths:[8,8,16,16] => "ϾЁψ"

  • Converting single integers to UTF-16 string: you can customize codecs as you wish to create any kind of serialization/deserialization protocol.

Features:

  • Output code points range is [0x0030, 0x8030) (with length = 2^15 = 32768), so control characters [0x0000, 0x0020) and surrogate code points [0xD800, 0xDFFF] will never appear in the output, which could be critical or ineffective while serialization/deserialization.
  • Dense binary packing: for example, 3 integer values with bit_depth=15 will occupy exactly 3 UTF-16 code units, and 10 values with bit_depth=3 will occupy exactly 2 UTF-16 code units.
  • Arrays serialization/deserialization can be performed with variadic bit depths for each element.
  • Meta information can be stored within the encoded string for future decoding without knowing a particular encoding codec.
  • Lightweight, acceptable performance (see TravisCI output for measurement results).

What the utf15.js is not:

  • It's not an lz-string analogue: it won't compress your data in terms of binary size (4 bytes UInt32 value may become 6 bytes UTF-16 string).
  • It's not an archiver: see previous point.
  • It's not a JSON.stringify replacement: utf15.js can process unsigned integers ∈ [0,Number.MAX_SAFE_INTEGER], arrays of such integers, and only them.

Examples:

Here some useful examples of library use cases, see more in tests.js:examples() section.

Global RoomPosition packing

const values = [3, 101, 97, 24, 42];    // room W101N97 x:24 y:42
const depths = [2,   7,  7,  6,  6];    // max = [4,128,128,64,64]

// Codec for position coding, oh, exploitable!
const map_codec = new Codec({ depth:depths, array:1 });

enc = map_codec.encode(values);
dec = map_codec.decode(enc);
assert(arr_eq(values, dec));
console.log(values, `[str.length=${enc.length}]`, dec);

> [ 3, 101, 97, 24, 42 ] '[str.length=2]' [ 3, 101, 97, 24, 42 ]

Directions packing

const values = [0,1,2,3,4,5,6,7,6,5];
const depth = 3; // 3 bits, max 8 values, range [0,7]

const dir_codec = new Codec({ depth, array:1 });

enc = dir_codec.encode(values);
dec = dir_codec.decode(enc);
assert(arr_eq(values, dec));
console.log(values, `[str.length=${enc.length}]`, dec);

> [ 0, 1, 2, 3, 4, 5, 6, 7, 6, 5 ] '[str.length=3]' [ 0, 1, 2, 3, 4, 5, 6, 7, 6, 5 ]

Single integer packing

const value = 1337;

// depth === 11 bits, value < 2^depth, enough
const codec = new Codec({ depth:11 });
enc = codec.encode(value);
dec = codec.decode(enc);
assert(value === dec);
console.log(value, enc, dec);

> 1337 'թ' 1337

ID packing

const
    id = '5a216a3df33b4e435ca8c5ab',    // object.id example
    values = [];                        // array of hex numbers

for(let i = 0, len = id.length; i < len; ++i) {
    const point = parseInt(id[i], 16);
    values.push(point);
}

const id_codec = new Codec({ depth:4, array: 1 });
enc = id_codec.encode(values);
dec = id_codec.decode(enc);

const decoded_id = dec.reduce((acc, x)=> (acc + x.toString(16)), '');

assert(id === decoded_id);
console.log(`${id}.len=${id.length}`, `[str.len=${enc.length}]`, decoded_id);

> 5a216a3df33b4e435ca8c5ab.len=24 [str.len=8] 5a216a3df33b4e435ca8c5ab

About

Library (codec generator) for packing integers to JavaScript UTF-16 strings

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published