Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use list of string lengths + list of strings instead of escaped null-terminated strings #11

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Use list of string lengths + list of strings instead of escaped null-terminated strings #11

wants to merge 1 commit into from

Conversation

arai-a
Copy link

@arai-a arai-a commented May 15, 2019

changed String dictionary section to the following,
now we don't have to do escape, nor look for 0x01/0x00 while reading.

StringPrelude ::= n=NumberOfStrings StringLength{n} StringDefinition{n}
NumberOfStrings ::= varnum
StringLength ::= varnum
StringDefinition ::= Bytes{corresponding StringLength}
Bytes ::= 0x00-0xff

@arai-a arai-a requested a review from dominiccooney May 15, 2019 13:36
@Yoric
Copy link

Yoric commented May 15, 2019

Also, I suspect that this will improve compression a little bit.

@arai-a
Copy link
Author

arai-a commented May 21, 2019

Checked the file size with facebook files in binjs-ref repository.
This uses empty external dictionary, so all strings are stored in
each binary file.

also, checked another variant that sorts the string dictionary by length first.

unfortunately this change mostly increases the file size.
sorting the strings by length first somewhat improves, but still increases.

Apparently the length part isn't much compression-friendly compared to null-terminator.
this needs some more experiment.

file original patched sorted by length
09PWOD-iG7U.js.bin 690 721 (104.49%) 706 (102.32%)
0FL4yAxztNI.js.bin 34512 35764 (103.63%) 35530 (102.95%)
0OShvvyv_0H.js.bin 21500 21871 (101.73%) 21911 (101.91%)
0QMTlI7NH0s.js.bin 9625 9842 (102.25%) 9612 ( 99.86%)
0YyVL58RB73.js.bin 4158 4188 (100.72%) 4229 (101.71%)
0_6_wz_uelN.js.bin 43665 45277 (103.69%) 44869 (102.76%)
0gVBItN8Q5A.js.bin 1307 1310 (100.23%) 1276 ( 97.63%)
0nNPGdaD0nT.js.bin 51971 54120 (104.13%) 53813 (103.54%)
0z7SVtqevWh.js.bin 36257 37278 (102.82%) 37008 (102.70%)
10x6WBItZMj.js.bin 128161 131090 (102.29%) 132381 (103.29%)
13JTyxJ1fHu.js.bin 30621 31221 (101.96%) 30956 (101.90%)
16RV_DNduLc.js.bin 20615 21163 (102.66%) 20910 (101.43%)
1E98iSLpnkH.js.bin 1673 1766 (105.56%) 1672 ( 99.94%)
1Fs4jyLrDMS.js.bin 3485 3519 (100.98%) 3510 (100.72%)
1PqKM6bvJiO.js.bin 98065 101420 (103.42%) 101409 (103.41%)
1Q7Vw4LF2Yr.js.bin 908 909 (100.11%) 894 ( 98.46%)
1WkLWwGfQa2.js.bin 268 273 (101.87%) 267 ( 99.63%)
1XzqGa3bKC9.js.bin 53950 55477 (102.83%) 55620 (103.10%)
1YrwUhW-w5Y.js.bin 13252 13780 (103.98%) 13716 (103.50%)
1eOj1Brg5D4.js.bin 7497 7772 (103.67%) 7629 (101.76%)
1iFRg9BeNmg.js.bin 2679 2681 (100.70%) 2606 ( 97.28%)
1kL74PTQCjO.js.bin 595 578 ( 97.14%) 592 ( 99.50%)
1sUao0c2PEe.js.bin 863 878 (101.74%) 870 (100.81%)
1vzDGthwISY.js.bin 19502 20054 (102.83%) 19717 (101.10%)
22K2uaUdQmP.js.bin 1535 1587 (103.39%) 1530 ( 99.67%)
26eXFdywVMp.js.bin 6253 6540 (104.59%) 6388 (102.16%)
2CDhOMRpesE.js.bin 6916 7078 (102.34%) 7043 (101.84%)
2YodW0RnyTo.js.bin 31908 33137 (103.85%) 32833 (102.90%)
2cMJ_CbnBwW.js.bin 7547 7854 (104.70%) 8157 (108.80%)
2qyyFNJaP7C.js.bin 3524 3648 (103.52%) 3518 ( 99.83%)
2rIE9zLrbX9.js.bin 5279 5528 (104.72%) 5333 (101.20%)
2yFnUvKYcZ7.js.bin 18662 19590 (104.97%) 19175 (102.75%)
3Iqnhvm-T6s.js.bin 15145 15600 (103.00%) 15359 (101.41%)
3MHDDokKiln.js.bin 14576 14977 (102.75%) 14730 (101.60%)
3QDMSHmYtRk.js.bin 6406 6693 (104.48%) 6557 (102.36%)
3cgkC0EM0Qf.js.bin 4081 4191 (102.70%) 4066 ( 99.63%)
3uldL8ZjFT8.js.bin 12291 12486 (101.59%) 12281 ( 99.92%)
3vNGxNuOcut.js.bin 7933 8090 (101.98%) 7890 ( 99.46%)
45TkpKn2ofu.js.bin 37211 38271 (102.85%) 38100 (102.39%)
4ezU-xdFZQt.js.bin 7158 7419 (103.65%) 7243 (101.19%)
4zUTiRaESYw.js.bin 11583 11908 (102.81%) 11620 (100.32%)
5HSUKQwIwZt.js.bin 42082 42732 (101.54%) 42487 (100.96%)
5IFSfwj8om.js.bin 5618 5919 (105.36%) 5803 (103.29%)
5KxTRfcMxIH.js.bin 8153 8440 (103.52%) 8194 (100.50%)
5sRTpHOwAvc.js.bin 22242 23007 (103.44%) 22900 (102.96%)
5vYiLw_dYIF.js.bin 8055 8342 (103.56%) 8105 (100.62%)
61p7yiXht2-.js.bin 50206 52256 (104.80%) 51889 (103.35%)
6JKpDbYzLnb.js.bin 6483 6589 (101.64%) 6403 ( 98.77%)
6JZhzyeEuCN.js.bin 8967 9315 (103.88%) 9276 (103.45%)
6KqUnE7jc9w.js.bin 7298 7492 (102.66%) 7282 ( 99.78%)
6O_ojd0SniS.js.bin 2125 2175 (102.35%) 2132 (100.33%)
6Vy5PMpF0Hg.js.bin 6056 6380 (105.35%) 6261 (103.39%)
6ZZ10cZ35bZ.js.bin 9899 10145 (102.49%) 10090 (101.93%)
6a1yJ8iDr5v.js.bin 1852 1887 (101.89%) 1824 ( 98.49%)
6d2pPNdw-Fq.js.bin 61748 63909 (103.50%) 63400 (102.68%)
6hfgpC9ic8F.js.bin 4110 4209 (102.41%) 4074 ( 99.12%)
6jeb8HxgHlP.js.bin 1083 1113 (102.77%) 1087 (100.37%)
794N6IR9E7.js.bin 12321 12810 (103.97%) 12531 (101.70%)
7BqLUPAKWMh.js.bin 1482 1515 (102.23%) 1395 ( 94.13%)
7E0Ot8Dy3-C.js.bin 6373 6606 (103.66%) 6478 (101.65%)
7HZgnEwNixO.js.bin 18245 19160 (105.20%) 18890 (103.54%)
7MLXMaSCgdy.js.bin 8713 8925 (102.43%) 8757 (100.50%)
7PfcIxnQ0TM.js.bin 3286 3407 (103.68%) 3343 (101.73%)
7SV69-MJz9d.js.bin 2300 2400 (104.35%) 2316 (100.70%)
7T4iM3nQFrv.js.bin 37291 38865 (104.22%) 40050 (107.40%)
7W2Y_j7-8XV.js.bin 7917 8202 (103.60%) 8079 (102.50%)
7d0Fr09BxRl.js.bin 14899 15232 (102.24%) 15024 (100.84%)
7hz53UCKK6t.js.bin 16424 16809 (102.34%) 16562 (100.84%)
7nilCjXkDsN.js.bin 1792 1809 (100.95%) 1759 ( 98.16%)
7rt70qPQEE4.js.bin 3435 3477 (101.22%) 3391 ( 98.72%)
80KtybBJSRq.js.bin 108805 110646 (101.69%) 111248 (102.25%)
8T10qOu6hA8.js.bin 12285 12780 (104.30%) 12531 (102.00%)
8YVRjv66YWN.js.bin 66536 69678 (104.72%) 71036 (106.76%)
8jdRgzkLyxN.js.bin 922 935 (101.41%) 948 (102.82%)
8vg_aNX2LTW.js.bin 7754 8079 (104.19%) 7870 (101.50%)
8yfRZjdcRpr.js.bin 33148 34132 (102.97%) 33829 (102.50%)
90KwH4neHw.js.bin 12031 12483 (103.76%) 12250 (101.82%)
93De-Q7NIMj.js.bin 2256 2288 (101.42%) 2233 ( 98.98%)
93PYB5GrxKi.js.bin 9943 10362 (104.21%) 10114 (101.72%)
99ZfiTBQjrD.js.bin 62218 63755 (102.47%) 63775 (102.50%)
9VUoI5Y8zgw.js.bin 90350 93890 (103.92%) 94459 (104.55%)
9itjkYhSqT7.js.bin 6264 6589 (105.19%) 6415 (102.41%)
9qcdbQ6N0vz.js.bin 782 848 (108.44%) 823 (105.24%)
A0nTvOA7SMq.js.bin 4718 4848 (102.76%) 4865 (103.12%)
ABozWUlfQQa.js.bin 1043 1058 (101.44%) 1046 (100.29%)
AhZnjAVEyh4.js.bin 7605 7986 (105.10%) 7868 (103.46%)
BSATYrRdEqw.js.bin 7613 7945 (104.36%) 7844 (103.30%)
BVuxfJBYYOy.js.bin 1096 1125 (102.65%) 1106 (100.91%)
BXCJygZcXT7.js.bin 11201 11619 (103.73%) 11331 (101.16%)
Ba5Z-3IbMrL.js.bin 8454 8709 (103.20%) 8469 (100.18%)
BdohC8uCF68.js.bin 51046 52128 (102.12%) 52163 (102.19%)
BhyGRBQE_ho.js.bin 4159 4295 (103.27%) 4137 ( 99.47%)
C0qPzI366B5.js.bin 5043 5162 (102.36%) 5177 (102.66%)
C4OnEvf_nsQ.js.bin 240 237 ( 98.75%) 237 ( 98.75%)
C4ZhzGqwQ96.js.bin 271963 279975 (102.95%) 285996 (105.16%)
CUH2YLmBB-4.js.bin 978 1035 (105.83%) 958 ( 97.96%)
Cq1EP4fUDbJ.js.bin 11467 11918 (103.93%) 11670 (101.77%)
CqyQckU2yo-.js.bin 9787 10168 (103.89%) 9954 (101.71%)
CvrtsqyNpWn.js.bin 6396 6516 (101.88%) 6474 (101.22%)
D4_q4AHaGdu.js.bin 8026 8152 (101.57%) 8013 ( 99.84%)
DBK4UL3WBG8.js.bin 1106 1163 (105.15%) 1126 (101.81%)
DGbUzkF4VDV.js.bin 148259 152801 (103.60%) 155838 (105.11%)
DJPPrzy7ff0.js.bin 5098 5328 (104.51%) 5179 (101.59%)
DJoMe0pettz.js.bin 27973 28523 (101.97%) 28375 (101.44%)
DKXUXo55mVb.js.bin 50285 51309 (102.40%) 51737 (102.89%)
DMoeyukxtiM.js.bin 21497 22177 (103.16%) 22263 (103.56%)
Dl0PcFY34M0.js.bin 713 754 (105.75%) 727 (101.96%)
Dt-P-nVSHp5.js.bin 19374 19567 (101.00%) 19268 ( 99.45%)
E6NydFiqMUh.js.bin 37517 39162 (104.38%) 40122 (106.94%)
EHU1s5cIT8a.js.bin 2334 2372 (101.63%) 2307 ( 98.84%)
EKPd8PWN5Sp.js.bin 10286 10402 (101.13%) 10315 (100.28%)
ES-H08hCe4R.js.bin 5175 5342 (103.23%) 5275 (101.93%)
ESNQ6wg-wVN.js.bin 13599 14155 (104.90%) 13935 (102.47%)
EXzH2oCF_WG.js.bin 11339 11819 (104.23%) 11526 (101.65%)
EenddqKIgxS.js.bin 737 733 ( 99.46%) 742 (100.68%)
Em3Kwiq345.js.bin 137995 142256 (103.90%) 143110 (103.71%)
EszNqiGd7DO.js.bin 5358 5480 (102.28%) 5366 (100.15%)
F13y1-OWrnG.js.bin 12371 12619 (102.00%) 12330 ( 99.67%)
FJbKFvWbYaR.js.bin 23969 24895 (103.86%) 24568 (102.50%)
FKhLGLrBw1b.js.bin 1918 1942 (101.25%) 1908 ( 99.48%)
FMmQ8tLHqBs.js.bin 10675 10884 (101.96%) 10581 ( 99.12%)
Fd1U8zdftY3.js.bin 600 611 (101.83%) 612 (102.00%)
FhECec447P7.js.bin 33137 34031 (102.70%) 33785 (101.96%)
FkHFmz2MgoW.js.bin 17624 18050 (102.42%) 17823 (101.13%)
FncnEVXq99f.js.bin 8922 9157 (102.63%) 8889 ( 99.63%)
FrMEaAgWus2.js.bin 92661 96184 (103.80%) 97644 (105.38%)
FtuH2CvvurL.js.bin 5408 5578 (103.14%) 5585 (103.27%)
FxBP7B2sGBz.js.bin 3348 3433 (102.54%) 3366 (100.54%)
G5FY0VSm5rb.js.bin 16597 17294 (104.20%) 16894 (101.79%)
GDWAoyDy3hk.js.bin 50920 51748 (101.63%) 51591 (101.32%)
GIvAi-mB56o.js.bin 18587 18802 (101.16%) 18524 ( 99.66%)
GMRb_AMn-2p.js.bin 16569 17221 (103.94%) 16834 (101.60%)
GWsQFO08Vaf.js.bin 35977 37186 (103.36%) 36690 (101.98%)
GZCn_tb8T0E.js.bin 19020 19392 (101.96%) 19075 (100.29%)
GbIFnbD5Hv3.js.bin 67352 70070 (104.40%) 70067 (104.30%)
Gj7hCH1JioK.js.bin 13058 13475 (103.19%) 13337 (102.14%)
GjhkPHrsA8I.js.bin 6706 6855 (102.22%) 6768 (100.92%)
Gr9UAc7ZBc_.js.bin 3147 3336 (106.10%) 3220 (102.32%)
Gw9cK2Tuwl9.js.bin 8063 8459 (104.91%) 8468 (105.20%)
Gwob1yTgNVa.js.bin 1717 1769 (103.30%) 1739 (101.28%)
Gzn5CpwLcFS.js.bin 23827 24384 (102.34%) 24422 (102.50%)
H0uohApc3Wi.js.bin 7675 7874 (102.59%) 7820 (101.89%)
H9yXsupYfcw.js.bin 7361 7631 (103.67%) 7417 (100.76%)
HJ9V2pE20_V.js.bin 2307 2355 (102.80%) 2294 ( 99.44%)
HKfkAAAZWLi.js.bin 10615 10858 (102.29%) 10852 (102.23%)
HNpOKWxxC6P.js.bin 8489 8626 (101.61%) 8446 ( 99.49%)
HUooP0yYGcn.js.bin 4162 4291 (103.10%) 4117 ( 98.92%)
HZ7rSN33iS3.js.bin 3208 3289 (102.52%) 3180 ( 99.13%)
Hckh2Rwyksb.js.bin 2641 2703 (102.35%) 2566 ( 97.16%)
HixpLV37dk5.js.bin 2105 2181 (103.61%) 2150 (102.14%)
Hn7pnMt7VJh.js.bin 31782 32500 (102.26%) 32958 (103.70%)
Htir4jq4jA2.js.bin 27063 27890 (103.60%) 27738 (102.49%)
IO4DIs8nJvq.js.bin 1311 1355 (103.36%) 1320 (100.69%)
IPYcVQm46mM.js.bin 8105 8479 (104.61%) 8257 (101.88%)
Ic0YY_ezK2_.js.bin 15536 15660 (100.80%) 15511 ( 99.84%)
Isozbn0R2c4.js.bin 107336 109454 (101.97%) 109868 (102.36%)
J3DZf3he5wl.js.bin 4690 4812 (102.60%) 4657 ( 99.30%)
JfqV8F6NUEv.js.bin 24139 24718 (102.40%) 24487 (101.44%)
JoKRIsLB0xX.js.bin 1103 1096 ( 99.37%) 1096 ( 99.37%)
Jyt80_jhkWr.js.bin 3360 3410 (101.49%) 3405 (101.34%)
K5NL9ldpeQ3.js.bin 50366 51452 (102.16%) 51888 (103.20%)
K66_o184Q7u.js.bin 3804 3900 (102.52%) 3846 (101.10%)
KOF4Tki1I_F.js.bin 98060 101245 (103.25%) 103866 (105.92%)
KjsolWwRYEq.js.bin 1764 1797 (101.87%) 1750 ( 99.21%)
Kkz0EfPx878.js.bin 10467 10913 (104.26%) 10662 (101.86%)
Ktjd_zCDiuX.js.bin 363 368 (101.38%) 356 ( 98.70%)
Ktjh1pUNe33.js.bin 15581 15983 (102.58%) 15710 (100.83%)
Kzw0nPC0dcW.js.bin 6696 6932 (103.52%) 6802 (101.58%)
L7jog_bXY11.js.bin 15194 15571 (102.48%) 15358 (101.80%)
LIaBBJHFKhS.js.bin 10186 10422 (102.32%) 10131 ( 99.46%)
LMWFQY0i5pg.js.bin 89784 93250 (103.86%) 93646 (104.30%)
LNGlQuUqYFf.js.bin 1145 1226 (107.70%) 1140 ( 99.56%)
LbdkYWnUPHH.js.bin 91615 93272 (101.81%) 93824 (102.41%)
LnPHIQfkPgJ.js.bin 152562 157387 (103.16%) 159612 (104.62%)
LquvhDYHVN_.js.bin 19327 19842 (102.66%) 19459 (100.68%)
M2wrki-9YRq.js.bin 1188 1168 ( 98.32%) 1206 (101.52%)
M4oEcTjMPIq.js.bin 17643 18344 (103.97%) 17980 (101.91%)
M6KutcQQx1J.js.bin 16539 16692 (100.93%) 16457 ( 99.50%)
MaQq-zjKQ9b.js.bin 37548 39031 (103.95%) 40085 (106.76%)
Mk-Oc98fTbS.js.bin 8993 9112 (101.32%) 9075 (100.91%)
MwM-eWv6gvY.js.bin 631 627 ( 99.37%) 648 (102.69%)
N7JSRkYI74q.js.bin 4803 5040 (104.93%) 4853 (101.40%)
N8rUHyk5FDI.js.bin 362 367 (101.38%) 367 (101.38%)
NEMgFJiGr4m.js.bin 6384 6517 (102.80%) 6340 ( 99.31%)
NP0I7Z0wjpg.js.bin 19345 19656 (101.61%) 19376 (100.16%)
NQRnhW7HDwe.js.bin 5804 5936 (102.27%) 6000 (103.38%)
NhNKyiBLxFg.js.bin 931 953 (102.36%) 929 ( 99.79%)
NjPxAnvhaki.js.bin 7813 7965 (101.95%) 7770 ( 99.45%)
Nk1WKHlapUy.js.bin 14924 15523 (104.10%) 15285 (102.42%)
NpdlEKtiuKA.js.bin 7227 7364 (101.90%) 7188 ( 99.46%)
O0D5wz4enLV.js.bin 882 934 (105.90%) 866 ( 98.19%)
O4w_IQdYWJc.js.bin 19597 20060 (102.36%) 19769 (100.88%)
OBF7Dw0EY2M.js.bin 5671 5830 (102.80%) 5674 (100.50%)
OTbMrmdj35V.js.bin 92107 93913 (101.96%) 94478 (102.57%)
OfXHPJrGMDY.js.bin 40772 41434 (101.62%) 41524 (101.84%)
P67MB8xnjDM.js.bin 133073 137469 (103.30%) 139557 (104.87%)
PQ2A0ClEJeK.js.bin 43595 45186 (103.65%) 44739 (102.62%)
PRW6RA4PqsI.js.bin 8561 8711 (101.75%) 8526 ( 99.59%)
PikWb-DRclC.js.bin 30076 31215 (103.79%) 30919 (102.80%)
Q-wSNUwkAAw.js.bin 4147 4329 (104.39%) 4327 (104.34%)
Q0bfmxur30T.js.bin 8992 9173 (102.10%) 9078 (100.96%)
QWha_PeC1CG.js.bin 9994 10205 (102.11%) 9989 ( 99.95%)
QYPVIGjITMs.js.bin 9612 10016 (104.20%) 9758 (101.52%)
QjHlKARx4Jl.js.bin 12446 12706 (102.90%) 12438 ( 99.94%)
QnG5Ibr3nN0.js.bin 7107 7301 (102.73%) 7184 (101.80%)
R2saIb6hdbu.js.bin 5086 5337 (104.94%) 5229 (102.81%)
RYStwPOyGt_.js.bin 2371 2433 (102.61%) 2359 ( 99.49%)
RixyttL2E_J.js.bin 8555 8714 (101.86%) 8506 ( 99.43%)
RwNV_ZmMYXx.js.bin 9516 9817 (103.16%) 9595 (100.83%)
SEFS3r1PhQT.js.bin 8987 9339 (103.92%) 9120 (101.48%)
SK1wkfeDq_h.js.bin 5227 5399 (103.29%) 5298 (101.36%)
SXpXdn7Gm3h.js.bin 301 306 (101.66%) 303 (100.66%)
SaMs8YKhI12.js.bin 5828 5917 (101.53%) 5779 ( 99.16%)
Sat5Kefz26k.js.bin 52050 53495 (102.78%) 53612 (103.00%)
Sls0fuKBDLY.js.bin 14062 14619 (103.96%) 14302 (101.71%)
Stwps2pjgwA.js.bin 2140 2201 (102.85%) 2130 ( 99.53%)
T1SAR34A6Ia.js.bin 13312 13514 (101.52%) 13784 (103.55%)
TG5ISf94Tpi.js.bin 6275 6561 (104.56%) 6366 (101.45%)
TNaDhPViNKe.js.bin 22743 23415 (102.95%) 23089 (101.52%)
TQ2Bb_pkfns.js.bin 6971 7203 (103.33%) 7035 (100.92%)
TRjk530Uq-4.js.bin 285 296 (103.86%) 282 ( 98.95%)
TWZbP8KTL1Y.js.bin 6165 6316 (102.45%) 6103 ( 98.99%)
TX5joFGrGzI.js.bin 29239 29646 (101.39%) 29465 (100.77%)
TZauC440LtG.js.bin 8943 9133 (102.12%) 9023 (100.89%)
Tl6W2-vTx42.js.bin 9852 10251 (104.50%) 9995 (101.45%)
TmjxlwXMF5n.js.bin 7091 7323 (103.27%) 7120 (100.41%)
Tn6It5Tj6cj.js.bin 1324 1372 (103.63%) 1312 ( 99.90%)
TpuLn4b3sfe.js.bin 14544 14994 (103.90%) 14718 (101.20%)
U7Q-LP06v2T.js.bin 653 696 (106.58%) 677 (103.68%)
UG4i5MCq6gc.js.bin 18615 19587 (105.22%) 19205 (103.17%)
UNo8Jh862bD.js.bin 45136 45869 (101.62%) 45806 (101.48%)
US5MebMc5rZ.js.bin 764 771 (100.92%) 751 ( 98.30%)
UrEYIrNxED2.js.bin 11756 11943 (101.59%) 11792 (100.31%)
V70FxA0Lwxv.js.bin 847 861 (101.65%) 850 (100.35%)
VC34lfZRcL-.js.bin 13628 14070 (103.24%) 13883 (101.87%)
VICqwwD8Kdu.js.bin 49541 50581 (102.10%) 50473 (101.88%)
VJwyx7mgPzM.js.bin 7562 7701 (101.84%) 7512 ( 99.34%)
VM49E-CNN5.js.bin 23293 24189 (103.85%) 23903 (102.62%)
VZ1-qzzy-qS.js.bin 8481 8824 (104.40%) 8636 (101.83%)
VdWwIQ0QYkA.js.bin 10291 10739 (104.35%) 10637 (103.36%)
VfL_uzuaCDL.js.bin 52245 54075 (103.50%) 53782 (102.94%)
VkGQgXkt7PT.js.bin 67029 68587 (102.32%) 68944 (102.86%)
VlyqIhqDfTj.js.bin 2725 2784 (102.17%) 2716 ( 99.67%)
VmHp78YPiBO.js.bin 2067 2143 (103.68%) 2067 (100.00%)
VnfzLsPn6WI.js.bin 4929 5150 (104.48%) 5015 (101.74%)
VqKl4ypOdsk.js.bin 409 412 (100.73%) 409 (100.00%)
VzTqr5yULRw.js.bin 3060 3130 (102.29%) 2985 ( 97.55%)
W26NPhE5J26.js.bin 279 295 (105.73%) 291 (104.30%)
W_OjOIf_p9e.js.bin 31827 32250 (101.33%) 32093 (100.84%)
Wboj_Y1w3Yu.js.bin 7975 8171 (102.46%) 7988 (100.16%)
WdEpJk_Tabb.js.bin 5764 5883 (102.60%) 5754 ( 99.83%)
WhA-dvAnZjd.js.bin 2113 2137 (101.14%) 2082 ( 98.53%)
WjlhBpoVp6U.js.bin 953 988 (103.67%) 952 ( 99.90%)
Wm9D4-HKkyS.js.bin 1491 1509 (101.21%) 1515 (101.61%)
WwrtoTdWMcs.js.bin 737 757 (102.71%) 752 (102.40%)
Wy7ru1prJ9p.js.bin 31933 32519 (101.84%) 32689 (102.37%)
WyHjBg4RHih.js.bin 7819 8092 (103.49%) 7885 (100.84%)
X6ft_Rw2FGg.js.bin 48833 50198 (102.80%) 50063 (102.52%)
X6xoI89q7wL.js.bin 4460 4566 (102.38%) 4447 ( 99.71%)
X70FmpmPl6U.js.bin 30955 32173 (103.93%) 31958 (103.24%)
X9KwaXAh8OA.js.bin 90068 93363 (103.66%) 94918 (105.38%)
XCZjSu6H2x6.js.bin 26293 27121 (103.15%) 26988 (102.64%)
XCaeFT9FnaM.js.bin 113207 115430 (101.96%) 115684 (102.19%)
XEfmAEzPxw5.js.bin 986 991 (100.51%) 962 ( 97.57%)
XJJST1rrUzj.js.bin 56768 58985 (103.91%) 58668 (103.35%)
Xajl4Z-HMDd.js.bin 5533 5652 (102.15%) 5535 (100.40%)
XvJRVgBJV33.js.bin 12844 13430 (104.56%) 13181 (102.62%)
XyxjBO0E-Ae.js.bin 8546 8922 (104.40%) 8687 (101.65%)
Y1-O927cK2t.js.bin 893 898 (100.56%) 916 (102.58%)
Y2wwAspvKK7.js.bin 1567 1596 (101.85%) 1581 (100.89%)
YB2lc13pFUI.js.bin 6140 6282 (102.31%) 6111 ( 99.53%)
YHDRl984zRW.js.bin 1822 1755 ( 96.32%) 1694 ( 92.97%)
YJgePE2Auby.js.bin 5747 5977 (104.00%) 5822 (101.31%)
YUkQ93LjurT.js.bin 8125 8353 (102.81%) 8185 (100.74%)
Yg3McEltNjp.js.bin 24221 24743 (102.16%) 24815 (102.45%)
YjDPNITqxhG.js.bin 268575 275980 (102.76%) 282382 (105.14%)
Z14o104Q29.js.bin 13118 13653 (104.80%) 13364 (101.88%)
Z2Kbh46LXZZ.js.bin 1966 2014 (102.44%) 1951 ( 99.24%)
Z5QGTLkc9kz.js.bin 73201 76472 (104.47%) 78935 (107.83%)
ZOeRtUIf90F.js.bin 20116 20733 (103.70%) 20414 (101.48%)
ZZfnEdoRgj0.js.bin 6183 6449 (104.30%) 6300 (101.89%)
ZbAYJatCyNZ.js.bin 69839 70862 (101.46%) 70817 (101.40%)
_0Gjqmw4Koo.js.bin 619 642 (103.72%) 617 ( 99.68%)
_3UqloxC0CW.js.bin 9488 9730 (102.55%) 9502 (100.15%)
_5tA9NCCPxQ.js.bin 12531 12791 (102.70%) 12528 ( 99.98%)
_HRTS9FAj70.js.bin 1413 1461 (103.40%) 1416 (100.21%)
_Kp2t1yMvG5.js.bin 25545 26463 (103.59%) 26343 (103.12%)
_obLyhmNwMA.js.bin 1432 1457 (101.75%) 1419 ( 99.90%)
_poPb23m2lW.js.bin 4590 4721 (102.85%) 4553 ( 99.19%)
a1OMwkLkxJy.js.bin 6445 6596 (102.34%) 6534 (101.38%)
a4sxR7Xqudt.js.bin 5239 5478 (104.56%) 5344 (102.00%)
aCUxlUPri-u.js.bin 14657 15324 (104.55%) 15092 (102.97%)
aCkKlOuYdAY.js.bin 423 429 (101.42%) 417 ( 98.58%)
aVmpARAebEj.js.bin 4215 4412 (104.67%) 4370 (103.68%)
afJUg3Oi2lo.js.bin 71159 71997 (101.18%) 72042 (101.24%)
ahTNFjdbHPj.js.bin 4700 4868 (103.57%) 4749 (101.40%)
akZ6romKPXS.js.bin 11849 12385 (104.52%) 12145 (102.50%)
aoGTCgaRyfh.js.bin 28577 29004 (101.49%) 28614 (100.13%)
bj2quEws0mS.js.bin 2966 3065 (103.34%) 3007 (101.38%)
bjY7glcVhij.js.bin 2455 2466 (100.45%) 2422 ( 98.66%)
bkspEohDWXI.js.bin 50795 52482 (103.32%) 52392 (103.14%)
bujNxQtqJfH.js.bin 12081 12572 (104.60%) 12267 (101.54%)
c0Qh0douchE.js.bin 6439 6719 (104.35%) 6594 (102.41%)
c4s5MUgdk4p.js.bin 8007 8174 (102.90%) 8097 (101.12%)
cEJxYmeVQFY.js.bin 14505 15110 (104.17%) 15119 (104.23%)
cLLmKH52bRw.js.bin 52889 54240 (102.55%) 54377 (102.81%)
cM2rbC5wYIj.js.bin 35313 35946 (101.79%) 36126 (102.30%)
ctOKVDJPRdx.js.bin 9492 9693 (102.12%) 9554 (100.65%)
d1P6nQfYXKD.js.bin 4104 4223 (102.90%) 4076 ( 99.32%)
d3u2FAOQCGP.js.bin 847 804 ( 94.92%) 775 ( 91.50%)
dQuPAucPva9.js.bin 2791 2905 (104.80%) 2777 ( 99.50%)
dhzIxMhJmUQ.js.bin 3420 3492 (102.11%) 3459 (101.14%)
dijW0z0ucm-.js.bin 17228 17519 (101.69%) 17302 (100.43%)
dr6aBMYM2V2.js.bin 45964 47307 (102.92%) 47178 (102.64%)
drJc7buhOtE.js.bin 18384 18770 (102.10%) 18609 (101.22%)
e7ra_72iXdV.js.bin 10335 10779 (104.30%) 10841 (104.90%)
eIJD4bQ5h-1.js.bin 5012 5145 (102.65%) 5004 ( 99.84%)
eZjQt87u8t1.js.bin 20004 20643 (103.19%) 20226 (101.11%)
eop6dvOMLTq.js.bin 41123 41900 (101.89%) 41914 (101.92%)
f2TlpbviZca.js.bin 7366 7695 (104.47%) 7520 (102.90%)
f53VGV2iT0I.js.bin 5267 5462 (103.70%) 5339 (101.37%)
fB_MiIP7NHd.js.bin 1648 1681 (102.00%) 1737 (105.40%)
fG-u4hmOE3Q.js.bin 10776 11117 (103.16%) 10914 (101.28%)
fLczu6LzRHG.js.bin 31511 32486 (103.90%) 32226 (102.27%)
fNXDCJu-V4g.js.bin 702 718 (102.28%) 692 ( 98.58%)
fSg_zxv_Kod.js.bin 2997 3086 (102.97%) 2997 (100.00%)
fVXETrH3Cq8.js.bin 10357 10616 (102.50%) 10546 (101.82%)
fW2Il-BeJSj.js.bin 7973 8115 (101.78%) 7896 ( 99.30%)
fws9yWG1beh.js.bin 67898 70947 (104.49%) 73099 (107.66%)
g-8q9Q5HAct.js.bin 8030 8468 (105.45%) 8587 (106.94%)
gDiLJ3Nh5Qk.js.bin 342 353 (103.22%) 344 (100.58%)
gKycf5rW4sU.js.bin 43371 44904 (103.53%) 44651 (102.95%)
gN5Uk4UvOZT.js.bin 3477 3574 (102.79%) 3485 (100.23%)
gV8VMvIotf7.js.bin 10413 10590 (101.70%) 10388 ( 99.76%)
gvMgr3QKvQ8.js.bin 19101 19683 (103.50%) 19431 (101.73%)
gvv_dhImewC.js.bin 9319 9638 (103.42%) 9423 (101.12%)
h-DmjvKXmR3.js.bin 13947 14002 (100.39%) 13842 ( 99.25%)
hNVa87u4kKS.js.bin 7543 7824 (103.73%) 8116 (107.60%)
hZKxYrkgzm4.js.bin 12816 13291 (103.71%) 12975 (101.24%)
hba0DhGeB63.js.bin 15820 16397 (103.65%) 16104 (101.80%)
hgw22H68N_l.js.bin 397899 405736 (101.97%) 412286 (103.62%)
hrAa4KpUxy3.js.bin 16592 17313 (104.35%) 17089 (103.00%)
huzeXy9rojb.js.bin 7701 7890 (102.45%) 7708 (100.90%)
i-G9J9_yx_.js.bin 1782 1775 ( 99.61%) 1727 ( 96.91%)
i-WhEfiHI2C.js.bin 47182 48470 (102.73%) 48657 (103.13%)
i0Ny6SirQCR.js.bin 9258 9525 (102.88%) 9336 (100.84%)
i1qU3JKPcMP.js.bin 12371 12807 (103.52%) 12567 (101.58%)
i6YOUjNhJvc.js.bin 7512 7679 (102.22%) 7600 (101.17%)
iEgP7Vv3MrH.js.bin 50477 51197 (101.43%) 51250 (101.53%)
iIIyGDRwUkJ.js.bin 6763 6886 (101.82%) 6665 ( 98.55%)
iLiXoQia_Hl.js.bin 17430 17920 (102.81%) 17589 (100.91%)
iOxfHlfLzMO.js.bin 1422 1443 (101.48%) 1389 ( 97.68%)
iVpq6no1sbf.js.bin 8281 8553 (103.28%) 8351 (100.85%)
ibLV5-CufiC.js.bin 3246 3295 (101.51%) 3194 ( 98.40%)
ie0_9cKR_JF.js.bin 1513 1556 (102.84%) 1523 (100.66%)
igvPVIqEFTP.js.bin 2876 2988 (103.89%) 2847 ( 98.99%)
jbEWliZOUvS.js.bin 2672 2718 (101.72%) 2700 (101.50%)
jgEfwJ3Zhg9.js.bin 8636 9064 (104.96%) 9014 (104.38%)
joglmeAu8Qv.js.bin 6193 6404 (103.41%) 6285 (101.49%)
jtWi6ZdW2bb.js.bin 2539 2646 (104.21%) 2569 (101.18%)
kKvdSscVFrv.js.bin 568 570 (100.35%) 550 ( 96.83%)
kPs479ufKyg.js.bin 166901 167425 (100.31%) 166973 (100.40%)
kZto1c9w8wC.js.bin 978 1021 (104.40%) 1000 (102.25%)
kfpHN646tDg.js.bin 2410 2498 (103.65%) 2380 ( 98.76%)
lVlq0lDwsF8.js.bin 17955 18078 (100.69%) 17865 ( 99.50%)
lVt5GmUyc2l.js.bin 80131 81743 (102.10%) 82006 (102.34%)
lg3_xIkYYKW.js.bin 19123 19661 (102.81%) 19293 (100.89%)
llDDV2rQ1Rz.js.bin 15456 15619 (101.50%) 15438 ( 99.88%)
lqQs_yLOe3X.js.bin 37346 38418 (102.87%) 38186 (102.25%)
mCXECSTwhF8.js.bin 17894 18117 (101.25%) 17842 ( 99.71%)
mWseK_aEOAs.js.bin 3479 3517 (101.90%) 3501 (100.63%)
mcqy6o0Vfti.js.bin 6842 7122 (104.90%) 6975 (101.94%)
mtYekkRCibq.js.bin 7204 7351 (102.40%) 7218 (100.19%)
n9Nfrfb5B_x.js.bin 13318 13847 (103.97%) 13539 (101.66%)
nI5PnHjEJyf.js.bin 31638 32254 (101.95%) 32094 (101.44%)
nMMjSesfBr4.js.bin 30382 31535 (103.80%) 31276 (102.94%)
nsO8_i4Z7F5.js.bin 4875 4980 (102.15%) 4844 ( 99.36%)
o7BSlkO7w7m.js.bin 2318 2296 ( 99.50%) 2207 ( 95.21%)
o7G0EczmD-6.js.bin 89991 93082 (103.43%) 92774 (103.90%)
o7wHnb8FJMh.js.bin 1972 2047 (103.80%) 1968 ( 99.80%)
oBpvYNZ2__3.js.bin 29781 30704 (103.10%) 30535 (102.53%)
oPUD9dTegEK.js.bin 17637 18356 (104.80%) 18015 (102.14%)
ocuLS-3Ixa4.js.bin 2249 2303 (102.40%) 2228 ( 99.70%)
odqwSDzA1Jh.js.bin 6722 7045 (104.81%) 6798 (101.13%)
op6CSGtgdwj.js.bin 6766 6884 (101.74%) 6714 ( 99.23%)
p4ThgvBjGnC.js.bin 13630 14009 (102.78%) 13787 (101.15%)
pDkCdAFgfot.js.bin 6742 6866 (101.84%) 6696 ( 99.32%)
pJi5pUFExsr.js.bin 3688 3768 (102.17%) 3661 ( 99.27%)
q1aykcXg1rv.js.bin 274991 283221 (102.99%) 289301 (105.20%)
q77Ec0mnQGL.js.bin 34736 35457 (102.80%) 35305 (101.64%)
q7gCUANZcPB.js.bin 4244 4329 (102.00%) 4196 ( 98.87%)
q9l7bWgIkZv.js.bin 1913 1967 (102.82%) 1949 (101.88%)
qH44GCO2FNp.js.bin 1576 1611 (102.22%) 1560 ( 98.98%)
qMTnOD4x9ke.js.bin 6217 6438 (103.55%) 6348 (102.11%)
qWlf3ZNfI4H.js.bin 2139 2193 (102.52%) 2133 ( 99.72%)
q_nM_ysfdDG.js.bin 24871 25552 (102.74%) 25646 (103.12%)
qbjgvCcmoa9.js.bin 7711 8006 (103.83%) 7885 (102.26%)
qwqtBhLzQ8Y.js.bin 166793 167302 (100.31%) 167048 (100.15%)
qzNly4KQTtU.js.bin 14025 14636 (104.36%) 14309 (102.20%)
r37ueZKpGKr.js.bin 8150 8534 (104.71%) 8578 (105.25%)
rL4GrOjmDqx.js.bin 3897 4066 (104.34%) 3953 (101.44%)
rc8ZmQFeS-7.js.bin 561 583 (103.92%) 572 (101.96%)
rcq57bhogFy.js.bin 860 878 (102.90%) 899 (104.53%)
rlnOA5ljgx-.js.bin 10421 10707 (102.74%) 10654 (102.24%)
sDDEkC3xfE8.js.bin 89039 92014 (103.34%) 93296 (104.78%)
sON6H-TQTSe.js.bin 12297 12601 (102.47%) 12267 ( 99.76%)
sa9HdQH-ogl.js.bin 9696 9915 (102.26%) 9679 ( 99.82%)
su4qR9fQIup.js.bin 56075 58256 (103.89%) 57915 (103.28%)
szg5Oa3h81p.js.bin 2981 3087 (103.56%) 2952 ( 99.30%)
t6u1v8nErjo.js.bin 4649 4734 (101.83%) 4607 ( 99.10%)
tJSUYmYnSG7.js.bin 29531 30553 (103.46%) 30449 (103.11%)
tPmvSn_H54o.js.bin 68514 69925 (102.60%) 70512 (102.92%)
tTkC-lq6BfZ.js.bin 141791 145998 (102.97%) 146613 (103.40%)
t_FKyKc7pIv.js.bin 97776 101310 (103.61%) 101520 (103.83%)
tia2K7wPxLm.js.bin 300 309 (103.00%) 299 ( 99.67%)
tjI9cfQd8sN.js.bin 15693 16090 (102.53%) 15870 (101.13%)
tyuO9ARb66s.js.bin 5625 5789 (102.92%) 5790 (102.93%)
uH6g_g6fsUY.js.bin 583 603 (103.43%) 582 ( 99.83%)
uHyvALTm4SZ.js.bin 35548 36946 (103.93%) 36775 (103.45%)
uMjfl_iCR.js.bin 1246 1389 (111.48%) 1224 ( 98.23%)
uil7OtZB_-S.js.bin 6097 6437 (105.58%) 6308 (103.46%)
ujVr5juu70U.js.bin 41940 42728 (101.88%) 42530 (101.41%)
unVYcbptooG.js.bin 1494 1533 (102.61%) 1419 ( 94.98%)
v-e81b_ymxI.js.bin 27534 28455 (103.34%) 28324 (102.87%)
v7nEa2wTc3k.js.bin 726 766 (105.51%) 763 (105.10%)
v8xeS_mshuc.js.bin 596 590 ( 98.99%) 606 (101.68%)
vJA96NEnKK9.js.bin 9756 9969 (102.18%) 9788 (100.33%)
vYjO33iDmup.js.bin 10518 10911 (103.74%) 10707 (101.80%)
v_23FSndsFZ.js.bin 143809 147937 (102.87%) 148946 (103.57%)
vdc-skgedI5.js.bin 63034 64791 (102.79%) 64715 (102.67%)
vrkEs5HVSSU.js.bin 2828 2891 (102.23%) 2859 (101.10%)
w04cnnPb3Tg.js.bin 5940 6079 (102.34%) 5983 (100.72%)
w4Vt3niBWH1.js.bin 5790 6011 (103.82%) 5941 (102.61%)
wTxM8iBz3c2.js.bin 8921 9328 (104.56%) 9137 (102.42%)
wfQF2RfxAkc.js.bin 1386 1361 ( 98.20%) 1334 ( 96.25%)
whJhzNwHJHe.js.bin 3191 3302 (103.48%) 3238 (101.47%)
x2UpV3umnzT.js.bin 17943 18140 (101.10%) 18139 (101.90%)
x4eLrd85LUN.js.bin 28037 28471 (101.55%) 28324 (101.20%)
x6-eCrmd1V9.js.bin 1350 1427 (105.70%) 1339 ( 99.19%)
xBmgk_xc-wG.js.bin 34797 36130 (103.83%) 35986 (103.42%)
xJS31ptCQbq.js.bin 279 290 (103.94%) 276 ( 98.92%)
xMrUJfyTCUg.js.bin 2230 2285 (102.47%) 2218 ( 99.46%)
xRT6_TBJBkw.js.bin 5437 5651 (103.94%) 5544 (101.97%)
xZNnvjL182T.js.bin 17518 18237 (104.10%) 17869 (102.00%)
x_EKm-Adi7Q.js.bin 21152 21957 (103.81%) 21696 (102.57%)
xdlg631Qxdp.js.bin 1425 1463 (102.67%) 1407 ( 98.74%)
xkpiQ5LiVoI.js.bin 1398 1388 ( 99.28%) 1401 (100.21%)
xo_xIOh_juS.js.bin 4597 4712 (102.50%) 4586 ( 99.76%)
xsyj3Z3mAiS.js.bin 11407 11614 (101.81%) 11407 (100.00%)
y-05tHqVj8P.js.bin 34247 35712 (104.28%) 35520 (103.72%)
y5R7cnYctJv.js.bin 1615 1555 ( 96.28%) 1565 ( 96.90%)
yKclkKsDE2H.js.bin 687 690 (100.44%) 659 ( 95.92%)
yRA0kDZHvwL.js.bin 440 446 (101.36%) 457 (103.86%)
y_L9TziFm87.js.bin 9665 9854 (101.96%) 9645 ( 99.79%)
ydXXbqxw_8l.js.bin 951 983 (103.36%) 925 ( 97.27%)
ynWyXLLi-Ij.js.bin 609 610 (100.16%) 610 (100.16%)
zC72O8ihM25.js.bin 5300 5487 (103.53%) 5366 (101.25%)
zXID_s3O-Wa.js.bin 16472 16964 (102.99%) 16675 (101.23%)
zlhcYuPRsUE.js.bin 7450 7702 (103.38%) 7524 (100.99%)
zm0DxINWRmX.js.bin 36167 36997 (102.29%) 36758 (101.63%)
zuCQsug4SlJ.js.bin 9711 9922 (102.17%) 9738 (100.28%)
zxjnIvNA5Hu.js.bin 6821 7045 (103.28%) 6935 (101.67%)

@vdjeric
Copy link

vdjeric commented May 21, 2019

Thank you for running this analysis @arai-a

@arai-a
Copy link
Author

arai-a commented May 21, 2019

there are 2 problems that this is supposed to solve:

  • handling of 0x00 (terminator)
    • to get each string, file decoder should look for 0x00 through the entire string dictionary (possibly incrementally while decoding brotli stream)
    • if the length is known, decoder just have to read that amount of bytes (of course, the decoder should read the length first, that is varint, but that's simpler and the code already used in other places)
  • handling of 0x01 (escape sequence)
    • to decode 0x01-escaped 0x01 and 0x00 code point, file decoder should look for 0x01 through the entire string dictionary, even if there's no escaped 0x01 or 0x00 in strings
    • also, replacing 0x01-escaped 0x01/0x00 to raw 0x01/0x00 isn't quick operation

the latter issue about "even if there's no escaped 0x01 or 0x00 in strings" can be solved by separating dictionary for strings that contains 0x01/0x00 and others. but having 2 dictionary structure just for that purpose doesn't sound reasonable for me.
(I hope there's better solution...)

then, there is the following thing that in either way the decoder have to do:

  • check the validity of WTF-8 encoding of the strings
    • to handle lone surrogate in source JS file, BinAST file should use WTF-8 instead of UTF-8
    • when reading from file, the decoder should throw if there's any string that isn't valid WTF-8
  • decode to UTF-16 if necessary
    • at least SpiderMonkey is not using WTF-8 internally, so any strings that contains lone surrogate should be decoded to UTF-16 (actually, any strings that is non-ASCII should)

So, in any way we have to iterate over the strings at least once (or twice if we decode),
and 0x00 and 0x01 handling might be possible to be merged with that operation, but
WTF-8 validation and decoding are provided as separate fundamental functions in codebase, and it's better if we can avoid modifying that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dominiccooney dominiccooney Awaiting requested review from dominiccooney

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@arai-a @Yoric @vdjeric