hsynz

english | 中文版

hsynz is a library for delta update using sync algorithm, like zsync.
rsync over http(s); implement the sync algorithm on client side, and server side only need http(s) cdn. support compressor zstd & zlib, support large file & directory(folder), support muti-thread.

Recommended scenarios: Very large number of older versions or where older versions are not available (not saved or modified, etc.) so that all deltas cannot be calculated in advance.

The server uses hsync_make to process the latest version of the data once, generating a summary info file(hsyni) of the new version of the data in chunks, and optionally compressing the new version of the data in chunks to get the release file(hsynz), which would be the hsynz equivalent if the new version of the original file were not compressed.

The client first downloads the hsyni file from the server or another user's share, calculates the updated blocks it needs to download based on its old version, and learns the location of these blocks in hsynz based on the information in hsyni, selects a communication method to download them on demand from the server's hsynz file, and merges the downloaded blocks with the existing data locally to get the latest version of the data.

hsync_demo provides a test client demo for local file testing.
hsync_http provides a download client demo with http(s) support for sync update from a server that provides an http(s) file download service(e.g CDN, support HTTP/1.1 muti range Requests).
Tip: You can also customise other communication methods for sync.

---

Compare with zsync

• In addition to supporting source and target as files, support is also provided for directories(folders).
• In addition to supporting compressed release package by zlib; also supported zstd compressor, providing better compression ratio, i.e. smaller downloaded patch package.
• The server-side make support multi-threaded parallel acceleration.
• The client-side diff speed has been optimized, and also support multi-threaded parallel acceleration.

---

Releases/Binaries

Download from latest release : Command line app for Windows, Linux, MacOS; and .so lib for Android.
( release files build by projects in path hsynz/builds )

Build it yourself

Linux or MacOS X

$ cd <dir>
$ git clone --recursive https://github.com/sisong/hsynz.git
$ cd hsynz
$ make

Windows

$ cd <dir>
$ git clone --recursive https://github.com/sisong/hsynz.git

build hsynz/builds/vc/hsynz.sln with Visual Studio

libhsynz.so for Android

• install Android NDK
• $ cd <dir>/hsynz/builds/android_ndk_jni_mk
• $ build_libs_static.sh (or $ build_libs_static.bat on windows, then got *.so files)
• import file com/github/sisong/hsynz.java (from hsynz/builds/android_ndk_jni_mk/java/) & .so files, java code can call the sync patch function in libhsynz.so

---

hsync_make command line usage:

hsync_make: [options] newDataPath out_hsyni_file [out_hsynz_file]
  newDataPath can be file or directory(folder),
  if newDataPath is a file & no -c-... option, out_hsynz_file can empty.
options:
  -s-matchBlockSize
      matchBlockSize>=128, DEFAULT -s-2k, recommended 1024,4k,...
  -b-safeBit
      set allow patch fail hash clash probability: 1/2^safeBit;
      safeBit>=14, DEFAULT -b-24, recommended 20,32...
  -p-parallelThreadNumber
    DEFAULT -p-4;
    if parallelThreadNumber>1 then open multi-thread Parallel mode;
  -c-compressType[-compressLevel]
      set out_hsynz_file Compress type & level, DEFAULT uncompress;
      support compress type & level:
        -c-zlib[-{1..9}[-dictBits]]     DEFAULT level 9
            dictBits can 9--15, DEFAULT 15.
        -c-gzip[-{1..9}[-dictBits]]     DEFAULT level 9
            dictBits can 9--15, DEFAULT 15.
            compress by zlib, but out_hsynz_file is .gz file format.
        -c-zstd[-{10..22}[-dictBits]]   DEFAULT level 21
            dictBits can 15--30, DEFAULT 24.
  -C-checksumType
      set strong Checksum type for block data, DEFAULT -C-xxh128;
      support checksum type:
        -C-xxh128
        -C-md5
        -C-sha512
        -C-sha256
        -C-crc32
            WARNING: crc32 is not strong & secure enough!
  -n-maxOpenFileNumber
      limit Number of open files at same time when newDataPath is directory;
      maxOpenFileNumber>=8, DEFAULT -n-48, the best limit value by different
        operating system.
  -g#ignorePath[#ignorePath#...]
      set iGnore path list in newDataPath directory; ignore path list such as:
        #.DS_Store#desktop.ini#*thumbs*.db#.git*#.svn/#cache_*/00*11/*.tmp
      # means separator between names; (if char # in name, need write #: )
      * means can match any chars in name; (if char * in name, need write *: );
      / at the end of name means must match directory;
  -f  Force overwrite, ignore write path already exists;
      DEFAULT (no -f) not overwrite and then return error;
      if used -f and write path is exist directory, will always return error.
  --patch
      swap to hsync_demo mode.
  -v  output Version info.
  -h or -?
      output Help info (this usage).

hsync_http command line usage:

download   : [options] -dl#hsyni_file_url hsyni_file
local  diff: [options] oldPath hsyni_file hsynz_file_url -diff#outDiffFile
local patch: [options] oldPath hsyni_file -patch#diffFile outNewPath
sync  infos: [options] oldPath hsyni_file [-diffi#cacheTempFile]
sync  patch: [options] oldPath [-dl#hsyni_file_url] hsyni_file hsynz_file_url [-diffi#cacheTempFile] 
  oldPath can be file or directory(folder),
  if oldPath is empty input parameter ""
options:
  -dl#hsyni_file_url
    download hsyni_file from hsyni_file_url befor sync patch;
  -diff#outDiffFile
    create diffFile from ranges of hsynz_file_url befor local patch;
  -diffi#cacheTempFile
    saving diffInfo to cache file for optimize speed when continue sync patch;
  -patch#diffFile
    local patch(oldPath+diffFile) to outNewPath;
  -cdl
    continue download data from breakpoint;
    DEFAULT continue download mode is closed;
  -r-stepRangeNumber
    DEFAULT -r-32, recommended 16,20,...
    limit the maximum number of .hsynz data ranges that can be downloaded
    in a single request step;
    if http(s) server not support muti-ranges request, must set -r-1
  -p-parallelThreadNumber
    DEFAULT -p-4;
    if parallelThreadNumber>1 then open multi-thread Parallel mode;
    NOTE: now download data always used single-thread.
  -n-maxOpenFileNumber
      limit Number of open files at same time when oldPath is directory;
      maxOpenFileNumber>=8, DEFAULT -n-24, the best limit value by different
        operating system.
  -g#ignorePath[#ignorePath#...]
      set iGnore path list in oldPath directory; ignore path list such as:
        #.DS_Store#desktop.ini#*thumbs*.db#.git*#.svn/#cache_*/00*11/*.tmp
      # means separator between names; (if char # in name, need write #: )
      * means can match any chars in name; (if char * in name, need write *: );
      / at the end of name means must match directory;
  -f  Force overwrite, ignore write path already exists;
      DEFAULT (no -f) not overwrite and then return error;
      not support oldPath outNewPath same path!
      if used -f and outNewPath is exist file:
        if patch output file, will overwrite;
        if patch output directory, will always return error;
      if used -f and outNewPath is exist directory:
        if patch output file, will always return error;
        if patch output directory, will overwrite, but not delete
          needless existing files in directory.
  -v  output Version info.
  -h or -?
      output Help info (this usage).

hsync_demo command line usage:

This cmdline is used for local sync tests, replacing the actual URL remote file with local file, see the hsync_http usage.

---

hsynz vs zsync:

case list(download from OneDrive):

	newFile <-- oldFile	newSize	oldSize
1	7-Zip_22.01.win.tar <-- 7-Zip_21.07.win.tar	5908992	5748224
2	Chrome_107.0.5304.122-x64-Stable.win.tar <-- 106.0.5249.119	278658560	273026560
3	cpu-z_2.03-en.win.tar <-- cpu-z_2.02-en.win.tar	8718336	8643072
4	curl_7.86.0.src.tar <-- curl_7.85.0.src.tar	26275840	26030080
5	douyin_1.5.1.mac.tar <-- douyin_1.4.2.mac.tar	407940608	407642624
6	Emacs_28.2-universal.mac.tar <-- Emacs_27.2-3-universal.mac.tar	196380160	257496064
7	FFmpeg-n_5.1.2.src.tar <-- FFmpeg-n_4.4.3.src.tar	80527360	76154880
8	gcc_12.2.0.src.tar <-- gcc_11.3.0.src.tar	865884160	824309760
9	git_2.33.0-intel-universal-mavericks.mac.tar <-- 2.31.0	73302528	70990848
10	go_1.19.3.linux-amd64.tar <-- go_1.19.2.linux-amd64.tar	468835840	468796416
11	jdk_x64_mac_openj9_16.0.1_9_openj9-0.26.0.tar <-- 9_15.0.2_7-0.24.0	363765760	327188480
12	jre_1.8.0_351-linux-x64.tar <-- jre_1.8.0_311-linux-x64.tar	267796480	257996800
13	linux_5.19.9.src.tar <-- linux_5.15.80.src.tar	1269637120	1138933760
14	Minecraft_175.win.tar <-- Minecraft_172.win.tar	166643200	180084736
15	OpenOffice_4.1.13.mac.tar <-- OpenOffice_4.1.10.mac.tar	408364032	408336896
16	postgresql_15.1.src.tar <-- postgresql_14.6.src.tar	151787520	147660800
17	QQ_9.6.9.win.tar <-- QQ_9.6.8.win.tar	465045504	464837120
18	tensorflow_2.10.1.src.tar <-- tensorflow_2.8.4.src.tar	275548160	259246080
19	VSCode-win32-x64_1.73.1.tar <-- VSCode-win32-x64_1.69.2.tar	364025856	340256768
20	WeChat_3.8.0.41.win.tar <-- WeChat_3.8.0.33.win.tar	505876992	505018368

test PC: Windows11, CPU R9-7945HX, SSD PCIe4.0x4 4T, DDR5 5200MHz 32Gx2
Program version: hsynz 0.9.3, zsync 0.6.2 (more programs's testing see HDiffPatch)
test Program:
zsync run make with zsyncmake -b 2048 -o {out_newi} {new},
client sync diff&patch by zsync -i {old} -o {out_new} {newi} (all files are local)
zsync -z run make with zsyncmake -b 2048 -z -u {new.gz} -o {out_newi} {new}
hsynz run make with hsync_make -s-2k {new} {out_newi} [{-c-?} {out_newz}],
client sync diff&patch by hsync_demo {old} {newi} {newz} {out_new} (all files are local)
hsynz p1 run make without compressor & out_newz , add -p-1
hsynz p8 run make without compressor & out_newz , add -p-8
hsynz p1 -zlib run make with -p-1 -c-zlib-9 (run hsync_demo with -p-1)
hsynz p8 -zlib run make with -p-8 -c-zlib-9 (run hsync_demo with -p-8)
hsynz p1 -gzip run make with -p-1 -c-gzip-9 (run hsync_demo with -p-1)
hsynz p8 -gzip run make with -p-8 -c-gzip-9 (run hsync_demo with -p-8)
hsynz p1 -zstd run make with -p-1 -c-zstd-21-24 (run hsync_demo with -p-1)
hsynz p8 -zstd run make with -p-8 -c-zstd-21-24 (run hsync_demo with -p-8)

test result average:

Program	compress	make mem	speed	sync mem	max mem	speed
zsync	52.94%	1M	353.9MB/s	7M	23M	34MB/s
zsync -z	20.67%	1M	14.8MB/s	12M	37M	28MB/s
hsynz p1	51.05%	5M	2219.3MB/s	5M	19M	191MB/s
hsynz p8	51.05%	18M	4646.8MB/s	12M	27M	331MB/s
hsynz p1 -zlib	20.05%	6M	17.7MB/s	6M	21M	160MB/s
hsynz p8 -zlib	20.05%	30M	119.5MB/s	13M	29M	246MB/s
hsynz p1 -gzip	20.12%	6M	17.6MB/s	6M	21M	160MB/s
hsynz p8 -gzip	20.12%	30M	118.8MB/s	13M	29M	247MB/s
hsynz p1 -zstd	14.97%	531M	1.9MB/s	24M	35M	173MB/s
hsynz p8 -zstd	14.96%	3349M	10.2MB/s	24M	35M	278MB/s

input Apk Files for test:

case list:

	app	newFile <-- oldFile	newSize	oldSize
1		cn.wps.moffice_eng_13.30.0.apk <-- 13.29.0	95904918	94914262
2		com.achievo.vipshop_7.80.2.apk <-- 7.79.9	127395632	120237937
3		com.adobe.reader_22.9.0.24118.apk <-- 22.8.1.23587	27351437	27087718
4		com.alibaba.android.rimet_6.5.50.apk <-- 6.5.45	195314449	193489159
5		com.amazon.mShop.android.shopping_24.18.2.apk <-- 24.18.0	76328858	76287423
6		com.baidu.BaiduMap_16.5.0.apk <-- 16.4.5	131382821	132308374
7		com.dragon.read_5.5.3.33.apk <-- 5.5.1.32	45112658	43518658
8		com.ebay.mobile_6.80.0.1.apk <-- 6.79.0.1	61202587	61123285
9		com.eg.android.AlipayGphone_10.3.0.apk <-- 10.2.96	122073135	119046208
10		com.google.android.apps.translate_6.46.0.apk <-- 6.45.0	48892967	48843378
11		com.google.android.googlequicksearchbox_13.38.11.apk <-- 13.37.10	190539272	189493966
12		com.jingdong.app.mall_11.3.2.apk <-- 11.3.0	101098430	100750191
13		com.netease.cloudmusic_8.8.45.apk <-- 8.8.40	181914846	181909451
14		com.reddit.frontpage_2022.36.0.apk <-- 2022.34.0	50205119	47854461
15		com.sankuai.meituan.takeoutnew_7.94.3.apk <-- 7.92.2	74965893	74833926
16		com.sankuai.meituan_12.4.207.apk <-- 12.4.205	93613732	93605911
17		com.sina.weibo_12.10.0.apk <-- 12.9.5	156881776	156617913
18		com.smile.gifmaker_10.8.40.27845.apk <-- 10.8.30.27728	102403847	101520138
19		com.ss.android.article.news_9.0.7.apk <-- 9.0.6	54444003	53947221
20		com.ss.android.ugc.aweme_22.6.0.apk <-- 22.5.0	171683897	171353597
21		com.taobao.taobao_10.18.10.apk <-- 10.17.0	117218670	117111874
22		com.tencent.mm_8.0.28.apk <-- 8.0.27	266691829	276603782
23		com.tencent.mobileqq_8.9.15.apk <-- 8.9.13	311322716	310529631
24		com.tencent.mtt_13.2.0.0103.apk <-- 13.2.0.0045	97342747	97296757
25		com.tripadvisor.tripadvisor_49.5.apk <-- 49.3	28744498	28695346
26		com.twitter.android_9.61.0.apk <-- 9.58.2	36141840	35575484
27		com.ubercab_4.442.10002.apk <-- 4.439.10002	69923232	64284150
28		com.ximalaya.ting.android_9.0.66.3.apk <-- 9.0.62.3	115804845	113564876
29		com.xunmeng.pinduoduo_6.30.0.apk <-- 6.29.1	30896833	30951567
30		com.youdao.dict_9.2.29.apk <-- 9.2.28	110624682	110628778
31		org.mozilla.firefox_105.2.0.apk <-- 105.1.0	83078464	83086656
32		tv.danmaku.bili_7.1.0.apk <-- 7.0.0	104774723	104727005

changed test Program:
zsync ... make -b 2048 changed to -b 1024
hsynz ... make -s-2k changed to -s-1k

test result average:

Program	compress	make mem	speed	sync mem	max mem	speed
zsync	62.80%	1M	329.8MB/s	6M	12M	76MB/s
zsync -z	59.56%	1M	19.8MB/s	8M	19M	56MB/s
hsynz p1	62.43%	4M	1647.6MB/s	4M	9M	152MB/s
hsynz p8	62.43%	6M	2563.7MB/s	11M	18M	270MB/s
hsynz p1 -zlib	58.67%	5M	23.7MB/s	4M	11M	151MB/s
hsynz p8 -zlib	58.67%	29M	141.8MB/s	12M	19M	265MB/s
hsynz p1 -gzip	58.95%	5M	23.5MB/s	4M	11M	148MB/s
hsynz p8 -gzip	58.95%	29M	141.8MB/s	12M	19M	256MB/s
hsynz p1 -zstd	57.74%	534M	2.7MB/s	24M	28M	151MB/s
hsynz p8 -zstd	57.74%	3434M	13.2MB/s	24M	28M	265MB/s

---

Contact

housisong@hotmail.com