librsync 2.3.4

Overview

This is a patch release that includes an important fix for a bug introduced in v2.3.3 that broke librsync on some platforms. Upgrading from v2.3.3 is strongly recommended for most people, particularly on platforms where the unit tests fail.

Included Changes

• Fix #248 by putting #include "config.h" with /* IWYU pragma: keep */ in most src/*.c files. Add /* IWYU pragma: keep */ to includes in src/fileutil.c that are needed on some platforms but not others so we can remove the special exemptions to skip this file for the iwyu and iwyu-fix targets in CMakeLists.txt. Also add some typecasts to rollsum.[ch] and patch.c to silence warnings on Windows. (dbaarda, #249)

Known Issues

• The debian package configs are still very out of date.

• There is still no documentation on how to compile on or cross-compile for windows. See #171 for help.

librsync 2.3.3

Overview

This is a patch release that includes some minor performance improvements, but mostly includes documentation and development process improvements, particularly for windows. Upgrading from v2.3.2 is recommended for most people, particularly windows users.

Included Changes

• Fix #244 Add windows build to stable release. Updated CONTRIBUTING.md release instructions to be clearer and include instructions on how to upload the win64 install artifact from the github "Check" action. (dbaarda, #245)

• Update github actions and fix iwyu build target. Update checkout and upload-artifact to v3. Update lint.yml installed packages for fixed iwyu deps. Fix iwyu build target to ignore fileutil.c and use neater clang output with noisy "note:" output removed. Run make iwyu-fix to fix includes for tests/rabinkarp_perf.c. (dbaarda #243)

• Add missing word to README.md. (AvdN, #237)

• Make delta directly process the input stream if it has enough data. Delta operations will only accumulate data into the internal scoop buffer if the input buffer is too small, otherwise it will process the input directly. This makes delta calculations 5%~15% faster by avoiding extra data copying. (dbaarda, #234)

• Add .gitignore for .cmake created by LSP on Windows. (sourcefrog, #232)

• Upload build and install artifacts from Github actions. This means we get downloadable build and install artifacts for all platforms from the "Check" action. (sourcefrog, #231)

• Improve documentation so that Doxygen generates more complete documentation with diagrams, renders better, and is more navigable as markdown docs on GitHub. (dbaarda, #230)

• Add github action and make targets for clang-tidy and iwyu. Added clang-tidy and iwyu make targets for checking code and includes, and iwyu-fix for fixing includes. Added lint.yml GitHub action to run these checks. Fixed all clang-tidy and iwyu warnings except for fileutil.c with platform related include complications. Added consistent include guards to all headers. Updated and improved documentation in CONTRIBUTING.md to include these changes. (rizsotto, dbaarda, #229)

• Tidy rdiff integration test scripts. Made the filenames and shell arguments for test scripts consistent. (dbaarda, #227)

• Add better cmake build type configuration support. Added BuildType.cmake with better support for selecting the build type and making it default to Debug. (dbaarda, #226)

• Fix #215 Migrate from Travis to GitHub Actions. Added a check.yml GitHub action with updated test/platform matrix including full testing of rdiff on Windows. (rizsotto, dbaarda, #225)

• Fix bash test scripts to work on Windows. Tweaked cmake configuration and bash script tests so that full rdiff tests using libpopt from vcpkg work. Running cmake --target check with rdiff compiled now works on windows. (dbaarda, #224)

• Remove obsolete unused tests. Removed some obsolete mdfour test data files and check-rdiff perl script. (dbaarda, #223)

• Fix warning for later CMake versions. New CMake versions started complaining about the filename Findlibb2.cmake not matching the LIBB2 variables being used. (rizsotto, #221)

Known Issues

• The debian package configs are still very out of date.

• There is still no documentation on how to compile on or cross-compile for windows. See #171 for help.

librsync 2.3.2

Overview

This is a patch release that fixes some minor bugs, tidies the code for many compiler warnings, and improves windows compatibility. Upgrading from v2.3.1 is recommended for most people, and essential for people using platforms experiencing bugs #214 or #207.

Included Changes

• Fix #214 heap corruption for too small kbloom. This could have crashed delta operations for very small files/signatures. Strangely it didn't seem to cause problems for most compilers/platforms, but did trigger errors for new versions of MSVC. (ljusten, #213)

• Fix #207 and add Travis Windows checks and improve compatibility. Turn on -Wconversion -Wno-sign-conversion warnings for clang. Add MSVC compiler flags to turn off posix warnings. Make all code compile clean with no warnings on all Travis platforms. Added cmake config checking for windows io.h and improve fileutil.c for MSVC. Fix broken error handling in rs_file_copy_cb(). Improved trace output, making it less spamy and more consistent. Add patch checking for invalid literal lengths. Improve internal variable and argument types. Add explicit type conversions. (dbaarda, #208)

• Fix a bug so patch will now fail returning RS_CORRUPT on encountering a zero length copy command instead of hanging. Make copy_cb() copying more data than requested an assert-fail on debug builds, and a log-warning for release builds. Make trace output a little less spammy about copy_cb() return values. (dbaarda, #206)

Known Issues

• The debian package configs are still very out of date.

• There is still no documentation on how to compile on or cross-compile for windows. See #171 for help.

librsync 2.3.1

Overview

This is a patch release that adds peformance improvements, updates and tidies documentation, and fixes two build/test bugs on FreeBSD. Upgrading from v2.3.0 is optional for most people, and essential for people using platforms experiencing bugs #198 or #199 .

For an analysis and comparison of performance improvements in librsync with each version see;

https://github.com/dbaarda/librsync-tests/blob/master/RESULTS.rst

Included Changes

• Fix #198 cmake popt detection using pkg-config and #199 test scripts on FreeBSD. Fixes and tidies FindPOPT.cmake and Findlibb2.cmake to use pkg-config correctly and behave more like official FindPackage() cmake modules. Makes all test scripts use /bin/sh instead of /bin/bash. (dbaarda, mandree #200)

• Change default block_len to always be a multiple of the blake2b 128 byte blocksize for efficiency. Tidy and update docs to explain using rs_sig_args() and rs_build_hash_table(), add rs_file_*() utils, and document new magic types. Remove really obsolete entries in TODO.md. Update to Doxygen 1.8.16. (dbaarda, #195)

• Improve hashtable performance by adding a small optional bloom filter, reducing max loadfactor from 80% to 70%, Fix hashcmp_count stats to include comparing against empty buckets. This speeds up deltas by 20%~50%. (dbaarda, #192, #193, #196)

• Optimize rabinkarp_update() by correctly using unsigned constants and manually unrolling the loop for best performance. (dbaarda, #191)

Known Issues

• The debian package configs are still very out of date.

• There is still no documentation on how to compile on or cross-compile for windows. See #171 for help.

librsync 2.3.0

Overview

This is a minor release that adds rs_sig_args() for getting the recommended signature arguments based on the file size. which is used by rs_sig_file() and rdiff. This means rdiff by default will use a better block size based on the file size, and supports using --sum-size=-1 to use the smallest safe sum size (safe against random collisions, not against crafted hash-collision attacks). It removes old code that made the default strongsum size 8 when using --hash=md4, and it now defaults to 16 (the max for md4) for better protection against the risk of hash collision attacks and corruption. It also includes many code cleanups and testing improvements, including full C99 compliance. Upgrading from v2.2.1 is optional.

To generate signatures identical to the previous version's default arguments, you will need to use --block-size=2048 to select the old default block size. Also, when using --hash md4 you will need to add --sum-size=8 to select the old sum-size default when using md4 strongsums.

Included Changes

• Bump minor version from 2.2.1 to 2.3.0 to reflect additional rs_sig_args() and strong_len=-1 support.

• Add public rs_sig_args() function for getting the recommend signature args from the file size. Added support to rdiff for --sum-size=-1 to indicate "use minimum size safe against random block collisions". Added warning output for sum-sizes that are too small to be safe. Fixed possible rdiff bug affecting popt parsing on non-little-endian platforms. (dbaarda, #109)

• Fixed yet more compiler warnings for various platforms/compilers. (Adsun701, texierp, #187, #188)

• Improved cmake popt handling to find popt dependencies using PkgConfig. (ffontaine, #186)

• Tidied internal code and improved tests for netint.[ch], tube.c, and hashtable.h. (dbaarda, #183 #185).

• Improved C99 compatibility. Add -std=c99 -pedantic to CMAKE_C_FLAGS for gcc and clang. Fix all C99 warnings by making all code C99 compliant. Tidy all CMake checks, #cmakedefines, and #includes. Fix 64bit support for mdfour checksums (texierp, dbaarda, #181, #182)

• Usage clarified in rdiff (1) man page. (AaronM04, #180)

Known Issues

• The debian package configs are still very out of date.

• There is still no documentation on how to compile on or cross-compile for windows. See #171 for help.

librsync 2.2.1

Overview

This is a patch release that fixes a bug that makes delta operations hang for files larger than 4GB. Upgrading from v2.2.0 is essential.

Included Changes

• Fix #176 hangs calculating deltas for files larger than 4GB. (dbaarda, #177)

Known Issues

• The debian package configs are still very out of date.

• There is still no documentation on how to compile on or cross-compile for windows. See #171 for help.

librsync 2.2.0

Overview

This is a minor release that adds RabinKarp rollsum support as the new recommended default. This is a much better rolling hash with a much better distribution and less collisions, particularly for text files and small blocks. This results in faster delta calculations and a reduced risk of corruption from hash collisions when using small strongsum sizes. It also includes important fixes for compiling on/for windows, and some other minor improvements. Upgrading from v2.1.0 is essential for people compiling for windows, recommended for people who want a performance increase, and otherwise not urgent.

To generate signatures compatible with previous versions of rdiff you will need to use the --rollsum rollsum to select the old rollsum algorithm, or for librsync use the RS_BLAKE2_SIG_MAGIC or RS_MD4_SIG_MAGIC arguments when starting the signature with rs_sig_begin().

Included Changes

• Bump minor version from 2.1.0 to 2.2.0 to reflect additional RabinKarp rollsum support.

• Fix MSVC builds by adding missing LIBRSYNC_EXPORT to variables in librsync.h, add -DLIBRSYNC_STATIC_DEFINE to the sumset_test target, and correctly install .dll files in the bin directory. (adsun701, #161)

• Add RabinKarp rollsum support and make it the default. RabinKarp is a much better rolling hash, which reduces the risk of hash collision corruption and speeds up delta calculations. The rdiff cmd gets a new -R (rollsum|rabinkarp) argument with the default being rabinkarp, Use -R rollsum to generate backwards-compatible signatures. (dbaarda, #3)

• Use single-byte literal commands for small inserts in deltas. This makes each small insert use 1 less byte in deltas. (dbaarda, #120)

• Fix multiple warnings (cross-)compiling for windows. (Adsun701, #165, #166)

• Change rs_file_size() to report -1 instead of 0 for unknown file sizes (not a regular file). (dbaarda #168)

• Add cmake BUILD_SHARED_LIBS option for static library support. BUILD_SHARED_LIBS defaults to ON, and can be set to OFF using ccmake . to build librsync as a static library. (dbaarda #169)

• Fix compile errors and add .gitignore entries for MSVS 2019. Fixes hashtable.h to be C99 compliant. (ardovm #170)

Known Issues

• The debian package configs are still very out of date.

• There is no documentation on how to compile on or cross-compile for windows. See issues about this on github for help.

librsync 2.1.0

Overview

This is a minor release that exposes a few useful utility functions to librsync.h previously only used by rdiff. It also introduces proper exporting of only public symbols in the librsync.h header. The most important fix is for a memory leak when freeing signatures. Upgrading from v2.0.2 is highly recommended for people using librsync and experiencing memory leak problems, but otherwise not urgent.

Included Changes

• Bump minor version from 2.0.3 to 2.1.0 to reflect additions to librsync.h.

• Fix exporting of private symbols from librsync library. Add export of useful large file functions rs_file_open(), rs_file_close(), and rs_file_size() to librsync.h. Add export of rs_signature_log_stats() to log signature hashtable hit/miss stats. Improve rdiff error output. (dbaarda, #130)

• Updated release process to include stable tarballs. (dbaarda,#146)

• Remove redundant and broken --paranoia argument from rdiff. (dbaarda, #155)

• Fix memory leak of rs_signature_t->block_sigs when freeing signatures. (telles-simbiose, #147)

• Document delta file format. (zmj, #46)

• Fix up doxygen comments. (dbaarda, #151)

Known Issues

• The debian package configs are still very out of date.

librsync 2.0.2

Overview

This is a minor point release that includes mainly code cleanups and compatibility fixes for different platforms. Upgrading from v2.0.1 is recommended but not urgent for people who don't require any of these fixes.

Included Changes

• Improve CMake install paths configuration (wRAR, #133) and platform support checking when cross-compiling (fornwall, #136).

• Fix Unaligned memory access for rs_block_sig_init() (dbaarda, #135).

• Fix hashtable_test.c name collision for key_t in sys/types.h on some platforms (dbaarda, #134)

• Format code with consistent style, adding make tidy and make tidyc targets for reformating code and comments. (dbaarda, #125)

• Removed perl as a build dependency. Note it is still required for some tests. (dbaarda, #75)

• Update RPM spec file for v2.0.2 and fix cmake man page install. (deajan, #47)

Known Issues

• The debian package configs are still very out of date.

librsync 2.0.1

Overview

Although this is only a minor point release it is long overdue and includes a large number of performance improvements and bug fixes. It should be significantly faster than any earlier version, particularly for large files. It should also be more robust and reliable. People are strongly encouraged to upgrade.

Included Changes

• Extensively reworked Doxygen documentation, now available at
  http://librsync.sourcefrog.net/ (Martin Pool)

• Removed some declarations from librsync.h that were unimplemented or no
  longer ever useful: rs_work_options, rs_accum_value. Remove
  declaration of unimplemented rs_mdfour_file(). (Martin Pool)

• Remove shipped snprintf code: no longer acutally linked after changing to
  CMake, and since it's part of C99 it should be widely available.
  (Martin Pool)

• Document that Ninja (http://ninja-build.org/) is supported under CMake.
  It's a bit faster and nicer than Make. (Martin Pool)

• make check (or ninja check etc) will now build and run the tests.
  Previously due to a CMake limitation, make test would only run existing
  tests and could fail if they weren't built.
  (Martin Pool, #49)

• Added cmake options to exclude rdiff target and compression from build.
  See install documentation for details. Thanks to Michele Bertasi.

• popt is only needed when rdiff is being built. (gulikoza)

• Improved large file support for platforms using different variants
  of fseek (fseeko, fseeko64, _fseeki64), fstat (fstat64,
  _fstati64), and fileno (_fileno). (dbaarda, charlievieth,
  gulikoza, marius-nicolae)

• rdiff -s option now shows bytes read/written and speed. (gulikoza).
  For delta operations it also shows hashtable match statistics. (dbaarda)

• Running rdiff should not overwrite existing files (signatures, deltas and
  new patched files) by default. If the destination file exists, rdiff will
  now exit with an error. Add new option -f (--force) to overwrite existing
  files. (gulikoza)

• Improve signature memory allocation (doubling size instead of calling
  realloc for every sig block) and added support for preallocation. See
  streaming.md job->estimated_signature_count for usage when using the
  library. rdiff uses this by default if possible. (gulikoza, dbaarda)

• Significantly tidied signature handling code and testing, resulting in more
  consistent error handling behaviour, and making it easier to plug in
  alternative weak and strong sum implementations. Also fixed "slack delta"
  support for delta calculation with no signature. (dbaarda)

• stdint.h and inttypes.h from C99 is now required. Removed redundant
  librsync-config.h header file. (dbaarda)

• Lots of small fixes for windows platforms and building with MSVC.
  (lasalvavida, mbrt, dbaarda)

• New open addressing hashtable implementation that significantly speeds up
  delta operations, particularly for large files. Also fixed degenerate
  behaviour with large number of duplicate blocks like runs of zeros
  in sparse files. (dbaarda)

• Optional support with cmake option for using libb2 blake2 implementation.
  Also updated included reference blake2 implementation with bug fixes
  (dbaarda).

• Improved default values for input and output buffer sizes. The defaults are
  now --input-size=0 and --output-size=0, which will choose recommended
  default sizes based on the --block-size and the operation being performed.
  (dbaarda)

• Fixed hanging for truncated input files. It will now correctly report an
  error indicating an unexpected EOF was encountered. (dbaarda,
  #32)

• Fixed #13 so that faster slack delta's are used for signatures of
  empty files. (dbaarda,
  #13)

• Fixed #33 so rs_job_iter() doesn't need calling twice with eof=1.
  Also tidied and optimized it a bit. (dbaarda,
  #33)

• Fixed #55 remove excessive rs_fatal() calls, replacing checks for
  programming errors with assert statements. Now rs_fatal() will only
  be called for rare unrecoverable fatal errors like malloc failures or
  impossibly large inputs. (dbaarda,
  #55)

Known Issues

• The rpm and debian package configs are very out of date.