v3.0.0

Hello everyone!

This is a much-anticipated release. In the future, we will strive to avoid accumulating such a large number of changes. We will code frequently, ship frequently, and release frequently!

Noteworthy Changes

• Every database engine now benefits from failover storage. This technique can save the day if one of the storage systems goes down. However, do not forget to monitor them anyway.
• For Postgres, we have a WAL archiving daemon available. Its primary purpose is to improve the efficiency of parallel archiving. Additionally, WAL-G supports the partial restoration of a cluster. If you have large, unnecessary tables, you can exclude them from the restoration process, the Postgres will skip over the WAL records for these tables.
• Greenplum's PITR (Point-in-Time Recovery) is undergoing its maturation process. Partial restore is supported in a similar way to Postgres. We now have tools to verify the integrity of AO\AOCS files in storage without actually restoring backups. This technique is necessary to ensure developers can sleep well, as Greenplum backups are relatively new, and extra precautions need to be taken to ensure they are safe and reliable.
• MongoDB backups can now benefit from binary backups, which significantly reduce the impact of backups on a running workload.
• MySQL now supports incremental backups, also known as delta backups. However, this feature is still in beta mode and the API may change in the future.
• Also, we now have beta support for etcd. Changes necessary for real PITR have not yet been merged into etcd, and can be found at etcd-io/etcd#17233.

Detailed changes log

• Correct the --without-files-metadata docs by @usernamedt in #1327
• expose zstd compression by @heinerstilz in #1326
• [WIP] Multithread downloading directory by @hariton27sy in #1297
• Add WALG_GP_AOSEG_SIZE_THRESHOLD setting by @usernamedt in #1329
• Respect WALG_NETWORK_RATE_LIMIT globally by @usernamedt in #1330
• Restrict rate limiting only to backup-push/backup-fetch commands by @usernamedt in #1331
• Initial mvp mongodb binary backup by @KhurtinDN in #1285
• Add delta backups for Greenplum by @usernamedt in #1333
• Fix sorting in detailed mongodb backup list by @KhurtinDN in #1334
• Add @VGoshev and @KhurtinDN as codeowners of MongoDB part of Wal-G by @VGoshev in #1335
• Add tests, docs and minor fixes for Greenplum delta backups by @usernamedt in #1337
• Fix mongodb binary backup during mongod do checkpoint by @KhurtinDN in #1342
• Add rate limiter for mongodb by @KhurtinDN in #1343
• Add data catalog size for PostgreSQL/Greenplum backups by @usernamedt in #1344
• Add minor improvements for Greenplum by @usernamedt in #1345
• postgresq.md add cmake installation by @rogaliiik in #1348
• Support ARM64 builds on MacOS by @usernamedt in #1349
• Implement delete target backup for Greenplum by @usernamedt in #1350
• Add delete garbage command for Greenplum by @usernamedt in #1351
• Add limiters to Greenplum segment backup-push/backup-fetch by @usernamedt in #1352
• Fix failing docs builds by @usernamedt in #1354
• Fix distro name for ubuntu-20.04 in arm release by @williancolognesitrimble in #1355
• Bump golangci/golangci-lint-action from 3.2.0 to 3.3.0 by @dependabot in #1361
• Add minimal mongod config path option by @KhurtinDN in #1360
• Fix closing mongodb backup cursor by @KhurtinDN in #1359
• Replace mongodb binary backup to tar archive format by @KhurtinDN in #1357
• Fix binary backup of mongodb 4.2 by @KhurtinDN in #1364
• Add daemon mode for wal-push command by @rogaliiik in #1353
• Change error to warning when creating symlink, that already exists by @IncubusRK in #1320
• [Greenplum] do not fail backup on increment reading error by @usernamedt in #1365
• Only dump relevant env variables with LOG_LEVEL=DEVEL by @LeGEC in #1367
• Do not print secrets in debug logs by @LeGEC in #1368
• Bump golangci/golangci-lint-action from 3.3.0 to 3.3.1 by @dependabot in #1380
• enable using aws iam roles by @Qwiz in #1377
• notify_d watchdog for daemon service by @rogaliiik in #1371
• Use pure go implementation for zstd by @LeGEC in #1376
• Restore debug logging improvements by @usernamedt in #1389
• postgresql: fix data path mismatch on local backups by @LeGEC in #1383
• Fix Greenplum deleted AO segment file error handling by @usernamedt in #1391
• Implement max file size for mysql by @bularond in #1374
• Fix non-string config settings handling by @usernamedt in #1393
• Makefile: change compilation and linking of libbrotli to optional (and off by default) by @LeGEC in #1379
• add unit tests for the WriteBackupListDetails function by @tewboat in #1395
• Add recovery_target_timeline setting to Greenplum restore config by @usernamedt in #1398
• Add detailed json backup-list for Greenplum by @usernamedt in #1399
• Add restore-point-list command for gp by @rogaliiik in #1397
• Update golangci-lint version to 1.50.1 by @serprex in #1381
• Add restore-point-ts flag for Greenplum backup-fetch by @usernamedt in #1403
• Greenplum: populate pg_hba.conf on master host during restore by @usernamedt in #1404
• enable linters for cmd/redis internal/databases/redis and pkg/storages by @teewar2 in #1407
• Fix wal-g-mongo mongod restore procedure by @KhurtinDN in #1414
• Add mongo-rs-members parameter to binary mongo restore by @KhurtinDN in #1418
• [MongoDB] add opportunity restore mongodb backup with + 1 version by @Apelsin234 in #1417
• Add mongo-rs-name and mongo-rs-members parameters to binary mongo restore by @KhurtinDN in #1419
• add unit test for greenplum/backup_fetch_handler by @teewar2 in #1406
• fix bug with mixed up arguments by @Apelsin234 in #1421
• Postgres requires data dir to be 0700 or 0750. by @Aevin1387 in #1420
• Add support mongodb 6.0 by @KhurtinDN in #1416
• Fix connect to mongodb during restore procedure by @KhurtinDN in #1428
• Bump golangci/golangci-lint-action from 3.3.1 to 3.4.0 by @dependabot in #1426
• MDB-21194: [MongoDB] fix bug with purge garbage and change logic for retains by @Apelsin234 in #1429
• Implementation of the new binlog-server command by @Fizic in #1408
• Bump github.com/prometheus/client_golang from 1.7.1 to 1.11.1 in /internal/tools by @dependabot in #1433
• Bump golang.org/x/text from 0.3.6 to 0.3.8 in /internal/tools by @dependabot in #1437
• Implement retry download backup by @bularond in #1394
• adding push metrics as a defer in the upload method to make sure metrics are pushed even on failure by @francoispqt in #1423
• Revert #1423 by @usernamedt in #1442
• Implement database composer and partial fetch for Postgres by @Catmoonlight in #1434
• [MongoDB] fix retryable error from code to name by @Apelsin234 in #1444
• Bump actions/setup-go from 3 to 4 by @dependabot in #1446
• Adaptive timeout for mongodb restore by @Apelsin234 in #1449
• [Greenplum] Store metadata about backup restore points by @usernamedt in #1448
• Add Postgres partial restore by database names by @Catmoonlight in #1447
• Add tests postgres sentinel dto is incremental by @AlexUran in #1455
• Add tests for NewDtoSerializer by @denisSurkov in #1460
• feat: removing binlogs after using them by @Fizic in #1457
• Update golangci-lint in .github/workflows to 1.52.2 by @serprex in ...

2.0.2-rc1

This is a pre-release to test the aws iam roles config setting.

v2.0.1

Release notes:
This release mainly consists of various bug fixes and improvements.

Common:

• Add arm(aarch64) build pipeline for ubuntu20.04 binaries (#1304)

PostgreSQL:

• Add warning output on backup-fetch retries (#1325)
• Remove redundant check for --without-files-metadata flag (#1321)
• PostgreSQL 15 Support (#1319)
• Fix of deleting WAL-E backups (#1292)

MySQL/MariaDB:

• Fix a regression in MariaDB backup-push handler (#1313)
• Binlog cache fixes (#1281)

Greenplum:

• Optimize delete handler (#1300, #1302)
• Optimize storage listing queries during the backup push (#1312)
• Use AO files metadata instead of backup references (#1308)
• Refactor AO/AOCS storage metadata queries (#1280, #1278)

Storages:

• Azure authentication using the default credential chain (#1282)
• Update aws-sdk-go to v1.44.7 (#1262)
• Add check read/write to storage tools (#1275)

v2.0.0

Release notes:

This release contains a huge amount of new features, bug fixes, and performance improvements. It is worth noting that v2.0 Postgres delta backups are not backward-compatible with the pre-2.0 versions. It means that the v2.0 delta backup can't be restored with the v1.1 or earlier versions. Also, this release introduces the initial support for Greenplum physical backups.

WAL-G for MySQL now supports the parallel backup-push/backup-fetch and has an optimized binlog upload mechanism. WAL-G for SQLServer now supports the backup encryption and compression as well as the external storage backup import/export.

Common:

• Update lz4, brotli #1165, #1179
• Multiple libsodium fixes #1087, #1227
• Bash/zsh shell completions #1250, #1252
• Add a 'flags' subcommand #1033
• Disable CPU Limits in turbo mode #1121
• Metrics forwarding via statsd protocol #1240

SQLServer:

• Add SQLServerDBConcurrency setting to limit concurrent database operations #1067
• Add LATEST keyword handling #1110
• Add norecovery option for backup log command #1115
• Add SQLServer compression / encryption #1161
• Backup import and export to the external storage #1184

Postgres:

• Add CopyComposer for full backups optimization using remote copy API #1012
• Add streamed JSON sentinel serialization to decrease memory usage #1103
• Extract heavy files and tarfilesets from the backup sentinel #1114
• Introduce WALG_TAR_DISABLE_FSYNC to skip fsync during backup-fetch #1144
• Add reduced memory usage backup mode #1101
• Improvements and fixes for remote backup #1175
• Add pgBackRest backups restore support (beta) #1139, #1202
• Add 'wal-restore' command #977
• Add 'delete garbage' command #1217, #1218
• Return custom error code when WAL file does not exist #1195
• Add WALG_ALIVE_CHECK_INTERVAL to fail backup on Postgres process failure #1260
• Reformat LSN output #1259

MySQL:

• Add wal-g binlog-analyze #1085
• Check binlog GTIDs before pushing binlogs to S3 #1190
• Add --check-gtids to wal-g binlog-push #1085
• Add --until-binlog-last-modified-time option to wal-g-mysql binlog-replay #1154
• Improve MySQL restore performance #1131
• Allow backups on cascade replicas #1168
• Fix mysql backup of a lagging replica #1223
• Make mysql bilong-push status observable to 3rd party tools #1229

Greenplum:

• Initial physical backups support for GPDB 6.19.2+. API might change in the future versions.

Storages:

• Add get/put/rm/cat to storage tools #1069, #1164
• S3: Add download by range #1083
• S3: Add connection reset retries #1137
• S3: Add WALG_S3_MAX_RETRIES setting #1148
• S3: Fix S3_LOG_LEVEL setting #1158
• S3: AWS authentication fixes #1209
• S3: Yandex Cloud session token auth support #1213
• SSH: Ensure trailing delimiter in the storage prefix #1130
• Azure: Switch to the recommended azblob SDK #1135, #1171
• Azure: Fix ListFolder never returning error #1208
• Swift: Ignore files that were not found during ListFolder #1129

v2.0.0-rc1

This is a pre-release to test S3 authentication with mechanisms other than key/secret.

v1.1.3-rc

This pre-release introduces the Greenplum physical backups support.

v1.1.2-rc

This pre-release introduces the S3_LOG_LEVEL setting to help with debugging the S3 storage interaction issues.

v1.1.1-rc

This is a pre-release intended to test the s3 connection reset retries fix (#1137).

v1.1

Some interesting features/fixes:

• Include libsodium in the GitHub release builds. Exit with failure if libsodium is requested but WAL-G was not compiled with libsodium support #1062 CVE-2021-38599
• UserData is now required to be a valid JSON #1049
• backup-list now sorts backups by the astronomical time of the completition #907

Postgres:

• .history files overwrite prevention fix #1052
• wal-verify can now be run from a standby #1014
• wal-verify now ignores the permanent backups #1014, #1002

MongoDB:

• Add support for MongoDB 5.0 #1041

SQLServer:

• Fix MSSQL driver import #1038

MySQL:

• Add --turbo flag to disable limiters #1027
• Fix --detailed backup-list bug #1026
• Add --until flag to binlog-push #1005

Redis:

• Add 6.2 support & redis-cli bug workaround #1022

Storages

• S3: customer SSE key support #1042 (wal-g/storages#48)
• Azure: support AZURE_STORAGE_SAS_TOKEN without the leading '?' #1056
• Azure: disable the unnecessary syslog writes #1058

v1.0

Release notes:

This time we decided to bump the number to 1.0 to switch to a common versioning pattern.
WAL-G 1.0 is compatible with prior versions (v0.2.0+). This might change with later releases.
Also, WAL-G 1.0 can restore WAL-E backups and WALs.
Good news for non-PostgreSQL databases: WAL-G is now production-ready for MS SQL, MySQL databases, meanwhile the API can be changed in future releases. MongoDB and Redis support is in the beta stage.

Some interesting features/fixes:

• #780 PostgreSQL remote backups
• #905 Improve compatibility with PostgreSQL 13, introduced WALG_PREFETCH_DIR option
• #824 New option was added to wal-verify, now it requires an argument timeline|integrity
• #903 It's now not allowed to create non-permanent delta backups from permanent backups
• #842 Fix compatibility with S3 implementations that do not support ListObjectsV2
• #950 It's now again possible to tell WAL-G to rename .ready files to .done directly
• #898, #962 WAL prefetch is now slightly faster
• #858 Catchup-list command to list delta backups taken specifically for catchup
• #847 Support for Yandex Cloud KMS
• #817, #833 Multipart retries for GCS storage
• #873 Support for AzureUSGovernmentCloud, AzureChinaCloud, AzureGermanCloud
• #899 Add metadata for WAL files
• #913, #917 New option to delete single backup for PostgreSQL
• Add Redis backups. Now we have Redis support with backup-fetch, backup-push, backup-list and delete commands. For full info see Redis.md
• Add MySQL catchup mode: allow replay binlogs while uploading new ones
• Add SQLServer single database restore with renaming
• Huge amount of bug fixes, tests and improvements for Redis/MongoDB/SQLServer/etc

Thanks!