v3.0.1

This is a technical pre-release aimed at mitigating several CVEs in dependencies (#1675, #1715). These vulenrabilities have no known ways to be exploited against WAL-G. But these days we should keep CVE scanners calm.
Also this pre-release gives access to several features that didn't make it into 3.0.0 because were not tested enough (#1662, #1646, #1652).
As usual, handful of bugs fixed.

Thanks to everyone who contributed!

What's Changed

• Support DigitalOcean Spaces (NoSuchKey during ListFolder) by @turt2live in #1646
• Skip archive_command check on standby by @x4m in #1662
• fixes for ao check config by @debebantur in #1664
• Fix for GP incremental backups by @debebantur in #1666
• Add unit tests for internal.databases.greenplum.buildFetchCommand by @denisSurkov in #1661
• Catchup-send and catchup-receive commands by @x4m in #1652
• Collecting statistics about read/written bytes by @debebantur in #1670
• Add Docs for undocumented debug env variables by @sebastiangrill in #1640
• sanikadze:doc_gp_gp_pitr by @Sanikadze in #1672
• Implement lockFile for windows by @passcod in #1673
• upgrade golang.org/x/net to resolve CVE-2023-39325 by @devinturner in #1675
• Fix WALG_STATSD_EXTRA_TAGS by @nick-kentik in #1676
• Fix gp tests by @debebantur in #1677
• add path to recovery.conf for greenplum by @avkashin in #1679
• Clarified dev installing instruction on Ubuntu for go 1.16+ by @homper in #1681
• added IsTablespaceMapExists test by @NiripsaKakVsegda in #1680
• split mongodb restore on stages under flag by @Apelsin234 in #1685
• add support to redis acl user by @NotAmigo in #1682
• Add glob pattern support in storagetools by @ReshetnikovPavel in #1684
• add PGSSLCERT, PGSSLKEY, PGSSLROOTCERT support by @NiripsaKakVsegda in #1687
• WALG-1627. MySQL fix file double Close() on file by @ostinru in #1628
• add create upload input unit tests by @danilkaz in #1690
• add tests for SetUserData in postgres/generic_meta_interactor by @SixBlueBlack in #1697
• Fix lll lint violation, update golangci version by @serprex in #1686
• Exclude standby.signal by @munakoiso in #1702
• WALG-1665. User proper DSNs in MySQL documentation & validate DSN format it earlier by @ostinru in #1692
• unit test for internal/backup.go/FetchMetadata by @ValeriaIM in #1695
• Bump golangci/golangci-lint-action from 4.0.0 to 5.3.0 by @dependabot in #1703
• Save isIncremented attribute by @debebantur in #1704
• Add charset converter to convert non utf-8 char by @anisurrahman75 in #1705
• Bump golangci/golangci-lint-action from 5.3.0 to 6.0.1 by @dependabot in #1706
• WALG-1683. Save CPU arch / OS to mysql sentinel. by @ostinru in #1693
• added without metadata to gp by @debebantur in #1708
• Tests for isPagedFile by @Themplarer in #1710
• Added validation for S3 credentials by @Themplarer in #1711
• fixed hex parsing in timeline by @debebantur in #1712
• Added retention period for s3 by @debebantur in #1713
• Add few unit test on azure ConfigureStorage by @Loraan in #1714
• Add Uploader interface mock, test on UploadSentinel and make mock command by @Loraan in #1716
• Update modules in order to resolve CVEs. by @montaguethomas in #1715

New Contributors

• @turt2live made their first contribution in #1646
• @sebastiangrill made their first contribution in #1640
• @Sanikadze made their first contribution in #1672
• @passcod made their first contribution in #1673
• @devinturner made their first contribution in #1675
• @avkashin made their first contribution in #1679
• @homper made their first contribution in #1681
• @NiripsaKakVsegda made their first contribution in #1680
• @NotAmigo made their first contribution in #1682
• @danilkaz made their first contribution in #1690
• @SixBlueBlack made their first contribution in #1697
• @ValeriaIM made their first contribution in #1695
• @anisurrahman75 made their first contribution in #1705
• @Themplarer made their first contribution in #1710
• @Loraan made their first contribution in #1714
• @montaguethomas made their first contribution in #1715

Full Changelog: v3.0.0...v3.0.1

v3.0.0

Hello everyone!

This is a much-anticipated release. In the future, we will strive to avoid accumulating such a large number of changes. We will code frequently, ship frequently, and release frequently!

Noteworthy Changes

• Every database engine now benefits from failover storage. This technique can save the day if one of the storage systems goes down. However, do not forget to monitor them anyway.
• For Postgres, we have a WAL archiving daemon available. Its primary purpose is to improve the efficiency of parallel archiving. Additionally, WAL-G supports the partial restoration of a cluster. If you have large, unnecessary tables, you can exclude them from the restoration process, the Postgres will skip over the WAL records for these tables.
• Greenplum's PITR (Point-in-Time Recovery) is undergoing its maturation process. Partial restore is supported in a similar way to Postgres. We now have tools to verify the integrity of AO\AOCS files in storage without actually restoring backups. This technique is necessary to ensure developers can sleep well, as Greenplum backups are relatively new, and extra precautions need to be taken to ensure they are safe and reliable.
• MongoDB backups can now benefit from binary backups, which significantly reduce the impact of backups on a running workload.
• MySQL now supports incremental backups, also known as delta backups. However, this feature is still in beta mode and the API may change in the future.
• Also, we now have beta support for etcd. Changes necessary for real PITR have not yet been merged into etcd, and can be found at etcd-io/etcd#17233.

Detailed changes log

• Correct the --without-files-metadata docs by @usernamedt in #1327
• expose zstd compression by @heinerstilz in #1326
• [WIP] Multithread downloading directory by @hariton27sy in #1297
• Add WALG_GP_AOSEG_SIZE_THRESHOLD setting by @usernamedt in #1329
• Respect WALG_NETWORK_RATE_LIMIT globally by @usernamedt in #1330
• Restrict rate limiting only to backup-push/backup-fetch commands by @usernamedt in #1331
• Initial mvp mongodb binary backup by @KhurtinDN in #1285
• Add delta backups for Greenplum by @usernamedt in #1333
• Fix sorting in detailed mongodb backup list by @KhurtinDN in #1334
• Add @VGoshev and @KhurtinDN as codeowners of MongoDB part of Wal-G by @VGoshev in #1335
• Add tests, docs and minor fixes for Greenplum delta backups by @usernamedt in #1337
• Fix mongodb binary backup during mongod do checkpoint by @KhurtinDN in #1342
• Add rate limiter for mongodb by @KhurtinDN in #1343
• Add data catalog size for PostgreSQL/Greenplum backups by @usernamedt in #1344
• Add minor improvements for Greenplum by @usernamedt in #1345
• postgresq.md add cmake installation by @rogaliiik in #1348
• Support ARM64 builds on MacOS by @usernamedt in #1349
• Implement delete target backup for Greenplum by @usernamedt in #1350
• Add delete garbage command for Greenplum by @usernamedt in #1351
• Add limiters to Greenplum segment backup-push/backup-fetch by @usernamedt in #1352
• Fix failing docs builds by @usernamedt in #1354
• Fix distro name for ubuntu-20.04 in arm release by @williancolognesitrimble in #1355
• Bump golangci/golangci-lint-action from 3.2.0 to 3.3.0 by @dependabot in #1361
• Add minimal mongod config path option by @KhurtinDN in #1360
• Fix closing mongodb backup cursor by @KhurtinDN in #1359
• Replace mongodb binary backup to tar archive format by @KhurtinDN in #1357
• Fix binary backup of mongodb 4.2 by @KhurtinDN in #1364
• Add daemon mode for wal-push command by @rogaliiik in #1353
• Change error to warning when creating symlink, that already exists by @IncubusRK in #1320
• [Greenplum] do not fail backup on increment reading error by @usernamedt in #1365
• Only dump relevant env variables with LOG_LEVEL=DEVEL by @LeGEC in #1367
• Do not print secrets in debug logs by @LeGEC in #1368
• Bump golangci/golangci-lint-action from 3.3.0 to 3.3.1 by @dependabot in #1380
• enable using aws iam roles by @Qwiz in #1377
• notify_d watchdog for daemon service by @rogaliiik in #1371
• Use pure go implementation for zstd by @LeGEC in #1376
• Restore debug logging improvements by @usernamedt in #1389
• postgresql: fix data path mismatch on local backups by @LeGEC in #1383
• Fix Greenplum deleted AO segment file error handling by @usernamedt in #1391
• Implement max file size for mysql by @bularond in #1374
• Fix non-string config settings handling by @usernamedt in #1393
• Makefile: change compilation and linking of libbrotli to optional (and off by default) by @LeGEC in #1379
• add unit tests for the WriteBackupListDetails function by @tewboat in #1395
• Add recovery_target_timeline setting to Greenplum restore config by @usernamedt in #1398
• Add detailed json backup-list for Greenplum by @usernamedt in #1399
• Add restore-point-list command for gp by @rogaliiik in #1397
• Update golangci-lint version to 1.50.1 by @serprex in #1381
• Add restore-point-ts flag for Greenplum backup-fetch by @usernamedt in #1403
• Greenplum: populate pg_hba.conf on master host during restore by @usernamedt in #1404
• enable linters for cmd/redis internal/databases/redis and pkg/storages by @teewar2 in #1407
• Fix wal-g-mongo mongod restore procedure by @KhurtinDN in #1414
• Add mongo-rs-members parameter to binary mongo restore by @KhurtinDN in #1418
• [MongoDB] add opportunity restore mongodb backup with + 1 version by @Apelsin234 in #1417
• Add mongo-rs-name and mongo-rs-members parameters to binary mongo restore by @KhurtinDN in #1419
• add unit test for greenplum/backup_fetch_handler by @teewar2 in #1406
• fix bug with mixed up arguments by @Apelsin234 in #1421
• Postgres requires data dir to be 0700 or 0750. by @Aevin1387 in #1420
• Add support mongodb 6.0 by @KhurtinDN in #1416
• Fix connect to mongodb during restore procedure by @KhurtinDN in #1428
• Bump golangci/golangci-lint-action from 3.3.1 to 3.4.0 by @dependabot in #1426
• MDB-21194: [MongoDB] fix bug with purge garbage and change logic for retains by @Apelsin234 in #1429
• Implementation of the new binlog-server command by @Fizic in #1408
• Bump github.com/prometheus/client_golang from 1.7.1 to 1.11.1 in /internal/tools by @dependabot in #1433
• Bump golang.org/x/text from 0.3.6 to 0.3.8 in /internal/tools by @dependabot in #1437
• Implement retry download backup by @bularond in #1394
• adding push metrics as a defer in the upload method to make sure metrics are pushed even on failure by @francoispqt in #1423
• Revert #1423 by @usernamedt in #1442
• Implement database composer and partial fetch for Postgres by @Catmoonlight in #1434
• [MongoDB] fix retryable error from code to name by @Apelsin234 in #1444
• Bump actions/setup-go from 3 to 4 by @dependabot in #1446
• Adaptive timeout for mongodb restore by @Apelsin234 in #1449
• [Greenplum] Store metadata about backup restore points by @usernamedt in #1448
• Add Postgres partial restore by database names by @Catmoonlight in #1447
• Add tests postgres sentinel dto is incremental by @AlexUran in #1455
• Add tests for NewDtoSerializer by @denisSurkov in #1460
• feat: removing binlogs after using them by @Fizic in #1457
• Update golangci-lint in .github/workflows to 1.52.2 by @serprex in ...

2.0.2-rc1

This is a pre-release to test the aws iam roles config setting.

v2.0.1

Release notes:
This release mainly consists of various bug fixes and improvements.

Common:

• Add arm(aarch64) build pipeline for ubuntu20.04 binaries (#1304)

PostgreSQL:

• Add warning output on backup-fetch retries (#1325)
• Remove redundant check for --without-files-metadata flag (#1321)
• PostgreSQL 15 Support (#1319)
• Fix of deleting WAL-E backups (#1292)

MySQL/MariaDB:

• Fix a regression in MariaDB backup-push handler (#1313)
• Binlog cache fixes (#1281)

Greenplum:

• Optimize delete handler (#1300, #1302)
• Optimize storage listing queries during the backup push (#1312)
• Use AO files metadata instead of backup references (#1308)
• Refactor AO/AOCS storage metadata queries (#1280, #1278)

Storages:

• Azure authentication using the default credential chain (#1282)
• Update aws-sdk-go to v1.44.7 (#1262)
• Add check read/write to storage tools (#1275)

v2.0.0

Release notes:

This release contains a huge amount of new features, bug fixes, and performance improvements. It is worth noting that v2.0 Postgres delta backups are not backward-compatible with the pre-2.0 versions. It means that the v2.0 delta backup can't be restored with the v1.1 or earlier versions. Also, this release introduces the initial support for Greenplum physical backups.

WAL-G for MySQL now supports the parallel backup-push/backup-fetch and has an optimized binlog upload mechanism. WAL-G for SQLServer now supports the backup encryption and compression as well as the external storage backup import/export.

Common:

• Update lz4, brotli #1165, #1179
• Multiple libsodium fixes #1087, #1227
• Bash/zsh shell completions #1250, #1252
• Add a 'flags' subcommand #1033
• Disable CPU Limits in turbo mode #1121
• Metrics forwarding via statsd protocol #1240

SQLServer:

• Add SQLServerDBConcurrency setting to limit concurrent database operations #1067
• Add LATEST keyword handling #1110
• Add norecovery option for backup log command #1115
• Add SQLServer compression / encryption #1161
• Backup import and export to the external storage #1184

Postgres:

• Add CopyComposer for full backups optimization using remote copy API #1012
• Add streamed JSON sentinel serialization to decrease memory usage #1103
• Extract heavy files and tarfilesets from the backup sentinel #1114
• Introduce WALG_TAR_DISABLE_FSYNC to skip fsync during backup-fetch #1144
• Add reduced memory usage backup mode #1101
• Improvements and fixes for remote backup #1175
• Add pgBackRest backups restore support (beta) #1139, #1202
• Add 'wal-restore' command #977
• Add 'delete garbage' command #1217, #1218
• Return custom error code when WAL file does not exist #1195
• Add WALG_ALIVE_CHECK_INTERVAL to fail backup on Postgres process failure #1260
• Reformat LSN output #1259

MySQL:

• Add wal-g binlog-analyze #1085
• Check binlog GTIDs before pushing binlogs to S3 #1190
• Add --check-gtids to wal-g binlog-push #1085
• Add --until-binlog-last-modified-time option to wal-g-mysql binlog-replay #1154
• Improve MySQL restore performance #1131
• Allow backups on cascade replicas #1168
• Fix mysql backup of a lagging replica #1223
• Make mysql bilong-push status observable to 3rd party tools #1229

Greenplum:

• Initial physical backups support for GPDB 6.19.2+. API might change in the future versions.

Storages:

• Add get/put/rm/cat to storage tools #1069, #1164
• S3: Add download by range #1083
• S3: Add connection reset retries #1137
• S3: Add WALG_S3_MAX_RETRIES setting #1148
• S3: Fix S3_LOG_LEVEL setting #1158
• S3: AWS authentication fixes #1209
• S3: Yandex Cloud session token auth support #1213
• SSH: Ensure trailing delimiter in the storage prefix #1130
• Azure: Switch to the recommended azblob SDK #1135, #1171
• Azure: Fix ListFolder never returning error #1208
• Swift: Ignore files that were not found during ListFolder #1129

v2.0.0-rc1

This is a pre-release to test S3 authentication with mechanisms other than key/secret.

v1.1.3-rc

This pre-release introduces the Greenplum physical backups support.

v1.1.2-rc

This pre-release introduces the S3_LOG_LEVEL setting to help with debugging the S3 storage interaction issues.

v1.1.1-rc

This is a pre-release intended to test the s3 connection reset retries fix (#1137).

v1.1

Some interesting features/fixes:

• Include libsodium in the GitHub release builds. Exit with failure if libsodium is requested but WAL-G was not compiled with libsodium support #1062 CVE-2021-38599
• UserData is now required to be a valid JSON #1049
• backup-list now sorts backups by the astronomical time of the completition #907

Postgres:

• .history files overwrite prevention fix #1052
• wal-verify can now be run from a standby #1014
• wal-verify now ignores the permanent backups #1014, #1002

MongoDB:

• Add support for MongoDB 5.0 #1041

SQLServer:

• Fix MSSQL driver import #1038

MySQL:

• Add --turbo flag to disable limiters #1027
• Fix --detailed backup-list bug #1026
• Add --until flag to binlog-push #1005

Redis:

• Add 6.2 support & redis-cli bug workaround #1022

Storages

• S3: customer SSE key support #1042 (wal-g/storages#48)
• Azure: support AZURE_STORAGE_SAS_TOKEN without the leading '?' #1056
• Azure: disable the unnecessary syslog writes #1058