v3.0.0

Hello everyone!

This is a much-anticipated release. In the future, we will strive to avoid accumulating such a large number of changes. We will code frequently, ship frequently, and release frequently!

Noteworthy Changes

• Every database engine now benefits from failover storage. This technique can save the day if one of the storage systems goes down. However, do not forget to monitor them anyway.
• For Postgres, we have a WAL archiving daemon available. Its primary purpose is to improve the efficiency of parallel archiving. Additionally, WAL-G supports the partial restoration of a cluster. If you have large, unnecessary tables, you can exclude them from the restoration process, the Postgres will skip over the WAL records for these tables.
• Greenplum's PITR (Point-in-Time Recovery) is undergoing its maturation process. Partial restore is supported in a similar way to Postgres. We now have tools to verify the integrity of AO\AOCS files in storage without actually restoring backups. This technique is necessary to ensure developers can sleep well, as Greenplum backups are relatively new, and extra precautions need to be taken to ensure they are safe and reliable.
• MongoDB backups can now benefit from binary backups, which significantly reduce the impact of backups on a running workload.
• MySQL now supports incremental backups, also known as delta backups. However, this feature is still in beta mode and the API may change in the future.
• Also, we now have beta support for etcd. Changes necessary for real PITR have not yet been merged into etcd, and can be found at etcd-io/etcd#17233.

Detailed changes log

• Correct the --without-files-metadata docs by @usernamedt in #1327
• expose zstd compression by @heinerstilz in #1326
• [WIP] Multithread downloading directory by @hariton27sy in #1297
• Add WALG_GP_AOSEG_SIZE_THRESHOLD setting by @usernamedt in #1329
• Respect WALG_NETWORK_RATE_LIMIT globally by @usernamedt in #1330
• Restrict rate limiting only to backup-push/backup-fetch commands by @usernamedt in #1331
• Initial mvp mongodb binary backup by @KhurtinDN in #1285
• Add delta backups for Greenplum by @usernamedt in #1333
• Fix sorting in detailed mongodb backup list by @KhurtinDN in #1334
• Add @VGoshev and @KhurtinDN as codeowners of MongoDB part of Wal-G by @VGoshev in #1335
• Add tests, docs and minor fixes for Greenplum delta backups by @usernamedt in #1337
• Fix mongodb binary backup during mongod do checkpoint by @KhurtinDN in #1342
• Add rate limiter for mongodb by @KhurtinDN in #1343
• Add data catalog size for PostgreSQL/Greenplum backups by @usernamedt in #1344
• Add minor improvements for Greenplum by @usernamedt in #1345
• postgresq.md add cmake installation by @rogaliiik in #1348
• Support ARM64 builds on MacOS by @usernamedt in #1349
• Implement delete target backup for Greenplum by @usernamedt in #1350
• Add delete garbage command for Greenplum by @usernamedt in #1351
• Add limiters to Greenplum segment backup-push/backup-fetch by @usernamedt in #1352
• Fix failing docs builds by @usernamedt in #1354
• Fix distro name for ubuntu-20.04 in arm release by @williancolognesitrimble in #1355
• Bump golangci/golangci-lint-action from 3.2.0 to 3.3.0 by @dependabot in #1361
• Add minimal mongod config path option by @KhurtinDN in #1360
• Fix closing mongodb backup cursor by @KhurtinDN in #1359
• Replace mongodb binary backup to tar archive format by @KhurtinDN in #1357
• Fix binary backup of mongodb 4.2 by @KhurtinDN in #1364
• Add daemon mode for wal-push command by @rogaliiik in #1353
• Change error to warning when creating symlink, that already exists by @IncubusRK in #1320
• [Greenplum] do not fail backup on increment reading error by @usernamedt in #1365
• Only dump relevant env variables with LOG_LEVEL=DEVEL by @LeGEC in #1367
• Do not print secrets in debug logs by @LeGEC in #1368
• Bump golangci/golangci-lint-action from 3.3.0 to 3.3.1 by @dependabot in #1380
• enable using aws iam roles by @Qwiz in #1377
• notify_d watchdog for daemon service by @rogaliiik in #1371
• Use pure go implementation for zstd by @LeGEC in #1376
• Restore debug logging improvements by @usernamedt in #1389
• postgresql: fix data path mismatch on local backups by @LeGEC in #1383
• Fix Greenplum deleted AO segment file error handling by @usernamedt in #1391
• Implement max file size for mysql by @bularond in #1374
• Fix non-string config settings handling by @usernamedt in #1393
• Makefile: change compilation and linking of libbrotli to optional (and off by default) by @LeGEC in #1379
• add unit tests for the WriteBackupListDetails function by @tewboat in #1395
• Add recovery_target_timeline setting to Greenplum restore config by @usernamedt in #1398
• Add detailed json backup-list for Greenplum by @usernamedt in #1399
• Add restore-point-list command for gp by @rogaliiik in #1397
• Update golangci-lint version to 1.50.1 by @serprex in #1381
• Add restore-point-ts flag for Greenplum backup-fetch by @usernamedt in #1403
• Greenplum: populate pg_hba.conf on master host during restore by @usernamedt in #1404
• enable linters for cmd/redis internal/databases/redis and pkg/storages by @teewar2 in #1407
• Fix wal-g-mongo mongod restore procedure by @KhurtinDN in #1414
• Add mongo-rs-members parameter to binary mongo restore by @KhurtinDN in #1418
• [MongoDB] add opportunity restore mongodb backup with + 1 version by @Apelsin234 in #1417
• Add mongo-rs-name and mongo-rs-members parameters to binary mongo restore by @KhurtinDN in #1419
• add unit test for greenplum/backup_fetch_handler by @teewar2 in #1406
• fix bug with mixed up arguments by @Apelsin234 in #1421
• Postgres requires data dir to be 0700 or 0750. by @Aevin1387 in #1420
• Add support mongodb 6.0 by @KhurtinDN in #1416
• Fix connect to mongodb during restore procedure by @KhurtinDN in #1428
• Bump golangci/golangci-lint-action from 3.3.1 to 3.4.0 by @dependabot in #1426
• MDB-21194: [MongoDB] fix bug with purge garbage and change logic for retains by @Apelsin234 in #1429
• Implementation of the new binlog-server command by @Fizic in #1408
• Bump github.com/prometheus/client_golang from 1.7.1 to 1.11.1 in /internal/tools by @dependabot in #1433
• Bump golang.org/x/text from 0.3.6 to 0.3.8 in /internal/tools by @dependabot in #1437
• Implement retry download backup by @bularond in #1394
• adding push metrics as a defer in the upload method to make sure metrics are pushed even on failure by @francoispqt in #1423
• Revert #1423 by @usernamedt in #1442
• Implement database composer and partial fetch for Postgres by @Catmoonlight in #1434
• [MongoDB] fix retryable error from code to name by @Apelsin234 in #1444
• Bump actions/setup-go from 3 to 4 by @dependabot in #1446
• Adaptive timeout for mongodb restore by @Apelsin234 in #1449
• [Greenplum] Store metadata about backup restore points by @usernamedt in #1448
• Add Postgres partial restore by database names by @Catmoonlight in #1447
• Add tests postgres sentinel dto is incremental by @AlexUran in #1455
• Add tests for NewDtoSerializer by @denisSurkov in #1460
• feat: removing binlogs after using them by @Fizic in #1457
• Update golangci-lint in .github/workflows to 1.52.2 by @serprex in #1464
• support flag latest for backup and oplog reply by @Apelsin234 in #1463
• add tests to LSN Regex by @IvanSibirtsev in #1462
• Refactor storage abstractions by @usernamedt in #1467
• use ProtonMail/go-crypto/openpgp instead of golang.org/x/crypto/openpgp by @IvanSibirtsev in #1468
• fix: flag in mysql binlog-server by @Fizic in #1475
• [Postgres] Add failover storages for wal-push and wal-fetch by @usernamedt in #1466
• Add prefetch prefix for wal-prefetch cmd by @denisSurkov in #1471
• Make wal-g-mysql binlog-server work with new MySQL by @ostinru in #1474
• add GetLogsDstSettings and ConfigureCompressor tests by @amagoosebitch in #1479
• Correct mistakes and inaccuracies I found in the documentation by @rdjjke in #1478
• Add backup utility tests by @Cockamamie in #1481
• Implement st transfer command by @rdjjke in #1480
• Bump golangci/golangci-lint-action from 3.4.0 to 3.5.0 by @dependabot in #1484
• Fix panics in st transfer command when --overwrite flag is set by @rdjjke in #1485
• MongoDB binary restore: flag to not perform host reconfiguration by @ElectricR in #1488
• Enable database alive check for Postgresql by default by @usernamedt in #1486
• continue inactive PR #1453 (Tables partial restore for Postgresql) by @nikifkon in #1490
• export daemon message type & constants by @sergey-arefev in #1494
• Fix mariadb tests by @usernamedt in #1495
• Update Golang to 1.20 version by @anikin-aa in #1492
• Bump golangci/golangci-lint-action from 3.5.0 to 3.6.0 by @dependabot in #1496
• [Postgres] Fix races in alive storage checker by @usernamedt in #1498
• wal-fetch command in daemon mode by @sergey-arefev in #1499
• walg-daemon-client by @sergey-arefev in #1500
• fix action order in test jobs: checkout -> setup-go by @sergey-arefev in #1501
• Add strict requirements for default tablespace relfilenode parsing by @usernamedt in #1503
• Add partial restore for Greenplum by @nikifkon in #1489
• fix daemon client build ldflags values by @sergey-arefev in #1504
• [PG] wal-g daemon \ client: ArchiveNonExistence by @sergey-arefev in #1505
• Support failover storages in backup-push and backups in st transfer by @rdjjke in #1502
• fix SdNotify: send in goroutine by @sergey-arefev in #1508
• Add Greenplum AO/AOCS files deduplication age limit by @usernamedt in #1507
• Add WALG_ prefix to all the s3 related envs by @anikin-aa in #1512
• chore: remove refs to deprecated io/ioutil by @testwill in #1515
• Fix panic in the multistorage folder_reader.go by @usernamedt in #1519
• Fix typo in wal-g storage tool cat handler by @reshke in #1520
• Fix deadlock in GpTarBallComposer between AddHeader and addFileWorker by @mialinx in #1524
• Pin the CI golang version to 1.20.5 by @usernamedt in #1525
• Fix WAL-G prefetch leaving zombie processes in daemon mode by @usernamedt in #1526
• Dead lock fix by @noname0443 in #1530
• MariaDB Test Fix by @noname0443 in #1533
• Support failover storages in backup-fetch by @rdjjke in #1514
• support flag latest for backup-show by @Apelsin234 in #1529
• Now supported: additional headers for s3 requests by @debebantur in #1535
• Fix bug with failover storages in backup-fetch by @rdjjke in #1537
• MongoDB binary restore: Added flag for particular replica set member ids by @ElectricR in #1532
• Bump gopkg.in/yaml.v3 from 3.0.0-20210107192922-496545a6307b to 3.0.0 in /internal/tools by @dependabot in #1541
• Correct typo in PG doc by @Quentinchampenois in #1543
• Enhancement: Read file content from STDIN for wal-g st put. by @reshke in #1539
• WAL-G storage tool copy command by @reshke in #1531
• Don't fail commands if unable to write storage statuses cache file by @rdjjke in #1544
• [PG] Optimize docker-compose & dockerfiles for postgresql docker tests by @sergey-arefev in #1510
• Print storage names in backup-list by @rdjjke in #1538
• Feature/envelope pgp by @Alviner in #1547
• Fix daemon client size regression & add test by @usernamedt in #1552
• fix: use yc endpoint and optional file creds with envelope gpg by @Alviner in #1553
• Pass Context around Folder upload functions, add PG WAL push timeout by @ne2pit in #1546
• PG Integration tests and linter local dev improvements by @ne2pit in #1551
• Cancel long mongodb uploads by context by @rdjjke in #1548
• Allow to specify storage names in PG commands by @rdjjke in #1550
• fix: read exactly bytes by @Alviner in #1555
• add myself to reviewer for mongo and redis by @Apelsin234 in #1558
• Fix target storage name specifying in PG commands by @rdjjke in #1557
• fix: b64 encoded envelope key length by @Alviner in #1556
• Add gpg key id to the envelope encryption header by @Alviner in #1559
• Feature/use external key identifier by @Alviner in #1561
• upgrade github.com/yandex-cloud/go-sdk to fix CVE-2020-26160 by @zlcnju in #1563
• Bump actions/checkout from 3 to 4 by @dependabot in #1554
• Bump golangci/golangci-lint-action from 3.6.0 to 3.7.0 by @dependabot in #1534
• Don't require user $HOME dir to exist by @rdjjke in #1565
• Support failover storages in backup deletion by @rdjjke in #1560
• Improve storage fail detection by putting the object by @ne2pit in #1549
• Fix golangci-lint found issues on master by @usernamedt in #1567
• Don't trim slash at the beginning of ssh folder path by @rdjjke in #1568
• Establish ssh connections lazily by @rdjjke in #1569
• Add tests for GenericMetaFetcher.Fetch by @olafeonk in #1572
• Log storage names by @rdjjke in #1570
• Fix deleting backups from failover storages in PG by @rdjjke in #1573
• Don't throw error in SSH folder if deleting object doesn't exist by @rdjjke in #1574
• Respect upload context in bguploader by @usernamedt in #1575
• Introduce a setting to turn off storage alive checks by @rdjjke in #1579
• Add MongoDB 7.0 support. Remove MongoDB 4.2 support. Fix dropIndexes operation replay. by @dapqa in #1583
• Recheck storages and don't wait cache to expire if they all are dead by @rdjjke in #1582
• MySQL incremental backup by @ostinru in #1571
• Fixed UUID filtering for nested applyOps commands by @dapqa in #1596
• Reuse SSH connections opened once when daemon started by @rdjjke in #1586
• Chec AO/AOCS table EOF by @debebantur in #1594
• S3 throttling test by @debebantur in #1591
• Backup selector tests by @KosovGrigorii in #1603
• Initial ETCD + WAL-G implementation by @KosovGrigorii in #1605
• Fix delete garbage in gp by @debebantur in #1609
• check that delete garbage picks correct oldest backup by @KosovGrigorii in #1607
• Retry fetch file by @debebantur in #1608
• Detect alive storages by stats by @rdjjke in #1602
• Fix closing SSH storage + log conection errors by @rdjjke in #1617
• fixed sequential tests by @debebantur in #1618
• Partial restore wildcard by @debebantur in #1613
• Etcd walg wal push wal fetch by @KosovGrigorii in #1620
• Add namespace to greenplum partial-restore by @debebantur in #1624
• Bump actions/cache from 3 to 4 by @dependabot in #1630
• Bump actions/setup-go from 4 to 5 by @dependabot in #1606
• propagate error from ParseLSN function by @sivukhin in #1612
• MongoDB binary restore: Add resetup of shardIdentity field for mongod… by @ElectricR in #1622
• Add backup selection to ao/aocs check by @debebantur in #1631
• fixes for ao/aocs length check by @debebantur in #1633
• STORAGES doc using wrong Azure variable name by @fgendc in #1626
• Add --after option support for PostgreSQL delete retain command by @victor-tsykanov in #1424
• Add tests for partitionStrings by @ReshetnikovPavel in #1598
• MongoDB binary restore: use most recent to given oplog from oplog arc… by @ElectricR in #1638
• MongoDB binary restore: If shardIdentity document does not exist, ass… by @ElectricR in #1641
• Bump golangci/golangci-lint-action from 3.7.0 to 4.0.0 by @dependabot in #1643
• Pinned minio version in throttling test by @debebantur in #1645
• Avoid resorting to pg_resetwal in catchup test by @x4m in #1642
• GP, prevent multiple backups in the same time by @debebantur in #1634
• Fix MariaDB and GP tests by @ostinru in #1647
• Allow extra tags to be added to statsd metrics by @nick-kentik in #1649
• Fixed file walking in aochecks by @debebantur in #1654
• Added s3 response code metrics by @debebantur in #1653
• Bump softprops/action-gh-release from 1 to 2 by @dependabot in #1655
• Added config settings for handling S3 throttling by @debebantur in #1656
• Bump google.golang.org/protobuf from 1.26.0-rc.1 to 1.33.0 in /internal/tools by @dependabot in #1658
• Changed s3_response_code metric to gauge by @debebantur in #1660

New Contributors

• @heinerstilz made their first contribution in #1326
• @KhurtinDN made their first contribution in #1285
• @VGoshev made their first contribution in #1335
• @rogaliiik made their first contribution in #1348
• @williancolognesitrimble made their first contribution in #1355
• @IncubusRK made their first contribution in #1320
• @Qwiz made their first contribution in #1377
• @bularond made their first contribution in #1374
• @tewboat made their first contribution in #1395
• @teewar2 made their first contribution in #1407
• @Apelsin234 made their first contribution in #1417
• @Fizic made their first contribution in #1408
• @francoispqt made their first contribution in #1423
• @Catmoonlight made their first contribution in #1434
• @AlexUran made their first contribution in #1455
• @denisSurkov made their first contribution in #1460
• @IvanSibirtsev made their first contribution in #1462
• @amagoosebitch made their first contribution in #1479
• @rdjjke made their first contribution in #1478
• @Cockamamie made their first contribution in #1481
• @ElectricR made their first contribution in #1488
• @nikifkon made their first contribution in #1490
• @sergey-arefev made their first contribution in #1494
• @anikin-aa made their first contribution in #1492
• @testwill made their first contribution in #1515
• @noname0443 made their first contribution in #1530
• @debebantur made their first contribution in #1535
• @Quentinchampenois made their first contribution in #1543
• @Alviner made their first contribution in #1547
• @ne2pit made their first contribution in #1546
• @zlcnju made their first contribution in #1563
• @olafeonk made their first contribution in #1572
• @dapqa made their first contribution in #1583
• @KosovGrigorii made their first contribution in #1603
• @sivukhin made their first contribution in #1612
• @fgendc made their first contribution in #1626
• @victor-tsykanov made their first contribution in #1424
• @ReshetnikovPavel made their first contribution in #1598
• @nick-kentik made their first contribution in #1649

Full Changelog: v2.0.1...v3.0.0