Get 4.x caught up with all the commits in 3.11 (#1072)

* Remove redundant interfaces and swap log and notification lines (#1019)

* Remove EventGenerator Interface.

* Remove EventObserver Interface.

* Remove BackupEvent Interface.

* Send SNS notification on backup after logging to account for the possibility of an Exception while trying to notify.

* Use synchronized list for thread-safety (#1018)

This list of PartETag is modified on multiple threads, so it
needs to be thread-safe.
S3FileSystem already uses a synchronized list, so do the
same here.

* Log backup failures rather than ignoring them. (#1025)

* Update CHANGELOG in advance of 3.11.95

* Print cleaner stack trace on failure to upload. (#1027)

* Switch from com.google.inject to JSR-330 javax.inject annotations for better compatibility

* Update CHANGELOG.md

* Reveal property to enable auto_snapshot. (#1031)

* Fix backup verification race condition causing missing notifications (#1034)

* Remove metaproxy validation it is never null in practice.

* Remove DateRange validation. It is never null in practice.

* Remove debug logging.

* Remove latest backup metadata validation. It is never null in practice.

* Consolidate repeated code into private verifyBackup.

* Change method names to better reflect what they do.

* Update latestResult wherever possible.

* Rewrite logic in findLatestVerfiedBackup to make it look more like verifyBackupsInRange.

* Change signature of BackupNotificationMgr.notify to not depend on BackupVerificationResult.

* Return all verified BackupMetadata instead of BackupVerificationResult when verifying en masse. It has enough information to skip the call to find the most recently verified backup.

Also, fix some tests that broke in this process: remove the check for the snapshot time in TestBackupVerification that only makes sense when the Path is for a file that does not exist. Also, mock the appropriate functions in MockBackupVerification in TestBackupVerificationTask.

* Rename findLatestVerifiedBackup responding to review comments.

* Reveal hook to allow operators to restore just to the most recent snapshot (#1035)

* Remove unused code.

* Remove redundant comments and vertical whitespace.

* Remove debug comments and now-redundant logger, simplify if-else and tighten error message for code style.

* Use final where applicable and remove it where redundant.

* Remove redundant BackupRestoreException from getIncrementals method signature.

* Split getting incremental files and snapshot files into separate methods.

* Reveal hook to allow operators to restore to the last valid snapshot.

* Remove added non-shaded Guava dependency pursuant to review comments.

* minor code modifications to simplify the nfpriam spring boot migration

* Update CHANGELOG.md

* Update CHANGELOG.md

* make the constructor public

* Update CHANGELOG.md

* remove the instance info from the DI (#1042)

* Update CHANGELOG.md

* Always TTL backups. (#1038)

* Fix Github CI by explicitly creating necessary directories. (#1045)

* Change the interface of PriamScheduler (#1049)

Change the interface of PriamScheduler

* minor name change (#1051)

* Update CHANGELOG.md

* Increment cross regional duplicate tokens to replicate the policy we have been applying manually. (#1048)

* Increment cross regional duplicate tokens to replicate the policy we have been applying manually. Throw when duplicate tokens are created in region because that would be an obvious error and we should not add two nodes in the same region so closely together.

* Improve error message on intra-regional duplicate token.

* Update CHANGELOG in advance of 3.11.101

* Rollback #1042: Change the interface of EC2RoleAssumptionCredential (#1052)

* Fix snapshot location regression in SNS messages. (#1054)

* Update CHANGELOG in advance of 3.11.103

* change the CassandraMonitor to public (#1056)

* Update CHANGELOG.md

* Add new constructor (#1064)

* Update CHANGELOG.md

* Add disk_failure_policy config (#1065)

* Update CHANGELOG.md

* fix Gson serilization issue (#1067)

* Update CHANGELOG.md

* Make block_for_peers_timeout_in_secs a first-class tunable. (#1069)

* Update CHANGELOG in advance of 3.11.108

* Fix TokenRetrieverTest

---------

Co-authored-by: Ammar Khaku <akhaku@users.noreply.github.com>
Co-authored-by: Cheng Wang <chengw@netflix.com>
Co-authored-by: Cheng Wang <107727158+chengw-netflix@users.noreply.github.com>

CHANGELOG.md

@@ -1,4 +1,52 @@
 # Changelog
+## 2023/09/02 3.11.108
+Tune block_for_peers_timeout_in_secs #1069
+
+## 2023/08/18 3.11.107
+Fix Gson serilization issue #1067
+
+## 2023/08/03 3.11.106
+Add config for Cassandra disk_failure_policy #1066
+
+## 2023/07/24 3.11.105
+Add constructor for BackupFileSystemContext #1064
+
+## 2023/05/05 3.11.104
+Change the constructor of the CassandraMonitor to be public. #1056
+
+## 2023/05/05 3.11.103
+Fix snapshot location regression in SNS messages. #1054
+
+## 2023/04/25 3.11.101
+*V1 Backups will be removed in the next release*
+Increment cross regional duplicate tokens to replicate the policy we have been applying manually. (#1048)
+
+## 2023/04/24 3.11.100
+Always TTL backups. (#1038)
+Fix Github CI by explicitly creating necessary directories. (#1045)
+Change the interface of PriamScheduler. (#1049)
+Minor name change in the PriamScheduler. (#1051)
+
+## 2023/04/11 3.11.99
+Change the interface of EC2RoleAssumptionCredential (#1042)
+
+## 2023/03/30 3.11.98
+Change the CassandraOperations constructor to be public (#1037)
+
+## 2023/03/29 3.11.97
+Reveal property to enable auto_snapshot. (#1031)
+Fix backup verification race condition causing missing notifications (#1034)
+Reveal hook to allow operators to restore just to the most recent snapshot (#1035)
+Minor changes to some Java constructors to public for better dependency injection. (#1036)
+
+## 2023/02/28 3.11.96
+Switch from com.google.inject to JSR-330 javax.inject annotations for better compatibility (#1030)
+
+## 2023/1/21 3.11.95
+Log backup failures rather than ignoring them. (#1025) 
+Use synchronized list for thread-safety (#1018) 
+Remove redundant interfaces and swap log and notification lines on backup failure. (#1019) 
+
 ## 2022/11/15 3.11.94
 (#1013) Create operator-specifiable time such that if a backup file was written before then it is automatically compressed using SNAPPY before upload.
 (#1012) Ensure SI files get into meta file properly.

priam/src/main/java/com/netflix/priam/PriamServer.java

@@ -16,8 +16,6 @@
  */
 package com.netflix.priam;
 
-import com.google.inject.Inject;
-import com.google.inject.Singleton;
 import com.netflix.priam.backup.BackupService;
 import com.netflix.priam.backupv2.BackupV2Service;
 import com.netflix.priam.cluster.management.ClusterManagementService;
@@ -33,6 +31,8 @@
 import com.netflix.priam.utils.Sleeper;
 import com.netflix.priam.utils.SystemUtils;
 import java.io.IOException;
+import javax.inject.Inject;
+import javax.inject.Singleton;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 

priam/src/main/java/com/netflix/priam/aws/AWSMembership.java

@@ -25,13 +25,13 @@
 import com.amazonaws.services.ec2.model.*;
 import com.amazonaws.services.ec2.model.Filter;
 import com.google.common.collect.ImmutableSet;
-import com.google.inject.Inject;
-import com.google.inject.name.Named;
 import com.netflix.priam.config.IConfiguration;
 import com.netflix.priam.cred.ICredential;
 import com.netflix.priam.identity.IMembership;
 import com.netflix.priam.identity.config.InstanceInfo;
 import java.util.*;
+import javax.inject.Inject;
+import javax.inject.Named;
 import org.apache.commons.lang3.StringUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;

priam/src/main/java/com/netflix/priam/aws/RemoteBackupPath.java

@@ -20,7 +20,6 @@
 import com.google.common.base.Preconditions;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableSet;
-import com.google.inject.Inject;
 import com.netflix.priam.backup.AbstractBackupPath;
 import com.netflix.priam.compress.CompressionType;
 import com.netflix.priam.config.IConfiguration;
@@ -33,6 +32,7 @@
 import java.util.Date;
 import java.util.List;
 import java.util.Optional;
+import javax.inject.Inject;
 
 /**
  * Represents location of an object on the remote file system. All the objects will be keyed with a

priam/src/main/java/com/netflix/priam/aws/S3CrossAccountFileSystem.java

@@ -15,13 +15,13 @@
 
 import com.amazonaws.services.s3.AmazonS3;
 import com.amazonaws.services.s3.AmazonS3Client;
-import com.google.inject.Inject;
-import com.google.inject.Singleton;
-import com.google.inject.name.Named;
 import com.netflix.priam.aws.auth.IS3Credential;
 import com.netflix.priam.backup.IBackupFileSystem;
 import com.netflix.priam.config.IConfiguration;
 import com.netflix.priam.identity.config.InstanceInfo;
+import javax.inject.Inject;
+import javax.inject.Named;
+import javax.inject.Singleton;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 

priam/src/main/java/com/netflix/priam/aws/S3EncryptedFileSystem.java

@@ -18,11 +18,6 @@
 import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
 import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
 import com.amazonaws.services.s3.model.PartETag;
-import com.google.common.collect.Lists;
-import com.google.inject.Inject;
-import com.google.inject.Provider;
-import com.google.inject.Singleton;
-import com.google.inject.name.Named;
 import com.netflix.priam.backup.AbstractBackupPath;
 import com.netflix.priam.backup.BackupRestoreException;
 import com.netflix.priam.backup.DynamicRateLimiter;
@@ -39,8 +34,14 @@
 import java.nio.file.Path;
 import java.nio.file.Paths;
 import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Collections;
 import java.util.Iterator;
 import java.util.List;
+import javax.inject.Inject;
+import javax.inject.Named;
+import javax.inject.Provider;
+import javax.inject.Singleton;
 import org.apache.commons.io.IOUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -116,7 +117,7 @@ protected long uploadFileImpl(AbstractBackupPath path, Instant target)
         DataPart part =
                 new DataPart(config.getBackupPrefix(), remotePath, initResponse.getUploadId());
         // Metadata on number of parts to be uploaded
-        List<PartETag> partETags = Lists.newArrayList();
+        List<PartETag> partETags = Collections.synchronizedList(new ArrayList<>());
 
         // Read chunks from src, compress it, and write to temp file
         File compressedDstFile = new File(localPath.toString() + ".compressed");

priam/src/main/java/com/netflix/priam/aws/S3FileSystem.java

@@ -20,10 +20,6 @@
 import com.amazonaws.services.s3.S3ResponseMetadata;
 import com.amazonaws.services.s3.model.*;
 import com.google.common.base.Preconditions;
-import com.google.inject.Inject;
-import com.google.inject.Provider;
-import com.google.inject.Singleton;
-import com.google.inject.name.Named;
 import com.netflix.priam.aws.auth.IS3Credential;
 import com.netflix.priam.backup.AbstractBackupPath;
 import com.netflix.priam.backup.BackupRestoreException;
@@ -47,6 +43,10 @@
 import java.util.Iterator;
 import java.util.List;
 import java.util.concurrent.atomic.AtomicInteger;
+import javax.inject.Inject;
+import javax.inject.Named;
+import javax.inject.Provider;
+import javax.inject.Singleton;
 import org.apache.commons.io.IOUtils;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;

priam/src/main/java/com/netflix/priam/aws/S3FileSystemBase.java

@@ -22,7 +22,6 @@
 import com.amazonaws.services.s3.model.lifecycle.*;
 import com.google.common.collect.Lists;
 import com.google.common.util.concurrent.RateLimiter;
-import com.google.inject.Provider;
 import com.netflix.priam.backup.AbstractBackupPath;
 import com.netflix.priam.backup.AbstractFileSystem;
 import com.netflix.priam.backup.BackupRestoreException;
@@ -37,6 +36,7 @@
 import java.util.Optional;
 import java.util.concurrent.LinkedBlockingQueue;
 import java.util.stream.Collectors;
+import javax.inject.Provider;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 

priam/src/main/java/com/netflix/priam/aws/SDBInstanceData.java

@@ -20,12 +20,12 @@
 import com.amazonaws.services.simpledb.AmazonSimpleDB;
 import com.amazonaws.services.simpledb.AmazonSimpleDBClient;
 import com.amazonaws.services.simpledb.model.*;
-import com.google.inject.Inject;
-import com.google.inject.Singleton;
 import com.netflix.priam.config.IConfiguration;
 import com.netflix.priam.cred.ICredential;
 import com.netflix.priam.identity.PriamInstance;
 import java.util.*;
+import javax.inject.Inject;
+import javax.inject.Singleton;
 
 /** DAO for handling Instance identity information such as token, zone, region */
 @Singleton

priam/src/main/java/com/netflix/priam/aws/SDBInstanceFactory.java

@@ -18,13 +18,13 @@
 
 import com.amazonaws.AmazonServiceException;
 import com.google.common.collect.ImmutableSet;
-import com.google.inject.Inject;
-import com.google.inject.Singleton;
 import com.netflix.priam.identity.IPriamInstanceFactory;
 import com.netflix.priam.identity.PriamInstance;
 import com.netflix.priam.identity.config.InstanceInfo;
 import java.util.*;
 import java.util.stream.Collectors;
+import javax.inject.Inject;
+import javax.inject.Singleton;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 

priam/src/main/java/com/netflix/priam/aws/UpdateCleanupPolicy.java

@@ -16,15 +16,15 @@
  */
 package com.netflix.priam.aws;
 
-import com.google.inject.Inject;
-import com.google.inject.Singleton;
-import com.google.inject.name.Named;
 import com.netflix.priam.backup.IBackupFileSystem;
 import com.netflix.priam.config.IConfiguration;
 import com.netflix.priam.scheduler.SimpleTimer;
 import com.netflix.priam.scheduler.Task;
 import com.netflix.priam.scheduler.TaskTimer;
 import com.netflix.priam.utils.RetryableCallable;
+import javax.inject.Inject;
+import javax.inject.Named;
+import javax.inject.Singleton;
 
 /** Updates the cleanup policy for the bucket */
 @Singleton

priam/src/main/java/com/netflix/priam/aws/auth/EC2RoleAssumptionCredential.java

@@ -15,10 +15,10 @@
 
 import com.amazonaws.auth.AWSCredentialsProvider;
 import com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider;
-import com.google.inject.Inject;
 import com.netflix.priam.config.IConfiguration;
 import com.netflix.priam.cred.ICredential;
 import com.netflix.priam.identity.config.InstanceInfo;
+import javax.inject.Inject;
 
 public class EC2RoleAssumptionCredential implements ICredential {
     private static final String AWS_ROLE_ASSUMPTION_SESSION_NAME = "AwsRoleAssumptionSession";

priam/src/main/java/com/netflix/priam/aws/auth/S3RoleAssumptionCredential.java

@@ -16,10 +16,10 @@
 import com.amazonaws.auth.AWSCredentials;
 import com.amazonaws.auth.AWSCredentialsProvider;
 import com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider;
-import com.google.inject.Inject;
-import com.google.inject.Singleton;
 import com.netflix.priam.config.IConfiguration;
 import com.netflix.priam.cred.ICredential;
+import javax.inject.Inject;
+import javax.inject.Singleton;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 

priam/src/main/java/com/netflix/priam/backup/AbstractBackup.java

@@ -16,7 +16,6 @@
  */
 package com.netflix.priam.backup;
 
-import com.google.inject.Inject;
 import com.netflix.priam.config.IConfiguration;
 import com.netflix.priam.scheduler.Task;
 import com.netflix.priam.utils.SystemUtils;
@@ -29,6 +28,7 @@
 import java.util.HashSet;
 import java.util.Optional;
 import java.util.Set;
+import javax.inject.Inject;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;