PHOENIX-7108 Provide support for pruning expired rows of views using Phoenix level compactions #1855

jpisaac · 2024-03-12T00:07:58Z

No description provided.

jpisaac · 2024-03-12T00:11:29Z

Rebased old PR with the feature branch master.
Fixed ViewTTLIT test failures.

jpisaac · 2024-03-12T00:16:43Z

TableTTLIT failing with the following error

Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): java.io.IOException: Key a/0:_0/1710201666000/Put/vlen=1/seqid=137 followed by a smaller key a/0:/1710201636000/DeleteFamily/vlen=0/seqid=107 in cf 77875878a1beda46043a6f04753c8bdc/0
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:451)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349)
Caused by: java.lang.AssertionError: Key a/0:_0/1710201666000/Put/vlen=1/seqid=137 followed by a smaller key a/0:/1710201636000/DeleteFamily/vlen=0/seqid=107 in cf 77875878a1beda46043a6f04753c8bdc/0

@kadirozde will need your help to resolve this TF

phoenix-core/src/main/java/org/apache/phoenix/compile/WhereOptimizer.java

kadirozde · 2024-03-18T16:48:08Z

phoenix-core/src/main/java/org/apache/phoenix/coprocessor/CompactionScanner.java

@@ -63,8 +89,8 @@ public class CompactionScanner implements InternalScanner {
    private final Configuration config;
    private final RegionCoprocessorEnvironment env;
    private long maxLookbackWindowStart;
-    private long ttlWindowStart;
-    private long ttl;
+    //private long ttlWindowStart;


These commented lines should be removed

kadirozde · 2024-03-18T16:50:29Z

phoenix-core/src/main/java/org/apache/phoenix/coprocessor/CompactionScanner.java

-        this.emptyCF = emptyCF;
-        this.emptyCQ = emptyCQ;
+        this.emptyCF = SchemaUtil.getEmptyColumnFamily(table);
+        this.emptyCQ = table.getEncodingScheme() == PTable.QualifierEncodingScheme.NON_ENCODED_QUALIFIERS ?


Can add a new method called SchemaUtil.getEmptyColumnName(table)?

Added SchemaUtil.getEncodingScheme(table), I am assuming that's what u meant

I meant SchemaUtil.getEmptyColumnName(table) or maybe better to call it SchemaUtil.getEmptyColumnQualifier(table). Since we have SchemaUtil.getEmptyColumnFamily(table), we should also have SchemaUtil.getEmptyColumnQualifier(table)

kadirozde · 2024-03-18T16:54:47Z

phoenix-core/src/main/java/org/apache/phoenix/coprocessor/CompactionScanner.java

-        phoenixLevelRowCompactor = new PhoenixLevelRowCompactor();
-        hBaseLevelRowCompactor = new HBaseLevelRowCompactor();
+        emptyCFStore = familyCount == 1 || columnFamilyName.equals(Bytes.toString(emptyCF)) || localIndex;
+        // TODO: check if is it appropriate to throw an IOException here


It should be okay to generate an exception here. What is the reason for throwing an exception?

Exceptions can be thrown if SYSCAT is not available or calls to SYSCAT fails.

kadirozde · 2024-03-18T17:14:05Z

phoenix-core/src/main/java/org/apache/phoenix/coprocessor/CompactionScanner.java

+                this.isMultiTenant = table.isMultiTenant();
+
+                int startingPKPosition = 0;
+                if (this.isMultiTenant && this.isSalted && this.isIndexTable) {


Can we turn this into more readable if then else statements by checking only one condition per if statement? I think it is better to create a lookup table for this, for example, int[][][] StartingPKPosition = new int[2][2][2];

As discussed offline created a more verbose if then else statement for the 8 conditions.

kadirozde · 2024-03-29T04:58:54Z

phoenix-core/src/main/java/org/apache/phoenix/util/matcher/RowKeyMatcher.java

+			return node;
+		}
+
+		int index = (int) key[depth] % (int) R;


Would the following be sufficient to convert byte to unsigned int and simplify the code?
int index = key[depth] & 0xFF;

kadirozde · 2024-04-01T17:07:53Z

phoenix-core/src/main/java/org/apache/phoenix/util/matcher/RowKeyMatcher.java

+	private static final Logger LOGGER = LoggerFactory.getLogger(RowKeyMatcher.class);
+
+	public static final int R = 256;
+	private TrieNode root = new TrieNode();


The trie data structure implemented here is not efficient in terms of space and time. We should be using a compressed trie data structure. I suggest using PATRICIA, please see https://commons.apache.org/proper/commons-collections/apidocs/org/apache/commons/collections4/trie/PatriciaTrie.html

We do not have to replace the current implementation of trie with PATRICIA in this PR. We can do it in a separate improvement jira later.

kadirozde · 2024-04-01T17:26:37Z

phoenix-core/src/main/java/org/apache/phoenix/util/ViewUtil.java

+                        }
+                        Connection
+                                tableConnection =
+                                QueryUtil.getConnectionOnServer(tenantProps, configuration);


The connection is not closed here. I suggest opening it within a try statement. Also, can we use one global connection instead of a separate connection for every view?

jpisaac · 2024-05-25T01:29:16Z

FYI @kadirozde I think the PR is almost ready except to removing debug logging and additional docs and comments

Jacob Isaac and others added 15 commits March 11, 2024 15:38

Initial integration of ViewTTL with PhoenixRowTracker

34d2163

Added view utils

f207424

More view util changes

d706960

Fixed parent TTL issue

60a689c

Added IT tests for ViewTTL

4ede4b5

Code cleanup

ee29300

Code refactoring and cleanup

16308ca

Test refactoring and code cleanup

180b61d

Added Local Index tests and test cleanup

cbbeaf6

Added support for view indexes and test refactoring

e8c7f3d

Removed some debug only classes

f3ce633

Addressed initial review comments and test refactoring

8f013f3

Adding tests

eb8cd26

Added PHOENIX-7040 (ADDENDUM) files

6fdd95c

Refacfored code for overlapping cases, naming and tests

f9f8508

jpisaac requested review from kadirozde and lokiore March 12, 2024 00:16

Fixed test failures in ViewTTLIT after rebase

eb36c81

jpisaac force-pushed the PHOENIX-7108-PR-2 branch from a992a81 to eb36c81 Compare March 13, 2024 01:14

kadirozde reviewed Mar 18, 2024

View reviewed changes

jpisaac added 2 commits March 19, 2024 19:55

Addressed comments, test failures and some code refactoring

016766a

Fixed namespace enabled test failure in NamespaceEnabledStatsCollectorIT

5a7e14e

kadirozde reviewed Mar 29, 2024

View reviewed changes

kadirozde reviewed Apr 1, 2024

View reviewed changes

Adding missing test file after rebase the previous PR (PR-1733)

6668a92

jpisaac force-pushed the PHOENIX-7108-PR-2 branch from 0177a5a to 6668a92 Compare April 1, 2024 18:57

Addressed some of the recent comments from Kadir

8f22312

jpisaac added 4 commits April 23, 2024 19:37

Refactored code and add tests for region boundary optimizations

1a5b0a0

Fixed null bytes checks when salted tables are involved

c81841d

Refactored code and add tests for tenant view batching

a73d00c

Added support for view_index batching and various tenant_id types

48a792f

Refactored code and tests to support various TenantID datatypes

7cd76af

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PHOENIX-7108 Provide support for pruning expired rows of views using Phoenix level compactions #1855

PHOENIX-7108 Provide support for pruning expired rows of views using Phoenix level compactions #1855

jpisaac commented Mar 12, 2024

jpisaac commented Mar 12, 2024 •

edited

jpisaac commented Mar 12, 2024

kadirozde Mar 18, 2024

kadirozde Mar 18, 2024

jpisaac Mar 20, 2024

kadirozde Mar 20, 2024 •

edited

kadirozde Mar 18, 2024

jpisaac Mar 20, 2024

kadirozde Mar 18, 2024

jpisaac Mar 20, 2024

kadirozde Mar 29, 2024

kadirozde Apr 1, 2024

kadirozde Apr 1, 2024

kadirozde Apr 1, 2024

jpisaac commented May 25, 2024

PHOENIX-7108 Provide support for pruning expired rows of views using Phoenix level compactions #1855

Are you sure you want to change the base?

PHOENIX-7108 Provide support for pruning expired rows of views using Phoenix level compactions #1855

Conversation

jpisaac commented Mar 12, 2024

jpisaac commented Mar 12, 2024 • edited

jpisaac commented Mar 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kadirozde Mar 20, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpisaac commented May 25, 2024

jpisaac commented Mar 12, 2024 •

edited

kadirozde Mar 20, 2024 •

edited