Wip/wan tx grouping module 1 #7083

albertogpz · 2021-11-05T08:42:11Z

For all changes:

Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?
Has your PR been rebased against the latest commit within the target branch (typically develop)?
Is your initial contribution a single, squashed commit?
Does gradlew build run cleanly?
Have you written or updated unit tests to verify your changes?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

kirklund

Inlined a few comments. Overall looks like a good direction! Here are some general comments and tips:

Prefer interfaces over abstract methods.
Standardize on one library instead of mixing them (such as AssertJ for assertions).
Is there a better word than "txgrouping" for module name and packages?
Declare variables and APIs with interfaces rather than implementations.
Always look for an interface to use instead of an implementation (such as InternalCache instead of GemFireCacheImpl) in both tests and product code.
Always minimize the scope of anything (class, method, variable, constant).

kirklund · 2021-11-29T21:48:24Z

...ributedTest/java/org/apache/geode/internal/cache/wan/txgrouping/TxGroupingBaseDUnitTest.java

+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;


Please use AssertJ for all assertions. The assertEquals can just use assertThat(a).isEqualTo(b) and assertNotNull becomes assertThat(a).isNotNull();

kirklund · 2021-11-29T21:49:06Z

...ributedTest/java/org/apache/geode/internal/cache/wan/txgrouping/TxGroupingBaseDUnitTest.java

+
+  protected Properties createLocatorConfig(int systemId, int locatorPort, int remoteLocatorPort) {
+    Properties config = new Properties();
+    config.setProperty(MCAST_PORT, "0");


This is the default so you can delete the line with MCAST_PORT

kirklund · 2021-11-29T21:50:27Z

...ributedTest/java/org/apache/geode/internal/cache/wan/txgrouping/TxGroupingBaseDUnitTest.java

+    if (regionSize != r.keySet().size()) {
+      await().untilAsserted(() -> assertThat(r.keySet().size()).isEqualTo(regionSize));
+    }


I would delete the if-statement and just use the await. The if statement is redundant... the await will also immediately complete successfully if the if-statement is true.

kirklund · 2021-11-29T21:51:17Z

...ributedTest/java/org/apache/geode/internal/cache/wan/txgrouping/TxGroupingBaseDUnitTest.java

+      await()
+          .untilAsserted(() -> assertThat(regionQueue.size()).isEqualTo(expectedQueueSize));
+    }
+    ArrayList<Integer> stats = new ArrayList<>();


Remember to declare as the weakest interface you need:

List<Integer> stats = new ArrayList<>();

kirklund · 2021-11-29T21:52:54Z

...ributedTest/java/org/apache/geode/internal/cache/wan/txgrouping/TxGroupingBaseDUnitTest.java

+    assertThat(creates).isEqualTo(gatewayReceiverStats.getCreateRequest());
+  }
+
+  protected void doTxPutsWithRetryIfError(String regionName, final long putsPerTransaction,


I think you should try to simplify any new method with this many calculations, for-loops, do-whiles and try-blocks. If you really need all of those blocks then try extracting the nested blocks to their own short little methods.

kirklund · 2021-11-29T21:56:08Z

...ava/org/apache/geode/internal/cache/wan/txgrouping/parallel/TxGroupingParallelDUnitTest.java

+    try {
+      for (AsyncInvocation<?> asyncInvocation : asyncInvocations) {
+        asyncInvocation.await();
+      }
+    } catch (InterruptedException e) {
+      fail("Interrupted");
+    }


Don't catch an exception and use fail. Instead just add throws InterruptedException to the test and delete the try-catch lines.

kirklund · 2021-11-29T22:00:51Z

...ava/org/apache/geode/internal/cache/wan/txgrouping/parallel/TxGroupingParallelDUnitTest.java

+    } finally {
+      exp.remove();
+      exp1.remove();
+    }


IgnoredException implements AutoCloseable so you can use it in try-with-resources syntax. Using import-static and addIgnoredException(Class) tidies it up nicely:

import static org.apache.geode.test.dunit.IgnoredException.addIgnoredException; try (IgnoredException ie1 = addIgnoredException(RegionDestroyedException.class); IgnoredException ie2 = addIgnoredException(ForceReattemptException.class)) { // ... }

kirklund · 2021-11-29T22:03:40Z

...st/java/org/apache/geode/internal/cache/wan/txgrouping/serial/TxGroupingSerialDUnitTest.java

+        + statistics.getUnprocessedEventsRemovedByPrimary())).isEqualTo(eventsReceived);
+  }
+
+  private Boolean killPrimarySender(String senderId) {


The dunit testing framework originally required primitive wrappers, so I think you may have copied some of this from elsewhere. It's better to just use simple primitives for return types and parameters now, such as boolean instead of Boolean.

kirklund · 2021-11-29T22:06:35Z

...ava/org/apache/geode/cache/wan/internal/txgrouping/CommonTxGroupingGatewaySenderFactory.java

+import org.apache.geode.internal.cache.wan.GatewaySenderException;
+import org.apache.geode.internal.cache.wan.MutableGatewaySenderAttributes;
+
+public abstract class CommonTxGroupingGatewaySenderFactory {


I think you should make this an interface instead of an abstract class. The static validator method would then become a default implementation which would probably only ever be overridden by a test.

What would be the benefit of using an interface over an abstract class here?
Given that it provides helper static methods I tend to think it is better to define it is a class but I have no strong argument in favor of it.

kirklund · 2021-11-29T22:07:38Z

...pache/geode/cache/wan/internal/txgrouping/parallel/TxGroupingParallelGatewaySenderQueue.java

+    while (true) {
+      for (Iterator<Map.Entry<TransactionId, Integer>> iter =
+          incompleteTransactionIdsInBatch.entrySet().iterator(); iter.hasNext();) {
+        Map.Entry<TransactionId, Integer> pendingTransaction = iter.next();
+        TransactionId transactionId = pendingTransaction.getKey();
+        int bucketId = pendingTransaction.getValue();
+        List<Object> events =
+            peekEventsWithTransactionId(partitionedRegion, bucketId, transactionId);
+        for (Object object : events) {


Try to extract any nested loops to their own smaller methods. It helps keep things better organized and more readable.

jake-at-work · 2021-11-30T19:23:42Z

...rg/apache/geode/cache/wan/internal/txgrouping/serial/TxGroupingSerialGatewaySenderQueue.java

+  }
+
+  @Override
+  public synchronized void remove() throws CacheException {


At first glance it looks like a lot of the remove method content is reimplemented here. Is there a way break up the original with some extension points to deduce the duplication?

See Nordix#17

jake-at-work · 2021-11-30T19:27:40Z

geode-wan/src/main/java/org/apache/geode/cache/wan/internal/GatewaySenderFactoryImpl.java

@@ -256,7 +257,13 @@ public GatewaySenderFactory setEnforceThreadsConnectSameReceiver(

  static void validate(final @NotNull InternalCache cache,
      final @NotNull GatewaySenderAttributesImpl attributes) {
-    final int myDSId = cache.getDistributionManager().getDistributedSystemId();
+    int myDSId;
+    if (cache instanceof CacheCreation) {


Maybe CacheCreation should be fixed to behave like any other InternalCache rather than making consumer know subtile differences in behavior.

You mean to implement in CacheCreation the getDistributionManager() method as follows?

@Override public DistributionManager getDistributionManager() { return InternalDistributedSystem.getAnyInstance().getDistributionManager() }

jake-at-work · 2021-11-30T19:30:40Z

...geode/internal/cache/wan/txgrouping/cli/commands/CreateTxGroupingGatewaySenderDUnitTest.java

+    addIgnoredException("could not get remote locator");
+
+    String createCommandString =
+        "create gateway-sender --id=sender1 --remote-distributed-system-id=1 --parallel --group-transaction-events=true";


This doesn't seem modular yet. Is this testing deprecated --group-transaction-events and work is pending to add at --type=TxGroupingParallelGatewaySender?

You can find an example of modularizing this in the first POC. Follow the new type property on the create gateway-sender command.
https://github.com/pivotal-jbarrett/geode/blob/wip/wan-spi/geode-gfsh/src/main/java/org/apache/geode/management/internal/cli/commands/CreateGatewaySenderCommand.java#L83

Done. We'll have to see how we reflect this in the documentation as we will need documentation specific for the module.

jake-at-work · 2021-11-30T20:42:41Z

@albertogpz Can you please rebase this from develop. There as a fixed added there to address issues with Gradle builds.

jake-at-work · 2021-12-01T00:39:04Z

@albertogpz we need to extract out the TX batching specific elements form the GatewaySenderEventImpl object. I replaced it with a implementation defined metadata object. See TxBatchMetadata from my previous example on making this a module. This gets set on the event object when constructed via an override on AbstractGatewaySenderEventProcessor. See example here.

albertogpz · 2021-12-02T05:11:57Z

Inlined a few comments. Overall looks like a good direction! Here are some general comments and tips:

1. Prefer interfaces over abstract methods.

2. Standardize on one library instead of mixing them (such as AssertJ for assertions).

3. Is there a better word than "txgrouping" for module name and packages?

4. Declare variables and APIs with interfaces rather than implementations.

5. Always look for an interface to use instead of an implementation (such as InternalCache instead of GemFireCacheImpl) in both tests and product code.

6. Always minimize the scope of anything (class, method, variable, constant).

Thanks for your comments.
Regarding the name, I do not have one that I really like. Other alternatives could be: txeventsgrouping, grouptxevents, txaware,...
If you like any of these or have any other proposal, please, let me know.

albertogpz · 2021-12-02T15:43:08Z

@albertogpz we need to extract out the TX batching specific elements form the GatewaySenderEventImpl object. I replaced it with a implementation defined metadata object. See TxBatchMetadata from my previous example on making this a module. This gets set on the event object when constructed via an override on AbstractGatewaySenderEventProcessor. See example here.

@pivotal-jbarrett As I said in a previous e-mail, I do not think we can make these changes without breaking backward compatibility. The feature was production ready at 1.14 so we can expect that it is used by customers. Therefore, we cannot just ignore those changes when interworking with 1.14 servers.
Am I missing something?

…t's review

onichols-pivotal · 2022-02-05T08:02:13Z

this PR appears to be abandoned, can it be closed?

albertogpz · 2022-02-07T08:21:15Z

this PR appears to be abandoned, can it be closed?

It is not abandoned. Still waiting for comments after review changes. I have rebased the branch to remove the conflicts with develop.

albertogpz force-pushed the wip/wan-tx-grouping-module_1 branch from 1407b2a to 214dc76 Compare November 18, 2021 17:37

albertogpz requested review from jake-at-work and kirklund November 25, 2021 07:10

kirklund reviewed Nov 29, 2021

View reviewed changes

jake-at-work reviewed Nov 30, 2021

View reviewed changes

jake-at-work requested changes Nov 30, 2021

View reviewed changes

albertogpz force-pushed the wip/wan-tx-grouping-module_1 branch 3 times, most recently from 1850646 to ee9b459 Compare December 2, 2021 15:16

albertogpz force-pushed the wip/wan-tx-grouping-module_1 branch from f238c66 to d6d80b2 Compare December 14, 2021 10:32

albertogpz requested review from kirklund and jake-at-work December 14, 2021 11:12

albertogpz force-pushed the wip/wan-tx-grouping-module_1 branch 5 times, most recently from 520d955 to 1ac52a6 Compare December 15, 2021 14:48

jake-at-work added 8 commits February 3, 2022 16:09

GEODE-6588: Cleanup GatewaySenderAttributes

042013d

GEODE-6588: Cleanup AbstractGatewaySender

2457963

GEODE-6588: Cleanup GatewaySenderFactoryImpl

8c2d23e

MODULE: Extract interface for GatewaySenderAttributes.

ae74a42

MODULE: Split out some factories.

7a3d45c

MODULE: Extract out TX grouping factories.

35b072e

MODULE: Create TX grouping sender impls.

f7c0d21

MODULE: Remove AbstractGatwaySender.mustGroupTransactionEvents()

477a823

jake-at-work and others added 11 commits February 4, 2022 13:46

MODULE: Stupid simple module.

0be467b

MODULE: ServiceLoader

07884e8

MODULE: Move tests

5aa6bc7

MODULE: Move remaining tx-grouping functionality and DUnit tests

1959233

MODULE: Fix some GatewaySenderFactoryImpl tests

74f9274

MODULE: Do not allow changing group-transaction-events

a7416a3

MODULE: Add gfsh create tx-grouping gw-sender cases.

0c3d814

Updated with Kirk's first review comments

43194d0

remove() refactored as suggested on jbarrett's review

cbe859d

Use --type=TxGroupingParallelGatewaySender... as suggested in jbarret…

8c2f77d

…t's review

Added exceptions to public API changes

77db403

albertogpz force-pushed the wip/wan-tx-grouping-module_1 branch from 1ac52a6 to dd7f827 Compare February 7, 2022 08:20

albertogpz force-pushed the wip/wan-tx-grouping-module_1 branch 3 times, most recently from 7a03267 to 13ea059 Compare February 8, 2022 09:44

Fix tests

4ce6664

albertogpz force-pushed the wip/wan-tx-grouping-module_1 branch from 13ea059 to 4ce6664 Compare February 8, 2022 15:37

kirklund removed their request for review April 4, 2022 16:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wip/wan tx grouping module 1 #7083

Wip/wan tx grouping module 1 #7083

albertogpz commented Nov 5, 2021

kirklund left a comment

kirklund Nov 29, 2021

kirklund Nov 29, 2021

kirklund Nov 29, 2021

kirklund Nov 29, 2021

kirklund Nov 29, 2021

kirklund Nov 29, 2021

kirklund Nov 29, 2021

kirklund Nov 29, 2021

kirklund Nov 29, 2021

albertogpz Dec 1, 2021 •

edited

kirklund Nov 29, 2021

jake-at-work Nov 30, 2021

jake-at-work Nov 30, 2021

albertogpz Dec 2, 2021

jake-at-work Nov 30, 2021

albertogpz Dec 2, 2021

jake-at-work Nov 30, 2021

jake-at-work Dec 1, 2021

albertogpz Dec 14, 2021

jake-at-work commented Nov 30, 2021

jake-at-work commented Dec 1, 2021 •

edited

albertogpz commented Dec 2, 2021

albertogpz commented Dec 2, 2021 •

edited

onichols-pivotal commented Feb 5, 2022

albertogpz commented Feb 7, 2022

Wip/wan tx grouping module 1 #7083

Are you sure you want to change the base?

Wip/wan tx grouping module 1 #7083

Conversation

albertogpz commented Nov 5, 2021

For all changes:

kirklund left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

albertogpz Dec 1, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jake-at-work commented Nov 30, 2021

jake-at-work commented Dec 1, 2021 • edited

albertogpz commented Dec 2, 2021

albertogpz commented Dec 2, 2021 • edited

onichols-pivotal commented Feb 5, 2022

albertogpz commented Feb 7, 2022

albertogpz Dec 1, 2021 •

edited

jake-at-work commented Dec 1, 2021 •

edited

albertogpz commented Dec 2, 2021 •

edited