WIP: FEATURE: Rework CR CatchUp mechanism #4988

bwaidelich · 2024-04-11T15:56:44Z

related: #4746

WIP because:

Proper(er) mechanism to prevent multiple catchup runs (maybe integrate TASK: Prevent multiple catchup runs #4751)?
Mechanism to skip projections that are not up-to-date and don't trigger hooks on replay/individual catchup – (maybe integrate with Replace CatchUpHooks with ContentRepositoryHooks #4992)
needs to be properly coordinated with other pending PRs like TASK: Interdimensional relatives for node move #4993, WIP: FEATURE: Integrate ContentStream projection into ContentGraph #4980, FEATURE: ContentGraphAdapter for write side access #4979

as discussed in todays weekly

and introduce `neos/contentrepository-dbaltools` package

…b.com/neos/neos-development-collection into feature/4746-rework-catchup-mechanism

...and use `Doctrine\DBAL\Connection` directly

bwaidelich · 2024-04-16T13:13:28Z

Neos.ContentGraph.DoctrineDbalAdapter/Tests/Behavior/Features/Bootstrap/FeatureContext.php

@@ -44,7 +44,6 @@ public function __construct()
        self::bootstrapFlow();
        $this->contentRepositoryRegistry = $this->getObject(ContentRepositoryRegistry::class);

-        $this->setupCRTestSuiteTrait();


no longer needed, see

neos-development-collection/Neos.ContentRepository.TestSuite/Classes/Behavior/Features/Bootstrap/CRTestSuiteTrait.php

Line 88 in 9ee2fc8

protected function setupCRTestSuiteTrait(): void

..to avoid status from showing a diff everytime

...for now

bwaidelich · 2024-04-19T13:18:32Z

Neos.ContentRepository.Core/Classes/ContentRepository.php

+    {
+        $store = new SemaphoreStore();
+        $factory = new LockFactory($store);
+        $lock = $factory->createLock('catchup');


TODO: This lock needs to be CR specific (i.e. include $this->id)

bwaidelich · 2024-04-19T13:20:44Z

Neos.ContentRepository.Core/Classes/ContentRepository.php

@@ -153,6 +159,67 @@ public function projectionState(string $projectionStateClassName): ProjectionSta
        throw new \InvalidArgumentException(sprintf('A projection state of type "%s" is not registered in this content repository instance.', $projectionStateClassName), 1662033650);
    }

+    public function catchUpProjections(): void
+    {
+        $store = new SemaphoreStore();


This has to become an external dependency and/or coordinated with #4751

nezaniel · 2024-04-21T09:48:01Z

This already looks very promising and I like the direction this is taking!
What part is still missing / incomplete? Just the new catchup implementation, right?

bwaidelich · 2024-04-21T10:38:34Z

This already looks very promising and I like the direction this is taking!

Thanks for the feedback, appreciated!

What part is still missing / incomplete?

I added some more lines to the task list above

mhsdesign

Thats a lot of stuff but looks "good" so far (considering this is WIP) ... i went through all php classes by reading except the last 20 with only the +-1 changes

mhsdesign · 2024-04-22T19:14:37Z

...pository.BehavioralTests/Tests/Functional/Feature/WorkspacePublication/setup-is-running-flag

why do we need this file now? ^^

mhsdesign · 2024-04-22T19:15:27Z

Neos.ContentRepository.Core/Classes/ContentRepository.php

+        #print_r(debug_backtrace(DEBUG_BACKTRACE_IGNORE_ARGS));exit;
+        #\Neos\Flow\var_dump('HANDLE ' . $command::class);


mhsdesign · 2024-04-22T19:16:48Z

Neos.ContentRepository.Core/Classes/ContentRepository.php

+        return new class {
+            /**
+             * @deprecated backwards compatibility layer
+             */
+            public function block(): void
+            {
+            }
+        };


if we were to merge this like this - which i think is not necessary - we should use an actual class so its inspectable via phpstorm :) But i dont think we need b/c layer for new code

mhsdesign · 2024-04-22T19:17:57Z

Neos.ContentRepository.Core/Classes/ContentRepository.php

            $catchUpHook?->onBeforeEvent($event, $eventEnvelope);
            $projection->apply($event, $eventEnvelope);
+            // TODO this should happen in the inner transaction


todo over here :)

mhsdesign · 2024-04-22T19:18:20Z

Neos.ContentRepository.Core/Classes/ContentRepository.php

+            ];
+            $projectionsAndCatchUpHooks[$projectionClassName]['catchUpHook']?->onBeforeCatchUp();
+        }
+        #\Neos\Flow\var_dump('CATCHUP from ' . $lowestAppliedSequenceNumber->value);


.. lots of debug code and i guess todos?

mhsdesign · 2024-04-22T19:19:32Z

Neos.ContentRepository.Core/Classes/Infrastructure/DbalClientInterface.php

what is the idea behind the removal of this?

mhsdesign · 2024-04-22T19:20:01Z

Neos.ContentRepository.Core/Classes/Infrastructure/DbalCheckpointStorage.php

i guess this is now part of CheckpointHelper? Good think we have it in the Core and now our lower level package ^^

See #5005

in order to prevent nested transactions potentially leading to RuntimeException of A transaction is active already, can't commit events!

github-actions bot added the 9.0 label Apr 11, 2024

WIP: FEATURE: Rework CR CatchUp mechanism

3a2fc46

bwaidelich force-pushed the feature/4746-rework-catchup-mechanism branch from 1138434 to 3a2fc46 Compare April 11, 2024 15:57

bwaidelich added 8 commits April 12, 2024 18:42

Remove react/async dependency

99cd1e0

as discussed in todays weekly

Remove react/async dependency 2/2

e8a0bdd

Cosmetic fix to satisfy linter

d2382ee

Remove unused namespace imports

7dc35fc

Replace CheckpointStorageInterface

13e1238

and introduce `neos/contentrepository-dbaltools` package

Replace CheckpointStorageInterface

4c425cd

and introduce `neos/contentrepository-dbaltools` package

Merge branch 'feature/4746-rework-catchup-mechanism' of https://githu…

b1d28f6

…b.com/neos/neos-development-collection into feature/4746-rework-catchup-mechanism

Remove DbalClientInterface and implementations

ad57a91

...and use `Doctrine\DBAL\Connection` directly

bwaidelich commented Apr 16, 2024

View reviewed changes

bwaidelich added 7 commits April 16, 2024 15:16

Update composer manifest

b817b1d

Tweak checkpoint error message

cb653fd

Re-add neos/eventstore-doctrineadapter dependency

43da4c2

Mostly cosmetic tweaks to satisfy linter

bb1c498

Make checkpoint table check a platform specific check

e62cb87

..to avoid status from showing a diff everytime

Remove notion of "sync" from composer scripts and readme

7cc13ed

Remove ProjectionCatchUpTriggerInterface and implementations

fc6ec5c

...for now

bwaidelich force-pushed the feature/4746-rework-catchup-mechanism branch from 192256e to fc6ec5c Compare April 17, 2024 15:33

bwaidelich mentioned this pull request Apr 18, 2024

WIP: Replace CatchUpHooks with ContentRepositoryHooks #4995

Draft

1 task

bwaidelich commented Apr 19, 2024

View reviewed changes

bwaidelich mentioned this pull request Apr 22, 2024

BUGFIX: Use dedicated connection for EventStore #5005

Draft

mhsdesign reviewed Apr 22, 2024

View reviewed changes

bwaidelich added 3 commits April 24, 2024 17:37

Merge branch '9.0' into feature/4746-rework-catchup-mechanism

89eada5

Merge branch '9.0' into feature/4746-rework-catchup-mechanism

cc3a191

Merge branch '9.0' into feature/4746-rework-catchup-mechanism

7c8ae2c

bwaidelich added 6 commits April 26, 2024 17:15

Fix ContentStreamProjection checkpoint

ed47eb6

Merge branch '9.0' into feature/4746-rework-catchup-mechanism

592e78d

Merge branch '9.0' into feature/4746-rework-catchup-mechanism

4af1e3c

BUGFIX: Remove doctrine migrate and cr setup commands from behat scripts

4c68ca7

See #5005

Use dedicated connection for EventStore

7bfd941

in order to prevent nested transactions potentially leading to RuntimeException of A transaction is active already, can't commit events!

Re-use previously created event store instances

e2e8af6

bwaidelich mentioned this pull request May 6, 2024

WIP: Rework CatchUp and CatchUp hooks #5027

Closed

Merge branch '9.0' into feature/4746-rework-catchup-mechanism

fb9b897

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: FEATURE: Rework CR CatchUp mechanism #4988

WIP: FEATURE: Rework CR CatchUp mechanism #4988

bwaidelich commented Apr 11, 2024 •

edited

bwaidelich Apr 16, 2024

bwaidelich Apr 19, 2024

bwaidelich Apr 19, 2024

nezaniel commented Apr 21, 2024

bwaidelich commented Apr 21, 2024

mhsdesign left a comment •

edited

mhsdesign Apr 22, 2024

mhsdesign Apr 22, 2024

mhsdesign Apr 22, 2024

mhsdesign Apr 22, 2024

mhsdesign Apr 22, 2024

mhsdesign Apr 22, 2024

mhsdesign Apr 22, 2024

		#print_r(debug_backtrace(DEBUG_BACKTRACE_IGNORE_ARGS));exit;
		#\Neos\Flow\var_dump('HANDLE ' . $command::class);

WIP: FEATURE: Rework CR CatchUp mechanism #4988

Are you sure you want to change the base?

WIP: FEATURE: Rework CR CatchUp mechanism #4988

Conversation

bwaidelich commented Apr 11, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nezaniel commented Apr 21, 2024

bwaidelich commented Apr 21, 2024

mhsdesign left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwaidelich commented Apr 11, 2024 •

edited

mhsdesign left a comment •

edited