refactor(cc-tx-visualization): capture transactional data with RxJS #3100 #3208

brunoffmateus · 2024-04-15T18:46:47Z

The plugin now utilizes RxJS instead of RabbitMQ in transaction monitoring.
ReplaySubjects store and emit observed transactional data to subscribers.

RafaelAPB · 2024-04-16T15:32:26Z

Please rebase with the latest version of satp-dev

petermetz · 2024-04-16T16:05:03Z

packages/cactus-plugin-cc-tx-visualization/src/main/typescript/plugin-cc-tx-visualization.ts

+  fabricTxObservable: Observable<RunTxReqWithTxId>;
+}
+
+export class CcTxVisualization implements ICactusPlugin, IPluginWebService {


@brunoffmateus I'm looking at the CcTxVisualization class' source code but cannot see any mention of RxJS in the imports nor the class implementation. Could you please let me know where to look?

There is an import { Observable } from "rxjs"; in line 56.
Then besuTxObservable and fabricTxObservable are used in the constructor (lines 104/105) and in the methods watchRunTransactionV1Exchange (line 254) and watchRunTxReqWithTxId (line 269).

RafaelAPB · 2024-04-24T17:57:55Z

@brunoffmateus please don't forget to add me as a maintainer for this package (see https://github.com/brunoffmateus/blockchain-integration-framework/blob/satp-3100-cctx/CODEOWNERS).

Please also rename the plugin to cactus-plugin-ccmodel-hepheastus

Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com> Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cc-tx-viz): add methods to fabric connector Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com> fix(prometheus): metrics.ts leaks to global registry hyperledger#1202 1. Specified a `register` property of the gauges as an empty array so that it does not pollute the global namespace. This is how this is supposed to be done as per the docs of prom-client. 2. Once the change from 1) took place, the issue became that the metrics gathering code was still trying to hit up the global scope for the metrics, e.g. calling the get metrics methods directly on the promClient object instead of the registry that we create for each prmoetheus exporter object separately. So a little additional refactor ensued to fix this as well by making sure that we grab a reference of the registry object at construction time and then re-use that wherever needed instead of going through the global promClient object. 3. Added missing .startMetricsCollection calls in the plugin constructors to ensure that the prometheus exporter object gets initialized properly (this is where the registry gets created as well so without this there are crashes happening when one tries to access the metrics through the registry) Why though? The problem was that the metrics.ts file that we have for all the plugin's metrics constructs a new Metric (Gauge) object at import time which then defaults to registering the metric in the global registry of prom-client by default. The latter was causing crashes when separate versions of the same metrics.ts file are imported in a scenario were the API server imports plugins from a different directory (this issue is coming from the branch where I'm working on plugin sandboxing via the live-plugin-manager). Fixes hyperledger#1202 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com> feat(cc-tx-viz): added test for multiple prometheus metrics on fabric Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com> feat(cc-tx-viz): add metric model Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com> feat(cc-tx-viz): added croschainEventLog models Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com> feat(fabric): added endpoint for list of transaction receipts Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com> feat(test-tooling): add RabbitMQ test server feat(cc-tx-viz): add RabbitMQ support; update model feat(cc-tx-viz): add preliminary support to Fabric and Besu build(deps): upgrade axios to latest to fix CVE Details CVE-2021-3749 high severity Vulnerable versions: <= 0.21.1 Patched version: 0.21.2 axios is vulnerable to Inefficient Regular Expression Complexity Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com> feat(cc-tx-viz): add methods to fabric connector Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com> feat(cc-tx-viz): add metric model Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com> xfix(cc-tx-viz): error implementation of IsVisualizable feat(cc-tx-viz): add methods to fabric and besu; update models Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com> feat(cc-tx-viz): update cross chain events log for fabric transactions Signed-off-by: Iulia Mihaiu <maramih.2007@yahoo.com> feat(cctxviz): fix fabric prometheus feat(cctxviz): fix besu prometheus feat(cctxviz): fix fabric test feat(cctxviz): update tx receipt feat(cctxviz): add test script Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): add test receipts feat(cctxviz): refactor CrossChainEventLog Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> test(cctxviz): refactor tests feat(cc-tx-viz): update model, add tests chore(cc-tx-viz): update dependencies Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cc-tx-viz): persist logs Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> test(cc-tx-viz): update test Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cc-tx-viz): update strinfify to csv Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> chore(cc-tx-viz): add csv folder Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> chore(cc-tx-viz): move csv folder Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cc-tx-viz): add test feat(cctxviz): add csv file, process mining script Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): proof of concept v1 Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> chore(cctxviz): update csv Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> docs(cctxviz): update readme Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): update tests Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): add carbon footprint Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): update tests Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): update model Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(fabric-connector): update default queue Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> test(cctxviz): fabric test Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): add parameters to fabric calls Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): support besu Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): update metrics Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> test(cctxvi): update fabric and besu tests Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> test(cctxviz): besu and fabric Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): support call transactions on fabric Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): add aggregation basis Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): aggregate cctx Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> chore: update yarn and remove package.lock Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): containerize draft Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): add revenue Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): update model playbook Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): add dummy use case playbook Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> test(cctxviz): update persist test and dummy uc Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): add timer Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): workload test Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): add more timers Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): add timer to map on connectors Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): update testing framework Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): vis plot wip Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): evaluate baseline Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): update plots Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> test(cctxviz): add test files Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> feat(cctxviz): update testing framework Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> test(cctxviz): time-efficient polling receipts Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> chore(cctxviz): add test chaincode Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> test(cctxviz): update test Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> test(cctxviz): use case v1 Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> test(cctxviz): update test framework Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> build(cc-tx-visualization): added missing fabric-contract-api This will fix one compilation error. Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com> chore(deps): updated yarn.lock after rebase onto upstream/main Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com> build(cc-tx-visualization): deleted redundant test fixtures Also added the test-tooling package as a dev dependency which should fix a few of the compiler errors we were getting on account of missing dependencies. Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com> fix: resolved build errors for cctxviz branch - test-tooling package changed that needs to be reverted before commmit to main branch - added import statement for test data generationt xfix(cc-tx-viz): error implementation of IsVisualizable chore: update yarn and remove package.lock Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> test(cctxviz): testing framework Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> test(cctxviz): test framework Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> test(cctxviz): update framework Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> test(cctxviz): add graphs Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> test(cctxviz): add test Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> fix(cctxviz): fix bug Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> test(cctxviz): add process conformance Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt>

The plugin now utilizes RxJS instead of RabbitMQ in transaction monitoring. ReplaySubjects store and emit observed transactional data to subscribers. fix(cc-tx-visualization): rebase latest version Signed-off-by: Rafael Belchior <rafael.belchior@tecnico.ulisboa.pt> squash! - migrate a test case to Fabric v2.5.6 LTS AIO This is just a snippet of a change from the pair programming call with Bruno, it can be safely squashed, no worries. Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com> Signed-off-by: Bruno Mateus <brumat315@gmail.com>

RafaelAPB · 2024-04-25T07:33:13Z

@brunoffmateus Looks much better. Please rename the plugin too

RafaelAPB

Generally looks good, but there are a few concerns that need to be addressed:

Code does not compile
Tests need to be refactored to use Jest
Test connectors transaction polling via Observables (instead of using mocks).

RafaelAPB · 2024-04-25T10:17:27Z

.cspell.json

@@ -33,6 +33,8 @@
        "cccs",
        "ccep",
        "ccid",
+        "cctx",
+        "cctxviz",


we will remove this as we will rename the plugin

my bad, missed 3. good job

RafaelAPB · 2024-04-25T10:17:45Z

CODEOWNERS

@@ -1,5 +1,6 @@
 * @petermetz @takeutak @izuru0 @jagpreetsinghsasan @vramakrishna @sandeepnRES @outSH

+packages/cactus-plugin-cc-tx-visualization @RafaelAPB


rename plugin

RafaelAPB · 2024-04-25T10:18:04Z

jest.config.js

@@ -121,5 +121,9 @@ module.exports = {
    `./examples/cactus-example-carbon-accounting-backend/src/test/typescript/integration/admin-enroll-v1-endpoint.test.ts`,
    `./examples/cactus-example-supply-chain-backend/src/test/typescript/integration/supply-chain-backend-api-calls.test.ts`,
    `./examples/cactus-example-supply-chain-backend/src/test/typescript/integration/supply-chain-cli-via-npm-script.test.ts`,
+    `./packages/cactus-plugin-cc-tx-visualization/src/test/typescript/integration/cctxviz-generate-use-case-dummy-baseline-events.test.ts`,


please re-implement the tests to use Jest, and remove this

RafaelAPB · 2024-04-25T10:18:11Z

tsconfig.json

@@ -130,6 +130,9 @@
    {
      "path": "./packages/cactus-test-verifier-client/tsconfig.json"
    },
+    {
+      "path": "./packages/cactus-plugin-cc-tx-visualization/tsconfig.json"


don't forget rename

RafaelAPB · 2024-04-25T10:23:43Z

packages/cactus-plugin-cc-tx-visualization/src/main/typescript/models/carbon-footprint.ts

+  return 30;
+}


please try to obtain this dynamically for EVM chains

if fabric or besu, return default

RafaelAPB · 2024-04-25T10:27:10Z

packages/cactus-plugin-cc-tx-visualization/src/main/typescript/plugin-cc-tx-visualization.ts

+  public async saveModel(
+    modelType: CrossChainModelType,
+    model: string,
+  ): Promise<void> {
+    this.crossChainModel.saveModel(modelType, model);
+  }


we should also have methods to:

read a serialized CrossChainModelType

read the output from pm4py

convert pm4py into CrossChainModelType, and vice versa

I recommend you to create a new directory, something like pm4py-adapter, where you put the code that handles interactions between this plugin and pm4py

RafaelAPB · 2024-04-25T10:28:33Z

packages/cactus-plugin-cc-tx-visualization/src/test/csv/dummy-use-case-6-events.csv

+1;txID2;2024-04-15T18:11:42.598Z;FABRIC_2;FabricContractInvocationType.SEND;LockAsset;"[""asset1""]";A;0;0.00018;39;0
+1;txID3;2024-04-15T18:11:42.599Z;FABRIC_2;FabricContractInvocationType.SEND;CreateAsset;"[""1"",""100""]";A;0;0.00018;60;0
+1;txID4;2024-04-15T18:11:42.600Z;FABRIC_2;FabricContractInvocationType.SEND;TransferAsset;"[""asset1"",""targetAccount""]";A;0;0.00018;61;0
+1;txID5;2024-04-15T18:11:42.601Z;FABRIC_2;FabricContractInvocationType.SEND;TransferAsset;"[""asset1"",""targetAccount2""]";A;0;0.00018;61;0


have these tests all start with example-dummy. don't forget to change the file name on the csv creation logic in the tests

RafaelAPB · 2024-04-25T10:28:57Z

packages/cactus-plugin-cc-tx-visualization/src/test/json/dummy-use-case-6-events.json

@@ -0,0 +1,102 @@
+[
+  {


see CSV comment

RafaelAPB · 2024-04-25T10:29:27Z

...tion/src/test/typescript/integration/cctxviz-generate-use-case-dummy-baseline-events.test.ts

@@ -0,0 +1,241 @@
+import test, { Test } from "tape-promise/tape";


refactor to use Jest; refactor to use new functionality

RafaelAPB · 2024-04-25T10:30:31Z

...tion/src/test/typescript/integration/cctxviz-generate-use-case-dummy-baseline-events.test.ts

+import {
+  FabricContractInvocationType,
+  RunTxReqWithTxId,
+} from "@hyperledger/cactus-plugin-ledger-connector-fabric";


Where is a connector being initialized? we need to test with real connectors, not mocks. Is the observable functionality enabled in the respective connector?

brunoffmateus requested review from petermetz, takeutak, izuru0, jagpreetsinghsasan, VRamakrishna, sandeepnRES and outSH as code owners April 15, 2024 18:46

RafaelAPB force-pushed the satp-dev branch from 0ba4a01 to c7497b5 Compare April 16, 2024 14:12

brunoffmateus requested a review from RafaelAPB as a code owner April 16, 2024 14:12

brunoffmateus force-pushed the satp-3100-cctx branch from 0cf79a6 to 4bd50b7 Compare April 16, 2024 15:50

petermetz reviewed Apr 16, 2024

View reviewed changes

RafaelAPB force-pushed the satp-dev branch from 72d63ab to db916c5 Compare April 23, 2024 10:16

RafaelAPB requested a review from petermetz April 24, 2024 17:48

maramih and others added 2 commits April 24, 2024 19:06

brunoffmateus force-pushed the satp-3100-cctx branch from 4bd50b7 to a2e32f7 Compare April 24, 2024 18:08

RafaelAPB requested changes Apr 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(cc-tx-visualization): capture transactional data with RxJS #3100 #3208

refactor(cc-tx-visualization): capture transactional data with RxJS #3100 #3208

brunoffmateus commented Apr 15, 2024

RafaelAPB commented Apr 16, 2024

petermetz Apr 16, 2024

brunoffmateus Apr 17, 2024

RafaelAPB commented Apr 24, 2024

RafaelAPB commented Apr 25, 2024

RafaelAPB left a comment

RafaelAPB Apr 25, 2024

RafaelAPB Apr 26, 2024

RafaelAPB Apr 25, 2024

RafaelAPB Apr 25, 2024

RafaelAPB Apr 25, 2024

RafaelAPB Apr 25, 2024

RafaelAPB Apr 25, 2024

RafaelAPB Apr 25, 2024

RafaelAPB Apr 25, 2024

RafaelAPB Apr 25, 2024

RafaelAPB Apr 25, 2024

RafaelAPB Apr 25, 2024

RafaelAPB Apr 25, 2024

		@@ -1,5 +1,6 @@
		* @petermetz @takeutak @izuru0 @jagpreetsinghsasan @vramakrishna @sandeepnRES @outSH

		packages/cactus-plugin-cc-tx-visualization @RafaelAPB

		@@ -0,0 +1,241 @@
		import test, { Test } from "tape-promise/tape";

refactor(cc-tx-visualization): capture transactional data with RxJS #3100 #3208

Are you sure you want to change the base?

refactor(cc-tx-visualization): capture transactional data with RxJS #3100 #3208

Conversation

brunoffmateus commented Apr 15, 2024

RafaelAPB commented Apr 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RafaelAPB commented Apr 24, 2024

RafaelAPB commented Apr 25, 2024

RafaelAPB left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment