Support for pattern matching across fields

[embano1]

What is your idea?

This has come up in several user discussions, especially around change data capture (CDC) that is often used with EventBridge Pipes and EventBridge Kafka Sink Connector, so posting here as a feature request.

Currently it is not possible to pattern match across fields in a JSON object e.g., a DynamoDB stream event (snippet) where I want to filter events that have a common (or different) value across fields, such as the value of "S" in "NewImage" and "OldImage" below.

{  
  "NewImage" : { 
    "Status" : { 
      "S" : "OK"
    } 
  },
  "OldImage": { 
    "Status": { 
      "S": "ERROR" 
    } 
  }  
}

Another (non-CDC) example:

{
  "detail": {
    "brand": "Ford",
    "model": "Focus",
    "colour": "Blue",
    "interior-colour": "Black"
  }
}

Suggested pattern:

{
  "detail": {
    "interior-colour": [ { "anything-but": "$.detail.colour" } ]
  }
}

Would you be willing to make the change?

No

Additional context

Add any other context (such as images, docs, posts) about the idea here.

[baldawar]

One follow up thought on this issue; if we were to support more payload formats (like avro or protobuf), then using $. as a indicator that we're using jsonpath would be confusing. We can work around this by following matcher like pattern. For example, [ { 'jsonpath' : '$.detail.colour' } ].

[timbray]

I agree with Rishi on syntax. I worry about implementation. If there are a bunch of these rules, the match-finding performance obviously becomes linear in their number. Right?

[embano1]

I agree with Rishi on syntax. I worry about implementation

+1

If there are a bunch of these rules, the match-finding performance obviously becomes linear in their number. Right?

That is definitely a concern, so there might be limits we have to impose around e.g., the number of such comparisons and path depth in a rule

[baldawar]

Wildcard is a good reference. Depending on how many you use, the performance degrades but there's a way to understand the worst-case complexity upfront via the evaluator. We could offer a similar solution or update the existing evaluator.

There's also some improvements we can make by trading off other dimensions. Take this rule

{
    "first": {
        "second": [{
            "jsonpath": "$.detail.colour"
        }]
    }
}

This can be recompiled to two sub-rules

{
    "$equal": [{
            "jsonpath": "$.detail.colour"
        },
        {
            "jsonpath": "$.first.second"
        }
    ]
}

and the perform computation similar to $or

event-ruler/src/main/software/amazon/event/ruler/JsonRuleCompiler.java

Line 240 in ac0b0d7

private static void parseIntoOrRelationship(final List<Map<String, List<Patterns>>> rules,

or by making both of these as terminal rules for a match

event-ruler/src/main/software/amazon/event/ruler/ACTask.java

Line 61 in 1928acc

Set<Double> terminalSubRuleIds = nameState.getTerminalSubRuleIdsForPattern(pattern);

.

There will always be some cost but if many folks have to build their own inefficient wrappers around ruler to support this behaviour, then its best to just include it in.