Avoiding duplicate linked facts without infinite loops?

[douglasg14b]

The data streams I'm working with are not always pure, and often contain events that are functionally duplicates of each other.

This results in the session throwing an exception that a fact has already been added. However, for a linked fact, I cannot add a .Not() condition that enforces that the fact produced by this rule cannot refire if it has the same Id of an already existing fact (The rule can fire many time, but only for different fact output). The .Not() causes an infinite loop of fact retraction & insertion, since it's linked. If it was unlinked this would work 🤔

I was reading this & the linked thread: #234

But am still scratching my head on how to avoid duplicate facts inside the rules engine, when those facts are linked.

[snikolayev]

@douglasg14b I assume you are using Yield to produce linked facts. So, the way I understand the problem is that your rule files and Yields a fact that is already in the session, which causes an update of that fact, causing any rules that match that fact to fire multiple times.
The Yield action is actually a syntactical sugar for code that looks something like this:

If linked fact does not exist yet
then insert linked
else update linked

So, instead of using Yield action, you can use Do action and manually code deduplication logic:

if linked fact does not exist yet
then insert linked
else if linked fact is meaningfully different
then update linked
else do nothing

Use IContext methods to check if the linked fact exists and to insert/update linked.
Let me know if this helps.

[douglasg14b]

Hi! I think this'll work, I'll give it a shot.

My explanation was kind of ambiguous I realized:

Events -> Match Rule -> Rule Creates Linked Fact -> Rule has condition that it cannot create the same fact with the same Id again (.Not()) -> Condition triggers because fact exists -> fact is retracted -> rule triggers again -> fact added -> fact retracted -> fact added ...etc

[douglasg14b]

Alright, tried this, but running into a problem.

    public override void Define()
    {
        L1TwilioTaskRouterEvent queueEntered = null!;
        TaskCreatedFact taskCreated = null!;

        When()
            .Match(() => taskCreated)
            .Match(() => queueEntered, x => x.EventType == "task-queue.entered", x => x.TaskSid == taskCreated.TaskSid);

        Then()
            .Do(x => YieldINotfExists(x, new QueueEnteredFact(queueEntered)));
    }



    private void YieldINotfExists(IContext context, QueueEnteredFact newFact)
    {
        if (context.GetLinked(newFact) == null)
        {
            context.InsertLinked(newFact, newFact);
        }
    }

QueueEnteredFact is a record and as such it serves as it's own key.

The logic above results in a:

An item with the same key has already been added. Key: QueueEnteredFact { Value = RemoteExpertData.Features.Ingestion.TwilioEvents.Entities.L1TwilioTaskRouterEvent, TaskSid = [redacted], Sid = [redacted], Timestamp = [redacted], TaskQueueName = [redacted]}

If I debug it at the point of exception I also see no linked facts under context._session:

Even more curious, there is only 1 fact provided that matches EventType == "task-queue.entered" 🤔 (Though in debugging this rule gets hit twice)

How can the fact be a duplicate if context.GetLinked() returns null? I'm quite confused now 😅

[snikolayev]

It looks like there is confusion about the linked fact key - it simply identifies the linked fact within the current rule activation, in case the rule yields more than one type of fact. In your case you are only yielding one thing, so you can put anything there, or just put a string unique within the current rule, like InsertLinked("1", newFact);

A linked fact is still a fact, and so it has to be unique in the session, as determined by its equality comparison. If it's a record, it's equality comparison is just a structural value equality comparison.
In your case, is there already a fact in the session (linked or not) that would be object.Equal(..) to this one?

[douglasg14b]

It looks like there is confusion about the linked fact key - it simply identifies the linked fact within the current rule activation, in case the rule yields more than one type of fact.

Ohhh, gotcha, thanks for that explanation. I was confused when I went to the source code and it was just a string & a number 😅

---

In your case, is there already a fact in the session (linked or not) that would be object.Equal(..) to this one?

Yes, the core problem is that I may receive duplicate events from a stream. Which will cause duplicate facts to be created (Such as QueueEnteredFact), causing exceptions. I want to ensure that QueueEnteredFact is only created once per distinct "task-queue.entered" event. The events themselves are not records, and do not have GetHashCode() overridden so I tend to only notice the duplication when I go to create my own facts.

This is a problem in many other rules as well, not just this one. QueueEnteredFact doesn't actually have to be linked, but I have others that do, so for illustrative purposes we're assuming this must be a linked fact.

if linked fact does not exist yet
then insert linked
else if linked fact is meaningfully different
then update linked
else do nothing

if linked fact does not exist yet is where I appear to be stuck. I thought that was achievable with IContext.GetLinked() but as you explained that's not session level 🤔

        // First approach was to naively do this
        // Creates an infinite insert -> retract loop
        When()
            .Match(() => taskCreated)
            .Match(() => queueEntered, x => x.EventType == "task-queue.entered", x => x.TaskSid == taskCreated.TaskSid)
            .Not<QueueEnteredFact>(x => x.Sid != queueEntered.Sid);

[snikolayev]

Another option is to add a Query that collects all forward chained facts that match the same condition. The benefit of Query -> Collect is that it matches an empty collection, so does not destroy the activation as Not clause does

IEnumerable<QueueEnteredFact> prior = default;
...
.Match(() => taskCreated)
.Match(() => queueEntered, ....)
.Query(() => prior, q => q
    .Match<QueueEnteredFact(x => ....)
    .Collect())

[douglasg14b]

            .Query(() => priors, query => query // Duplicate prevention
                .Match<CustomerNotPresentFact>(x => x.TaskSid == taskCreated.TaskSid)
                .Collect());

        Filter() // Duplicate prevention
            .Where(() => !priors.Any());

This works, however it's quite in-elegant, and I'm ending up adding these to almost every low-level rule. And doesn't really work well for more complex rules.

I wonder if there is a more elegant way to do this, or a way for me to abstract this to make it less... gross?