RFC: Rule system v2

[OliverSkroblin]

In this RFC, I would like to discuss how the Rule System should evolve in the coming years.

1. First, I would like to list the problems with the current system that I have encountered in recent years.
2. Then, I will outline what, in my opinion, could be a possible path for the development of the Rule System.
3. Finally, I will describe the impact that these changes would have on the system.

Current issues

• We need to load the cart in each request to evaluate the rules
• We need to load all rules in each request to calculate the cart
• ... This leads to a chicken-egg issue where you can create rules circles which are not valid and would lead to an endless loop
• The system was designed to be based on a storefront session, but adapted to admin sessions too
  • Many issues in the past where we have to restore orders or to request more data from database
• Price rules are heavy to calculate and extrem Inefficient and slow
  • To much rules leads to queries with over hundret IF-ELSE conditions
• Multiple cart calculation are necessary to “nearly” calculate the correct cart

Possible solution

Current rule system will be splitted into three areas content, checkout, flow:

• Content rules ...

  • are designed for providing different content for different user session
  • are evalulate in each request and stored globally
  • are compatible with the http cache
  • can be used to hide or display content (cms elements, products, categories, ...)
  • have no access to the cart scope
  • have access to customer scope

• Checkout rules ...

  • are designed for using during the customer cart prozess and for the after-order process
  • evaluated inside the cart process
  • are not compatible with the http cache and can not be used for "content"
  • can be used for promotions, risk management, surcharges, ...
  • have access to the cart scope
  • have access to the customer scope

• Flow rules ...

  • are designed to executed within flows
  • will only be evaluated when they are queried inside a flow
  • have access to two different scopes:
    • Session scope: customer-id + token (for cart)
    • Order scope: customer-id + order-id (for order)
  • have access to the database connection and can fetch own data

Consequences

• It is no more necessary to load the cart in a "storefront" request, until the customer gets into the checkout process
  • This leads to an extrem performance boost
• Rules are separated for the different purposes
  • This leads to a clear API of each rule class and which data is when available
• Flows conditions becomes much more transparent in case of when which condition will be evaluated
• We just have to calculate the cart one time and not inside a loop
  • Chicken-egg issue will be solved
• We have to "train" customers and developers what's the difference between this systems
• We will loose much flexiblity because not each rule type have access like the current system before
  • For example: Hide a category until you have something inside the cart

The above system is intended to serve as a foundation and standard implementation (also for our SaaS product).

Of course, it will still be possible to manipulate the corresponding input parameters for the different rule types and to individualize them in projects.

[kleinmann]

I like the idea of more specific rule scopes, since most rules in our projects are only used for one of those at a time.

I'm not certain that the chicken-egg cart calculation issue would be resolved, though, since the proposed Checkout rules would still affect each other, wouldn't they? Unless the entire flow would be changed.
I'm curious if there are any actual real-life examples where inter-dependent rules are used and have multiple changes in subsequent cart calculation/processor iterations. I don't have any myself off the top of my head.
If there are none, it would be a nice optimization to just skip that dependency resolution in a loop 🙂

[OliverSkroblin]

Hi Uwe,

thanks for the input. Yes, you could be right that certain configurations could still lead to a loop.
However, I also believe that the interpretation of the rules between the individual processors will significantly improve the probability of a further loop being required.

To be honest, I'm not sure if there are any real-life examples out there right now, and if there were, I'd be happy if people would share them here 👍

[SpiGAndromeda]

I think loop-problems could still exist. E.g. if a promotion is added by a rule but this changes an overall price of the order which changes something else and so on (please correct me if I am wrong).

The segregation of scopes is very nice as it allows further and more specific validations of created/applied rules (without adding much complexity because a validation wouldn't have to handle all scopes).
Maybe this segregation isn't enough. At least for the checkout rules. Segregating them into application categories (not a good name) that are applied in a specific order might solve even more problems and eliminate the need for loop-calculations. It might make the system less flexible but more transparent. Tests would be a lot more easier as well.

The application categories and the order of the application would have to be discussed.

[benmarks]

Fair points for sure. However - a system of precedents can be another dimension of complexity & inefficiency; ref. the legendary spaghetti graphviz of Magento 1's totals priority sorting.

[renegr-moonshiner]

Hej certainly, performance improvements would be great and as far as I understand this RFC points mostly down to two points:

• Performance Improvements
• Endless loop prevention

Please add to my thoughts if I missed a crucial point.

The one thing, that is really powerful (mixing contexts: the condition has a different context - eg. cart - as the result-context (e.g. category page)) such as in your example provided

For example: Hide a category until you have something inside the cart

Is something that is really amazing and is a thing that we would miss surely.

So coming back to the initial problem statements, what could be alternative solutions:

• Performance

There are various patterns (such as memoization) which could probably solve this issue as well?!
Memoization takes simply the "input" aka "context" of a given query to doublecheck if the certain query has been executed.

So from a data driven perspective, what is context aka input:

The Rule
The context (e.g. user, user-session, cart, saleschannel) - e.g. hashing the combination of the ids of these values is a quick and optimized way to compress the information of the context in a very tiny memory foodprint.

Putting that together into a memoization pattern, and store the desired outcome within this memoized entity could probably solve the performance topic.

Of course first calculation would take time.

Using well known caching patterns (in-memory, TTL, invalidation) could be used to solve the problem statement on how long and where should this be memoized (such as redis, memcache). I can not forsee how big the memory foodprint would be, so this might imply another problem? On the otherside the memoization of this rules during a storefront session, are usually very shortlived, as common visitor time does not exceed 5-10 minutes in many cases. Maybe even lower. There might be of course cases, where more minutes are appropriate, but if the TTL can be configured for rule-caching on a rule-basis. With this in mind it would be possible to fine tune performance/memory tradeoffs on a rule basis and on the basis of the customers needs.

For the other problem statement "endless loop prevention":

I can't share any bits right now to this, but there might be patterns available that are commonly used within compilers for languages, that need to take care to handle circular references. As I am not an expert in writing compilers, I can not pinpoint a specific pattern, that helps in detecting & preventing such cases.

My point to that is simply, maybe there's a pattern in a different domain, which proves a solid solution which can be applied to the mighty and powerful rule-builder-system as well.

---

Thanks for bringing this topic up for open discussion.

Full Disclaimer: My knowledge to the underlying implementation for the current rule builder is very limited. The thoughts presented here merely rely on trying to understand the actual problem statement, and seeking patterns in other domains that might solve this problem, without trying to change the whole system.

[OliverSkroblin]

Hey, thanks for the comment.

for the background knowledge of the rule system (maybe i add this in the RFC description), here is what's going on the system right now, when the customer is logged in or has products in the cart

• We load all rules
• We (loop) match all rules + calculate the cart
• Results in an offset of all rule ids
• Now content will be loaded (based on rules, or not)
• Now we have to write the response and the cache item for this request
  • At this point, we have to add all "matching" rules to a cache cookie
  • This leads to a permutation-hell for system which are massive using the rule system
  • And ATM, the rule system is used a lot and we have system with thounds of rules (cache hit rate 5%)

That's for the background.

Is something that is really amazing and is a thing that we would miss surely.

Yes, I can understand that, but this is the biggest performance killer of all. But it will still be possible for projects to implement exactly such requirements.

Therefore, my approach is to say: By default, the minimum set will be delivered with the best possible performance, and agencies can then optimize the systems for the customers so that only necessary things are included again, instead of the other way around.

There are various patterns (such as memoization) which could probably solve this issue as well?!

Yes, many performance problems in programming can be solved this way. But here the problem lies on a different level. It's not the problem that the code is slow, it's the problem that the code has to run on every page for every customer which is logged in or has something in the cart.

With a scalable system, you don't want to have as many customers as possible hit your application in the first place, but rather process them via a reverse proxy (varnish, fastly, etc). But since all rules are evaluated for every request, you have a very high cache permutation, which means that you no longer have any cache hits.

And the problem with this flexibility, which is currently activated by default, is simply that the customer can influence his system performance by configuring rules (perhaps only for promotions or flows) and has no idea about it.

[dithom]

Thanks for the explanation! Could you elaborate on writing all rules to a cache cookie and how this leads to a permutation hell for the system?

[OliverSkroblin]

Hey, sorry, it was a bit busy. Sure, I'd be happy to explain.

With the Rule System, you can check the following things (Conditions):

1. Customer logged in
2. Customer has already ordered something
3. Customer has something in the shopping cart

You can nest these conditions as you like in "rules".

Now, you use these rules (or conditions) for various content:

• Hide CMS element
• Product prices
• Show/hide categories

Customers entering the shop, can now potentially experience 6 different scenarios and view different content:

1. Not logged in
2. Not logged in + Something in the shopping cart
3. Logged in
4. Logged in + Something in the shopping cart
5. Logged in + Has ordered something
6. Logged in + Has ordered something + Something in the shopping cart

And they can walk through this state within a single session.

1. Go to shop (not logged in) (first-key)
2. Add something to cart (second-key)
3. login (third-key)
4. order (fourth-key)
5. add something to cart (six-key)

These are currently just 6 rules checking a boolean "Has/Is === yes/no." However, we have rules that can check various values like "Customer from Group X," "Has ordered X shoes in the last 3 months," etc.

All these configurations "could" lead to different shop content, meaning we need to store them under a different cache key.

Additionally, we have to consider delivering content in different languages and tax statuses, and with other currencies.

This essentially results in a cache hit rate of 1% (exaggerated).

[oerkel]

A small example of how we currently use it. Our plugin creates various rules to be used in the rule builder for payment methods. Our rule determines a value via REST that is compared with the value entered in the rule builder. The problem is that this rule may only be executed for a specific event (onFrontendCheckoutFinishOrder) after the customer has confirmed the terms and conditions by clicking on the order button. We therefore had to build a subscriber that now controls the execution of the rule.

[kleinmann]

This kinda feels like it's not a good problem match for the rule builder 🤔 Are you blocking payment method usage in this way? Is it a Shopware Cloud shop or self-hosted?
I see rule matching more like "request/context metadata", so something like a user's state that can be used for decision-making. Limiting it to a single event feels like it should not run via the rule system.

[JoshuaBehrens]

@kleinmann I just had a similar issue, where I had to evaluate a situation on the database. I don't want to beforehand analyze on the database if I don't know how the rule is structured as this influences the criteria I want to ask the database. Eventually I structured it always as last statement and used static in-memory cache.

[r4pt0s]

What about a slightly different approach for handling the rule evaluation.
There are already scopes which we can define where rules can get applied like the cart.
What if there would be some kind of rule dependencies like

cart rules => trigger rule evaluation for content rules

But content rules can not trigger evaluation of cart rules.

Which will basically mean, rule scopes get a tag applied like for a content rule scope for example

CART_SCOPE_RULE_AWARE

Then in the rule evaluation phase, there could be logic to trigger a set of rule evaluations with their dependency rule scopes.

This would keep the awesome dynamic "handle it everywhere" behavior + there could be helpful events for plugin developers to inject their own dependencies.

A performance boost will also be kept. Only dependency rule evaluations get executed but not those where a early return happens because of a not applied dependency. If you take it a step further, tags - or better said flags in a individual rule context - can get applied for each rule individually in the administration by orchestrating rule scope dependencies if a user has the right administration permissions.

Tags could maybe also have some kind of underlying class where further configurations can get applied for preventing endless loop runs of rule evaluations which are already processed since the last evaluation phase.

[OliverSkroblin]

Interesting approach. What I don't fully understand, is when you want to evaluate which rules.

All rules are currently evaluated as soon as the context and cart are loaded. Since the rules are stored in context, we currently always have to use the cart to create a complete SalesChannelContext object.

When exactly do you want to differentiate which rules are taken into account or not?

Then the question would be, how do you teach the reverse proxy (e.g. fastly) that depending on the route a different cache header is taken into account?

[r4pt0s]

That's a fair point.

For fresh sessions

If a customer arrives and starts a fresh session, the cached version gets used anyway, therfore the defaults apply or the cache object gets build.

For running sessions

Evaluation of the rules could happen as soon as the cart changes on demand and once the context gets tied to a customer account.
Also considered should be customer entity changes like changing the address.
This then should result in a new cache key and therefore a new cache object.

Basic idea

Evalution of rules not on each request, instead evaluation should be event based.

---

The default shopware cache routines will be applied in this case because of the context as far as I'm aware of but I could totally be wrong.

A entire different story would be rules like the datetime rule where you can set a date and a time range for the rule to resolve to true.

I'm not 100% about the reverse proxy and the cache headers though and if they might not work anymore with the idea.

Chances are that I'm a bit off track with the my idea because I might have missed side effects which I'm not 100% aware of right now.

[OliverSkroblin]

No, the idea is not wrong to calculate rules only when they could theoretically have changed. However, I see the issue that each developer would then have to take care of triggering that accordingly. Additionally, I see a risk when external rules (plugins or project rules) come into play.

Regarding the reverse proxy, I have written a longer comment above, explaining the cache hashing.

In theory, we could also simply say that when we build the Sales Channel Context, we no longer calculate the shopping cart but simply create a Rule Scope like this:

shopware/src/Core/Framework/Rule/RuleScope.php

Line 10 in ae7adf0

abstract class RuleScope

It would have almost the same effect as not loading the shopping cart in the storefront but only in the checkout process.

[OliverSkroblin]

Hey all,

I heard from @shyim that there have been some concerns about this RFC at various events.

I want to emphasize that we will not be completely removing any functionality from the system. However, our goal is to create a stable and performant base solution that can be easily adjusted to return to the previous feature set.

It's crucial for us to establish a solid foundation initially. As consumers of our API, you should only need to add things rather than having to disable unnecessary code from us.

Therefore, I want to highlight:

Yes, we plan to weaken the Rules by default, but with simple adjustments (php/plugin/.env/.yaml), it should be relatively easy to restore a comparable state. And this can be done concurrently, not months after deprecation or the break!

[OliverSkroblin]

See: #3613