Lazy expansion of json payloads that we get from user executors #735
Conversation
There was a problem hiding this comment.
Hi @fokion, thank your for your proposition.
If I well understand, you want to cover the following usecase:
- a user executor, doing assertions on big payload
- the user executor have to be used with small payload and big payload
- The unmarshal done in the parent testcase could be long
About the implementation:
We don't want to add a global flag, unless it's a new feature that we want to test and include in the next major release.
I don't understand the move: you deleted the unmarshal from the runExector on the output by adding it on the range on the Assignments. But, if you don't want a unmarshalling of a big payload, you just need to not add the variable in the output of the user executor.
I am aware that the unmarshal is not efficient, but I am not in favor of fixing this problem in this way.
types_executor.go
Outdated
| } | ||
| if len(m) > 0 { | ||
| for s, i := range m { | ||
| if !strings.Contains(strings.ToUpper(s), "__Type__") && !strings.Contains(strings.ToUpper(s), "__Len__") { |
There was a problem hiding this comment.
strings.ToUpper(s) will never match __Type__
Same for __Len__
Why do you need to filter Type and Len ?
There was a problem hiding this comment.
thanks will need to fix this.
| @@ -574,13 +576,43 @@ func processVariableAssignments(ctx context.Context, tcName string, tcVars H, ra | |||
|
|
|||
| for varname, assignment := range stepAssignment.Assignments { | |||
| Debug(ctx, "Processing %s assignment", varname) | |||
| lazilyLoadJSON := os.Getenv(LAZY_JSON_EXPANSION_FLAG) == FLAG_ENABLED | |||
| if lazilyLoadJSON { | |||
| if strings.Contains(assignment.From, "json") { | |||
There was a problem hiding this comment.
Why do you need to check the json parts in the From attribute? It's possible to have a json variable without a name containing json string.
There was a problem hiding this comment.
we want to check the assignments for the keys that we are generating from the unmarshalling. For example we are making new keys with the name result.systemoutjson for the key result.systemout the check here is to check if the key has that json substring in order to check if we need to unmarshal that payload.
|
The idea is that you can have a big JSON payload , by default we will try to unmarshal it and we may not use the new keys in any assertion. That will cause issues with performance as the constant unmarshalling , copying of values with increase memory and cpu usage. The point here is to unmarshal this payload lazily and for now have that feature tested under a flag so that we dont affect everyone. |
…yloads that we dont need Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>
Signed-off-by: Fokion Sotiropoulos <fokion.s@gmail.com>
fokion
force-pushed
the
feat/optional-json-expansion
branch
from
5cd5e97
to
98066d0
Compare
This is done in order to avoid expanding and storing big payloads that we dont need and we dont use for assertions. The flag is called
VENOM_NO_JSON_EXPANSION.