You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Although the recent change (#192) improved the performance of the signature requests for copy receipts, we're observing an increasing trend for copy receipt latencies, which started to cause performance issues in our integrations.
We have outlets with >200K queue items and a CNumerator of nearly 5K. We noticed that the latencies have direct correlation with the number of queue items, which can be seen in the table below.
The table shows the correlation of copy receipt latency (red, left y-axis) and number of queue items (yellow, right y-axis) for multiple outlets (x-axis):
Impact
The impact can be seen in the sub-sections for each store. Each sub-section shows the queue size, CNumerator, and a latency heatmap for:
All receipts - signature requests for all receipts
Copy receipts - signature requests for copy receipts
Other receipts - signature requests for non-copy receipts
TL;DR:
Copy receipt latency has a direct correlation with the queue size.
While the copy receipt request is being processed, it blocks the queue and impacts latency of other receipt requests.
This is expected due to the blocking nature of the queue; the main idea is to highlight the spikes in other receipt types are actually a result of copy receipt requests.
Growing copy receipt latencies can be clustered in 2 groups;
First group that has higher latency
Second group that has lower latency, which is mainly a subsequent copy receipt request. Subsequent copy receipt requests that are made in a certain window is always lower. This can be the result of an index cache on the data layer
Store A
Queue size: 216K
CNumerator: 4.7K
Latency heatmap for all receipts:
Latency heatmap for copy receipts:
Latency heatmap for other receipts:
When we focus on the copy receipt latencies, it can be seen that the latency of the subsequent copy receipt requests that are made in a certain window is always lower. This can be the result of an index cache on the data layer. The table below shows copy receipt requests that are grouped in equal time windows:
Findings:
The latency heatmap for copy receipts indicates that the latencies can be clustered in levels; 4-5s and 10-15s
The latency heatmap for other receipts indicates that the latencies are mostly in the same level (~500ms).
Latencies of other receipts seem to be similar (~500ms) for all receipt types. This can also be seen within table under the summary section.
In some cases, an increasing latency can be observed for other receipts. This is a result of the previous copy receipt request blocking the middleware while it is being processed.
Store B
Queue size: 194K
CNumerator: 3.7K
Latency heatmap for all receipts:
Latency heatmap for copy receipts:
Latency heatmap for other receipts:
Findings:
Same as store A.
Store I
Queue size: 42K
CNumerator: 0.9K
Latency heatmap for all receipts:
Latency heatmap for copy receipts:
Latency heatmap for other receipts:
Findings:
Copy receipt request latencies are on the same level (~2.4s-3.1s).
Other receipt request latencies are on the same level (~500ms) with spikes up to ~2.8s that is a side effect of the copy receipt request that is processed in the background.
Expectations
As SaaS and PaaS providers, our expectation is to have stable latencies regardless of the queue size. Growing queue size will cause delays or even interruptions in receipt printing, and as a result, degraded customer experience.
The text was updated successfully, but these errors were encountered:
Hi @mertcanyalhi, thanks a lot for these very detailed analysis, very appreciated! Indeed it looks like the scaling of the intermediate fix we had deployed for #192 did improve the situation, but still doesn't scale perfectly - although the rate of the performance deterioration became much slower, that's very obviously not enough.
We've implemented a second fix that decouples the copy receipt performance from the number of processed receipts and should change the complexity from O(n) to O(1). We're currently working on deploying this, ETA is somewhat about ~2 weeks in my opinion - I'll keep you posted. Sorry about the inconvenience, and again, thanks for the super helpful problem description. 😊
Summary
Although the recent change (#192) improved the performance of the signature requests for copy receipts, we're observing an increasing trend for copy receipt latencies, which started to cause performance issues in our integrations.
We have outlets with >200K queue items and a
CNumerator
of nearly 5K. We noticed that the latencies have direct correlation with the number of queue items, which can be seen in the table below.The table shows the correlation of copy receipt latency (red, left y-axis) and number of queue items (yellow, right y-axis) for multiple outlets (x-axis):
Impact
The impact can be seen in the sub-sections for each store. Each sub-section shows the queue size,
CNumerator
, and a latency heatmap for:TL;DR:
Store A
Latency heatmap for all receipts:
Latency heatmap for copy receipts:
Latency heatmap for other receipts:
When we focus on the copy receipt latencies, it can be seen that the latency of the subsequent copy receipt requests that are made in a certain window is always lower. This can be the result of an index cache on the data layer. The table below shows copy receipt requests that are grouped in equal time windows:
Findings:
Store B
Latency heatmap for all receipts:
Latency heatmap for copy receipts:
Latency heatmap for other receipts:
Findings:
Store I
Latency heatmap for all receipts:
Latency heatmap for copy receipts:
Latency heatmap for other receipts:
Findings:
Expectations
As SaaS and PaaS providers, our expectation is to have stable latencies regardless of the queue size. Growing queue size will cause delays or even interruptions in receipt printing, and as a result, degraded customer experience.
The text was updated successfully, but these errors were encountered: