New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimize memory collection for plugin manager #15847
Conversation
d7064b7
to
eac4ea7
Compare
Here is the report for 444c437 (saleor:ir-168-optimize-memory-cleanup) No differences were found.
# saleor.graphql.accountbenchmark account
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
addresses for federation query count 9 9 2
customers query 51 51 0
delete staff members 40 40 0
query staff user 22 22 3
staff create 22 22 3
staff update groups and permissions 35 35 5
users for federation query count 8 8 3
# saleor.graphql.accountbenchmark permission group
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
groups for federation query count 3 3 0
permission group create 24 24 0
permission group delete 29 29 3
permission group query 14 14 0
permission group update 43 43 2
permission group update remove users with manage staff 35 35 3
# saleor.graphql.appbenchmarks app extensions
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
app extensions 15 15 0
app extensions with filter[filter0] 14 14 0
app extensions with filter[filter1] 10 10 0
app extensions with filter[filter2] 14 14 0
app extensions with filter[filter3] 10 10 0
# saleor.graphql.appbenchmarks apps
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
apps for federation query count 9 9 3
apps with tokens and webhooks 9 9 0
# saleor.graphql.attributebenchmark attribute
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
attribute translation 6 6 0
attribute value translation 20 20 0
query attribute 6 6 0
query attributes 8 8 0
# saleor.graphql.channelbenchmark channel
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
channels query 5 5 0
# saleor.graphql.channelmutations channel update
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
channel update mutation remove warehouse 39 39 6
# saleor.graphql.checkoutbenchmark checkout mutations
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
add billing address to checkout 71 71 9
add checkout lines 77 77 12
add checkout lines with external shipping 180 180 75
add checkout lines with reservations 163 163 58
add delivery to checkout 74 74 12
add shipping to checkout 83 83 15
checkout email update 38 38 1
checkout payment charge 51 51 15
checkout shipping address update 78 78 9
checkout voucher code 95 95 16
complete checkout 238 238 79
complete checkout preorder 233 233 80
complete checkout with digital line 276 276 93
complete checkout with out of stock webhook 240 240 79
complete checkout with single line 240 240 79
create checkout 78 78 8
create checkout with reservations 138 138 24
customer complete checkout 251 251 85
customer complete checkout for cc 226 226 75
update checkout lines 70 70 12
update checkout lines with reservations 169 169 71
# saleor.graphql.checkoutbenchmark checkouts
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
staff multiple checkouts 6 6 0
# saleor.graphql.checkoutbenchmark homepage
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
user checkout details 58 58 5
user checkout details with tax app 78 78 26
# saleor.graphql.discountbenchmark promotion create
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
promotion create 26 26 1
# saleor.graphql.discountbenchmark promotion delete
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
promotion delete 25 25 1
# saleor.graphql.discountbenchmark promotion rule create
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
promotion rule create 24 24 1
# saleor.graphql.discountbenchmark promotion rule delete
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
promotion rule delete 21 21 0
# saleor.graphql.discountbenchmark promotion rule update
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
promotion rule update 32 32 0
# saleor.graphql.discountbenchmark promotion update
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
promotion update 13 13 0
# saleor.graphql.discountbenchmark promotions
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
promotions querytest promotions query 9 9 0
# saleor.graphql.discountbenchmark sales
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
sales query with channel slug 9 9 0
sales query without channel slug 7 7 0
# saleor.graphql.discountbenchmark voucher code bulk delete
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
voucher code bulk delete queries 20 20 2
# saleor.graphql.discountbenchmark vouchers
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
vouchers query with channel slug 20 20 0
vouchers query withot channel slug 19 19 0
# saleor.graphql.giftcardbenchmark gift card mutations
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
create never expiry gift card 27 27 4
gift card bulk activate by staff 15 15 3
update gift card 35 35 4
# saleor.graphql.giftcardbenchmark gift card queries
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
filter gift cards by products 14 14 3
filter gift cards by tags 12 12 3
filter gift cards by used by user 13 13 3
query gift card details 14 14 3
query gift cards 12 12 3
# saleor.graphql.orderbenchmark fulfillment
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
fulfillment query 6 6 0
# saleor.graphql.orderbenchmark fulfillment refund and return products
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
fulfillment refund products order lines 62 62 1
fulfillment return products order lines 104 104 8
# saleor.graphql.orderbenchmark order
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
staff multiple draft orders 7 7 0
staff multiple orders 7 7 0
staff order details 58 58 8
user order details 56 56 9
# saleor.graphql.orderbenchmark order bulk create
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
order bulk create 20 20 2
# saleor.graphql.orderbenchmark order fulfill
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
order fulfill 18 18 0
order fulfill with gift cards 18 18 0
# saleor.graphql.orderbenchmark order lines create
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
order lines create 95 95 17
order lines create variants on promotion 154 154 22
# saleor.graphql.pagebenchmark page type
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
page types for federation query count 3 3 0
query page type 18 18 6
query page types 25 25 12
# saleor.graphql.paymentbenchmark payment transactions
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
payment transactions 54 54 20
# saleor.graphql.productbenchmark category
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
categories children 3 3 0
categories for federation query count 3 3 0
category delete 48 48 4
category view 28 28 1
# saleor.graphql.productbenchmark collection
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
collection add products 20 20 5
collection bulk delete 27 27 3
collection view 7 7 0
collections for federation query count 4 4 0
create collection 35 35 5
delete collection 38 38 5
remove products from collection 17 17 5
retrieve collection channel listings 5 5 0
# saleor.graphql.productbenchmark homepage
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
report product sales 10 10 3
retrieve product list 4 4 0
# saleor.graphql.productbenchmark product
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
filter products by attributes 7 7 0
filter products by boolean attributes 14 14 0
filter products by gift card 12 12 1
filter products by numeric attributes 13 13 0
product create 48 48 3
product details 41 41 1
product translations 5 5 0
products for federation query count 6 6 1
products media for federation query count 5 5 0
products types for federation query count 2 2 0
retrieve channel listings 20 20 0
retrieve product attributes 8 8 0
retrieve product images 4 4 0
retrieve product media 4 4 0
retrive products with product types and attributes 7 7 0
update product 69 69 5
# saleor.graphql.productbenchmark product variant channel listing update
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
variant channel listing update 32 32 3
# saleor.graphql.productbenchmark variant
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
product variant bulk create 45 45 1
product variant create 59 59 3
products variants for federation query count 6 6 1
retrieve variant list 34 34 1
update product variant 60 60 7
# saleor.graphql.productbenchmark variant stocks
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
product variants stocks create 28 28 4
product variants stocks create with single webhook called 27 27 4
product variants stocks delete by id 29 29 4
product variants stocks delete by sku 29 29 4
product variants stocks delete with out of stock webhook many calls 25 25 3
product variants stocks update by sku 27 27 4
product variants stocks update byid 27 27 4
query product variants stocks 10 10 0
# saleor.graphql.producttest product sorting attributes
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
sort product not having attribute data 20 20 0
# saleor.graphql.shippingbenchmark shipping methods
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
vouchers query with channel slug 7 7 0
vouchers query without channel slug 7 7 0
# saleor.graphql.shippingbenchmark shipping zones
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
shipping zones query 6 6 0
# saleor.graphql.shopbenchmark homepage
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
retrieve shop 2 2 0
# saleor.graphql.warehousebenchmark stock bulk update
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
stocks bulk update queries count 31 31 5
# saleor.graphql.warehousebenchmark stocks
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
stocks query 5 5 0
# saleor.graphql.webhookbenchmark webhook events
test name left count right count duplicate count
------------------------------------------------------------------------ ----------- ----------- ---------------
webhooks 6 6 0
|
.../tests/e2e/checkout/shipping/test_checkout_invalidate_price_based_shipping_on_voucher_add.py
Outdated
Show resolved
Hide resolved
.../tests/e2e/checkout/shipping/test_checkout_invalidate_price_based_shipping_on_voucher_add.py
Outdated
Show resolved
Hide resolved
...l_shipping_methods_in_checkout/test_use_external_shipping_methods_in_checkout_core_1652.yaml
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@maarcingebala I don't remember the exact details, but I remember that when we were adding the webhook for shipping methods, we had some issues with the combination of sync shipping method webhook and a voucher. As a double-check I would suggest checking locally some cases with different types of vouchers.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
QA approved.
During incident 168 we discovered that some instances of PluginsManager have reference cycles and are not collected by the garbage collector after each request. In cases where many channels are used (e.g. over 300), this leads to significant memory usage and eventually OOM worker kills.
Reference cycles were caused by properties on CheckoutInfo that are evaluated as lazy objects (
SimpleLazyObject
). When initialized, they were holding the reference to PluginsManager which was not collected after request execution. To fix this, these properties were refactored to not use lazy objects::CheckoutInfo.all_shipping_methods
- changed intofunctools.cached_property
, this field can call external webhook and should be evaluated only once. Fields that were needed to calculate it were moved to CheckoutInfo as class members.update_delivery_method_lists_for_checkout_info
functions was changed to clean this property when any of the members needed to calculate it it changed.CheckoutInfo.valid_pick_up_points
- changed intofunctools.cached_property
, it only runs DB queries but they'll be cached, so we perform fewer in case the property is called again.CheckoutInfo.delivery_method_info
- changed into@property
, it uses cached property inside and doesn't need to be cached itself.Additionally, some
__del__
methods were added to make sure that certain objects are cleared from memory on object finalization:DataLoader.__del__
BasePlugin.__del__
PluginsManager.__del__
Lastly, the GraphQLView has now "finally" block where we clear the dataloaders that were cached in the request.
Impact
Docs
Pull Request Checklist
ADDED_IN_X
,PREVIEW_FEATURE
, etc.)