Performance issue on rendering of home feed

[adrienjoly]

Problem

A lot of time seems to be wasted between querying the logged user's subscriptions (cf orange blocks on the left) and posts (cf orange blocks on the right)

This gets worse on profiles that have a lot of subscriptions.

This performance problem is probably the root cause of recent performance and availability issues, cf:

• User profiles that take more than 1 minute to be provided (https://app.datadoghq.com/incidents/1)
• Database crash on Tue 2023/08/22 13:37:10 UTC (https://app.datadoghq.com/incidents/2)

[adrienjoly]

Logs when accessing my homefeed, after deploying v1.55.29:

0|app  | ▶ 2023-08-26T07:39:23.532Z GET / (userLibrary.controller) {}
0|app  | fetchByAuthors>arrayToSet_4d94501d1f78ac091dbc9b4d: 313.329ms
0|app  | prepareSidebar_fetchRecentActivity: 263.748ms
0|app  | ◀ 2023-08-26T07:39:23.532Z GET / responds 200 after 716 ms

This matches with the trace from Datadog's APM:

(source: https://app.datadoghq.com/apm/trace/1360823663931202456?colorBy=service&env=production&graphType=flamegraph&shouldShowLegend=true&sort=time&spanID=1360823663931202456&spanViewType=metadata&timeHint=1693035563520.005)

=> Observations:

• snip.arrayToSet takes almost half of full request's processing time! (313ms / 716ms)
• fetchRecentActivity mongodb query is also quite costly, and potentially worse on user accounts that are subscribed to thousands of users.

[adrienjoly]

Logs for same request, after optimisation (see commit above):

0|app  | ▶ 2023-08-26T08:26:26.654Z GET / (userLibrary.controller) {}
0|app  | fetchByAuthors>arrayToSet_4d94501d1f78ac091dbc9b4d: 1.185ms
0|app  | prepareSidebar_fetchRecentActivity: 10.33ms
0|app  | ◀ 2023-08-26T08:26:26.654Z GET / responds 200 after 63 ms

=> arrayToSet is 300x faster than the previous version based on reduce() 🎉

APM span confirms the improvement:

source: https://app.datadoghq.com/apm/trace/6625358628523903867?colorBy=service&env=production&graphType=flamegraph&shouldShowLegend=true&sort=time&spanID=6625358628523903867&spanViewType=metadata&timeHint=1693038386648.0042

[adrienjoly]

Next steps:

• let's make sure that home feed rendering time is reasonable, even with profiles that have many subscribers and/or subscriptions.
• remove calls to console.time() and console.timeEnd()

[adrienjoly]

the "activity" sidebar section seems to put high stress on our mongodb instance.

...which makes sense because it starts non-optimized and heavy db queries against all the users followed by the logged user.

=> after several outages in production, I decided to turn that feature off by commenting out the corresponding code, in 52536d3 and 17998a1

=> if there are no outages in the coming week, we should probably just get rid of this feature, for good, by deleting the commented code.

[adrienjoly]

Issue fixed by 4e15f75.

Performance problems reduced by disabling fetching of followed users' activity on feed/home page. (52536d3)

=> remaining tasks were transfered over to issue #682.