Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: resharding #3014

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

Feat: resharding #3014

wants to merge 10 commits into from

Conversation

Skillz4Killz
Copy link
Contributor

No description provided.

@github-actions github-actions bot added the pkg-gateway Affects the gateway package label Apr 28, 2023
@github-actions
Copy link

github-actions bot commented Apr 28, 2023

Benchmark

Detail results of benchmarks
Benchmark suite Base (b1a4739) Latest Head (f22a440) ebe35fb 9d17ea1 2ca8a49
[transformer] message cache check RSS 120.41 MB ±1% 123.92 MB ±1% 119.48 MB ±1% 116.13 MB ±1% 114.77 MB ±1%
[transformer] message cache check Heap Used 117.13 MB ±1% 117.53 MB ±1% 117.75 MB ±1% 117.27 MB ±1% 118.47 MB ±1%
[transformer] message cache check Heap Total 107.93 MB ±1% 105.83 MB ±1% 107.41 MB ±1% 107.67 MB ±1% 106.36 MB ±1%
[transformer] old message cache check RSS 79.65 MB ±1% 83.19 MB ±1% 81.77 MB ±1% 81.28 MB ±1% 80.37 MB ±1%
[transformer] old message cache check Heap Used 103.17 MB ±1% 101.7 MB ±1% 102.02 MB ±1% 101.41 MB ±1% 102.82 MB ±1%
[transformer] old message cache check Heap Total 79.88 MB ±1% 80.93 MB ±1% 77.26 MB ±1% 77.52 MB ±1% 80.93 MB ±1%
[Cache Plugin] RSS 0.49 MB ±3.47% 0.57 MB ±2.98% 0.57 MB ±2.98% 0.57 MB ±2.98% 0.58 MB ±2.98%
[Cache Plugin] Heap Used 10.77 MB ±1.44% 10.64 MB ±1.44% 10.73 MB ±1.44% 10.62 MB ±1.44% 10.77 MB ±1.44%
[Cache Plugin] Heap Total 0 MB ±0% 0 MB ±0% 0 MB ±0% 0 MB ±0% 0 MB ±0%
rest.simplifyUrl 201826 ops/sec ±0.36% 203472 ops/sec ±0.94% 200691 ops/sec ±0.31% 201413 ops/sec ±0.34% 200109 ops/sec ±0.38%
Camelize 1 event 5868 ops/sec ±0.32% 6049 ops/sec ±0.31% 5940 ops/sec ±0.35% 6083 ops/sec ±0.30% 5978 ops/sec ±0.32%
Snakelize 1 event 5876 ops/sec ±0.29% 6021 ops/sec ±0.31% 5946 ops/sec ±0.26% 6054 ops/sec ±0.25% 5965 ops/sec ±0.27%

@codecov
Copy link

codecov bot commented Apr 28, 2023

Codecov Report

Attention: Patch coverage is 25.78947% with 141 lines in your changes are missing coverage. Please review.

Project coverage is 79.40%. Comparing base (1e6f22e) to head (f22a440).
Report is 78 commits behind head on main.

❗ Current head f22a440 differs from pull request most recent head c4267b5. Consider uploading reports for the commit c4267b5 to get more accurate results

Files Patch % Lines
packages/gateway/src/manager.ts 25.78% 141 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3014      +/-   ##
==========================================
+ Coverage   76.72%   79.40%   +2.68%     
==========================================
  Files         193      182      -11     
  Lines       23384    21227    -2157     
  Branches      763      460     -303     
==========================================
- Hits        17941    16856    -1085     
+ Misses       5434     4364    -1070     
+ Partials        9        7       -2     
Flag Coverage Δ *Carryforward flag
bot 56.30% <ø> (+1.77%) ⬆️
bot-e2e 54.64% <ø> (+0.19%) ⬆️ Carriedforward from 9d17ea1
bot-unit 47.38% <ø> (-1.83%) ⬇️
discordeno 100.00% <ø> (+91.66%) ⬆️
discordeno-unit 100.00% <ø> (+91.66%) ⬆️
gateway 68.14% <25.78%> (-6.40%) ⬇️
gateway-integration 68.14% <25.78%> (-6.40%) ⬇️
gateway-unit 28.45% <18.94%> (-3.82%) ⬇️
rest 88.23% <ø> (+4.58%) ⬆️
rest-e2e 29.85% <ø> (+2.52%) ⬆️ Carriedforward from 9d17ea1
rest-unit 71.03% <ø> (-0.05%) ⬇️
types 100.00% <ø> (ø)
types-unit 100.00% <ø> (ø)
utils 96.69% <ø> (+13.47%) ⬆️
utils-unit 96.69% <ø> (+13.47%) ⬆️

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Skillz4Killz Skillz4Killz marked this pull request as ready for review April 29, 2023 18:26
Skillz4Killz and others added 2 commits September 25, 2023 10:29
Co-authored-by: Awesome Stickz <awesome@stickz.dev>
Co-authored-by: Awesome Stickz <awesome@stickz.dev>
Copy link

github-actions bot commented Dec 4, 2023

This pull request has gone stale for over a month. If this pull request is still useful, leave a comment below. Otherwise, it will be closed shortly.

@lts20050703
Copy link
Member

@Skillz4Killz Is there anything missing/not working?

@Skillz4Killz Skillz4Killz requested a review from a team as a code owner April 27, 2024 12:16
@github-actions github-actions bot added the website Affects the website label Apr 27, 2024
docker-apps/rest-passthrough/CHANGELOG.md Outdated Show resolved Hide resolved
Comment on lines +75 to +76
// TODO: fetch bot gateway info
const sessionInfo = await gateway.resharding.getSessionInfo()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this TODO done yet?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe so? The comment says to get bot gateway info and I think that's what it does since the first commit to this PR 🤔

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it though? The code inside getSessionInfo just throw an error and require the end user to override the handler? I thought resharding is enabled by default and should work out of the box?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like it's intended that way because we don't have a way to use rest in gateway right?

packages/gateway/src/manager.ts Show resolved Hide resolved
Comment on lines +131 to +134
// Ignore events until we are ready
events: {
async message(shard, payload) {
if (payload.t === 'READY') {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Ignore events until we are ready
events: {
async message(shard, payload) {
if (payload.t === 'READY') {
events: {
async message(shard, payload) {
// Ignore events until we are ready
if (payload.t === 'READY') {

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not entirely sure if this comment talks about gateway events or the events like messageCreate etc.

Comment on lines +165 to +166
// Mark that we are making an identify request so another is not made.
bucket.identifyRequests.push(resolve)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't seem to find any lines checking that "if we are making an identify request, don't make another one" or something like that?

Comment on lines +175 to +177
async updateGuildsShardId(guildIds, shardId) {
logger.warn(`[Resharding] Updating the following guild ids shard to #${shardId}: ${guildIds.join(', ')}`)
},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uhh is this function/method supposed to do something? Why is there only 1 logger.warn line?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we even need this function internally? I feel like this is another function that was supposed to be overrode by the user to update their guilds shard ids if they store it

Comment on lines +195 to +200
shard.events.message = async function (_, message) {
// Member checks need to continue but others can stop
if (message.t !== 'GUILD_MEMBERS_CHUNK') return
// Process only the chunking events
oldHandler?.(shard, message)
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wondering if anyone can tell me why "member checks need to continue".

packages/gateway/src/manager.ts Outdated Show resolved Hide resolved
website/docs/bigbot/step-2-rest.md Outdated Show resolved Hide resolved
website/docs/bigbot/step-3-gateway.md Outdated Show resolved Hide resolved
@Fleny113 Fleny113 marked this pull request as draft April 30, 2024 14:28
@github-actions github-actions bot removed the website Affects the website label May 1, 2024
Co-authored-by: LTS20050703 <lts20050703@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pkg-gateway Affects the gateway package
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants