Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

network dead? #2057

Closed
RazerStvH opened this issue Apr 22, 2024 · 20 comments
Closed

network dead? #2057

RazerStvH opened this issue Apr 22, 2024 · 20 comments

Comments

@RazerStvH
Copy link

Is everyone's network dead? I got this on the server all day:
Tunnel creation success rate: 1%
Routers: 25462 Floodfills: 1518 LeaseSets: 556
Client Tunnels: 8 Transit Tunnels: 19881

@Vort
Copy link
Contributor

Vort commented Apr 22, 2024

Network is under attack right now.
1 hop tunnels work relatively well, 3 hop tunnels are almost unusable.

@SergSel2006
Copy link

who ever need to attack the net...
Getting that situation on my routers too. Waiting for devs to reply

@Vort
Copy link
Contributor

Vort commented Apr 22, 2024

I suggest to wait for commits with mitigations and to build new binary from sources then.

@Vort
Copy link
Contributor

Vort commented Apr 22, 2024

Please try 6ce2c30. Protection should activate after 10 minutes of uptime.
For some OSes binaries are available for download from Actions page.
For other - should be built manually.

@SergSel2006
Copy link

SergSel2006 commented Apr 23, 2024

I heared that java I2P routers are not affected by this. I tested the java router and it worked fine.

@lcharles123
Copy link

Please try 6ce2c30. Protection should activate after 10 minutes of uptime. For some OSes binaries are available for download from Actions page. For other - should be built manually.

Im using this new build, working well for now. Thanks!

Uptime: 10 hours, 8 minutes, 0 seconds
Network status: Firewalled - Symmetric NAT
Tunnel creation success rate: 2%
Received: 583.72 MiB (9.41 KiB/s)
Sent: 357.96 MiB (6.60 KiB/s)
Transit: 29.15 MiB (0.00 KiB/s)
Data path: /root/.i2pd
Hidden content. Press on text to see.
Routers: 5502 Floodfills: 932 LeaseSets: 0
Client Tunnels: 10 Transit Tunnels: 2

@orignal
Copy link
Contributor

orignal commented Apr 23, 2024

I heared that java I2P routers are not affected by this. I tested the java router and it worked fine.

Because they ban all clones including real routers. I wouldn't call it "fine".

@twofaktor
Copy link

Bitcoin network affected too:

Apr 23 13:47:50 bitcoind[242178]: 2024-04-23T13:47:50Z [i2p] Creating persistent SAM session a274793822 with 127.0.0.1:7656
Apr 23 13:47:50 bitcoind[242178]: 2024-04-23T13:47:50Z [i2p] Error listening: Receive timeout (received 0 bytes without terminator before that)

@SergSel2006
Copy link

I heared that java I2P routers are not affected by this. I tested the java router and it worked fine.

Because they ban all clones including real routers. I wouldn't call it "fine".

Yes, this behaviour is not good overall, but the stategy is able to increase the tunnel creation sucess rate to 30%.
But now, as I am using patched i2pd, tunnels are working again as usual.

@SergSel2006
Copy link

Bitcoin network affected too:

Apr 23 13:47:50 bitcoind[242178]: 2024-04-23T13:47:50Z [i2p] Creating persistent SAM session a274793822 with 127.0.0.1:7656
Apr 23 13:47:50 bitcoind[242178]: 2024-04-23T13:47:50Z [i2p] Error listening: Receive timeout (received 0 bytes without terminator before that)

Obviously. Now anything that relates on i2p tunnels won't work because, well, there is no i2p tunnels.

@SilmorSenedlen
Copy link

SilmorSenedlen commented Apr 24, 2024

Network is under attack right now.

Just got OOB on Windows host:
I2Pd_DoS_New
I shooted this in process of killing .
When I saw that my browser had crashed, i2pd consume ~80 GiB of virtual memory.

Version: 2.50.2

Maybe this is also a problem with memory leaks, and not only because of the number of requests from the network.
As I mentioned in a similar previous issue, I would really like to have a limit on memory usage at the application level itself, since Windows, as far as I know, does not have any tools to set such restrictions in the user's environment.

@Vort
Copy link
Contributor

Vort commented Apr 24, 2024

When I saw that my browser had crashed, i2pd consume ~80 GiB of virtual memory.

I noticed similar problem with my node.

Maybe this is also a problem with memory leaks, and not only because of the number of requests from the network.

Attack triggers some (still unknown) bug in i2pd.
Not directly, but because of high load.

In my case leak started to happen after computer was overloaded.
Before that consumption was relatively stable at ~500MB.

@SilmorSenedlen
Copy link

Before that consumption was relatively stable at ~500MB.

Yep,my node usually fits the same mem boundaries.
The stable load for the last month has been ~5-6 MiB and ~30-40K transit tunnels.

@LLE8
Copy link

LLE8 commented Apr 25, 2024

In my experience, while normal operation (not under DDoS-attack) assumed memory leak is from 500KB to 1MB per day, but not every day.

@Vort
Copy link
Contributor

Vort commented Apr 25, 2024

In my experience, while normal operation (not under DDoS-attack) assumed memory leak is from 500KB to 1MB per day, but not every day.

This is, most likely, how heap fragmentation (problem common for many C++ programs) manifests itself.

In comparison, for two cases, which I observed recently, leak happened with speeds of ~1MB/s and ~1GB/s.

@SilmorSenedlen
Copy link

SilmorSenedlen commented Apr 25, 2024

Now I have found that the process has bloated again and consumes ~14 GiB of memory.

i2pd_Res_02
i2pd_Res_01

While I was looking at the virtual memory graph, it remained at the same level, although the Working Set grew by 4-8 KiB every few seconds.
Network activity: ~ 13-16k PPS
Now I see that Virtual Memory has grown in one leap.

@Vort
Copy link
Contributor

Vort commented Apr 25, 2024

I figured out that one of memory leaks (fast, happens during shutdown) appears at this line:

it = m_Peers.insert (std::pair<i2p::data::IdentHash, Peer>(ident, {r, ts})).first;

m_Peers there have size of 0xfffffffffffffff7 and it starts to call new function with 2GB blocks.

It's in fact the same problem as in #2058 - m_Peers needs more mutex locks.

@MakeCartWheel MakeCartWheel mentioned this issue Apr 25, 2024
@SilmorSenedlen
Copy link

SilmorSenedlen commented Apr 25, 2024

Looking at the memory consumption at this moment, at first I thought that the app was crashed and restarted.
But it turned out that ~15-16 GiB, which I observed during the day, collapsed at some point x_X
Although the amount of data being passed thought router and the number of tunnels has not decreased at all.
Are these some kind of "temporary" memory leaks that app can fix itself over time?

i2pd_Res_03
i2pd_Res_04

@SergSel2006
Copy link

SergSel2006 commented Apr 27, 2024

Looking at the memory consumption at this moment, at first I thought that the app was crashed and restarted. But it turned out that ~15-16 GiB, which I observed during the day, collapsed at some point x_X Although the amount of data being passed thought router and the number of tunnels has not decreased at all. Are these some kind of "temporary" memory leaks that app can fix itself over time?

There is no such thing as garbarage collector in C++, all memory in controlled by application. It feels like there was something with memory leak, but then that something was destroyed, freeing all leaked memory back.

@orignal
Copy link
Contributor

orignal commented Apr 27, 2024

It's a race condition that damages std structures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants