Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bitswap/client: memory leak on BlockPresenceManager #574

Open
Dreamacro opened this issue Feb 1, 2024 · 3 comments
Open

bitswap/client: memory leak on BlockPresenceManager #574

Dreamacro opened this issue Feb 1, 2024 · 3 comments
Assignees
@hacdias
Labels
kind/bug A bug in existing code (including security flaws) need/analysis Needs further analysis before proceeding P1 High: Likely tackled by core team if no one steps up

Comments

@Dreamacro
Copy link
Contributor

Recently, I observed that when using bitswap for fetching data, there is a large amount of memory consumption and no release after downloading.

I use pprof and find most heap is occupied here.

boxo/bitswap/client/internal/blockpresencemanager/blockpresencemanager.go

Line 40 in 89bceff

bpm.presence[c] = make(map[peer.ID]bool)

I try to print stat after bpm.presence[c] = make(map[peer.ID]bool) and I found bpm.presence would delete on BlockPresenceManager.RemoveKeys, so I added some logs.

I found that there is a race condition problem here.

  1. BlockPresenceManager.RemoveKeys has been called and remove the CidA
  2. BlockPresenceManager.ReceiveFrom has been called and created bpm.presence[CidA]
  3. bpm.presence[CidA] is no longer used or removed, and the map is getting bigger and bigger.
@Dreamacro Dreamacro added the need/triage Needs initial labeling and prioritization label Feb 1, 2024
@welcome Welcome
Copy link

welcome bot commented Feb 1, 2024

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment.
Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

  • "Priority" labels will show how urgent this is for the team.
  • "Status" labels will show if this is ready to be worked on, blocked, or in progress.
  • "Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.io if you just need general support.

@aschmahmann aschmahmann added need/analysis Needs further analysis before proceeding kind/bug A bug in existing code (including security flaws) and removed need/triage Needs initial labeling and prioritization labels Feb 6, 2024
@aschmahmann
Copy link
Contributor

@Dreamacro can you attach or link to a pprof dump for analysis? Also, which versions of boxo and go-libp2p are you using?

@aschmahmann aschmahmann added the P1 High: Likely tackled by core team if no one steps up label Feb 6, 2024
@Dreamacro
Copy link
Contributor Author

Dreamacro commented Feb 7, 2024 •
edited

@aschmahmann

I first provide a few fragments of pprof/heap. I can confirm a memory leak here because the leakage disappears when I replace map[cid.Cid]map[peer.ID]bool with a cache with time. Dreamacro@df0a587

heap fragments: (Although boxo 0.15 is shown below, it has been upgraded to boxo 0.17)

258: 1387008 [2230: 11988480] @ 0xcd2266 0xcd3d10 0xcd7025 0x1c257ec 0x1c257f0 0x1c549fc 0x1c58ade 0x1c58ffa 0x1c6df45 0x1c39d31 0x1b0a0df 0x1b09108 0x1b32687 0xd32fe1
#	0x1c257eb	github.com/ipfs/boxo/bitswap/client/internal/blockpresencemanager.(*BlockPresenceManager).updateBlockPresence+0x22b	github.com/ipfs/boxo@v0.15.0/bitswap/client/internal/blockpresencemanager/blockpresencemanager.go:48
#	0x1c257ef	github.com/ipfs/boxo/bitswap/client/internal/blockpresencemanager.(*BlockPresenceManager).ReceiveFrom+0x22f		github.com/ipfs/boxo@v0.15.0/bitswap/client/internal/blockpresencemanager/blockpresencemanager.go:30
#	0x1c549fb	github.com/ipfs/boxo/bitswap/client/internal/sessionmanager.(*SessionManager).ReceiveFrom+0x9b				github.com/ipfs/boxo@v0.15.0/bitswap/client/internal/sessionmanager/sessionmanager.go:158
#	0x1c58add	github.com/ipfs/boxo/bitswap/client.(*Client).receiveBlocksFrom+0x5fd							github.com/ipfs/boxo@v0.15.0/bitswap/client/client.go:336
#	0x1c58ff9	github.com/ipfs/boxo/bitswap/client.(*Client).ReceiveMessage+0x279							github.com/ipfs/boxo@v0.15.0/bitswap/client/client.go:380
#	0x1c6df44	github.com/ipfs/boxo/bitswap.(*Bitswap).ReceiveMessage+0x84								github.com/ipfs/boxo@v0.15.0/bitswap/bitswap.go:181
#	0x1c39d30	github.com/ipfs/boxo/bitswap/network.(*impl).handleNewStream+0x530							github.com/ipfs/boxo@v0.15.0/bitswap/network/ipfs_impl.go:431
#	0x1b0a0de	github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).SetStreamHandler.func1+0x3e					github.com/libp2p/go-libp2p@v0.32.1/p2p/host/basic/basic_host.go:598
#	0x1b09107	github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).newStreamHandler+0x6a7						github.com/libp2p/go-libp2p@v0.32.1/p2p/host/basic/basic_host.go:440
#	0x1b32686	github.com/libp2p/go-libp2p/p2p/net/swarm.(*Conn).start.func1.1+0xa6							github.com/libp2p/go-libp2p@v0.32.1/p2p/net/swarm/swarm_conn.go:137

109: 1185920 [3482: 37884160] @ 0xcd2266 0xcd3d10 0xcd7025 0x1c257ec 0x1c257f0 0x1c549fc 0x1c58ade 0x1c58ffa 0x1c6df45 0x1c39d31 0x1b0a0df 0x1b09108 0x1b32687 0xd32fe1
#	0x1c257eb	github.com/ipfs/boxo/bitswap/client/internal/blockpresencemanager.(*BlockPresenceManager).updateBlockPresence+0x22b	github.com/ipfs/boxo@v0.15.0/bitswap/client/internal/blockpresencemanager/blockpresencemanager.go:48
#	0x1c257ef	github.com/ipfs/boxo/bitswap/client/internal/blockpresencemanager.(*BlockPresenceManager).ReceiveFrom+0x22f		github.com/ipfs/boxo@v0.15.0/bitswap/client/internal/blockpresencemanager/blockpresencemanager.go:30
#	0x1c549fb	github.com/ipfs/boxo/bitswap/client/internal/sessionmanager.(*SessionManager).ReceiveFrom+0x9b				github.com/ipfs/boxo@v0.15.0/bitswap/client/internal/sessionmanager/sessionmanager.go:158
#	0x1c58add	github.com/ipfs/boxo/bitswap/client.(*Client).receiveBlocksFrom+0x5fd							github.com/ipfs/boxo@v0.15.0/bitswap/client/client.go:336
#	0x1c58ff9	github.com/ipfs/boxo/bitswap/client.(*Client).ReceiveMessage+0x279							github.com/ipfs/boxo@v0.15.0/bitswap/client/client.go:380
#	0x1c6df44	github.com/ipfs/boxo/bitswap.(*Bitswap).ReceiveMessage+0x84								github.com/ipfs/boxo@v0.15.0/bitswap/bitswap.go:181
#	0x1c39d30	github.com/ipfs/boxo/bitswap/network.(*impl).handleNewStream+0x530							github.com/ipfs/boxo@v0.15.0/bitswap/network/ipfs_impl.go:431
#	0x1b0a0de	github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).SetStreamHandler.func1+0x3e					github.com/libp2p/go-libp2p@v0.32.1/p2p/host/basic/basic_host.go:598
#	0x1b09107	github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).newStreamHandler+0x6a7						github.com/libp2p/go-libp2p@v0.32.1/p2p/host/basic/basic_host.go:440
#	0x1b32686	github.com/libp2p/go-libp2p/p2p/net/swarm.(*Conn).start.func1.1+0xa6							github.com/libp2p/go-libp2p@v0.32.1/p2p/net/swarm/swarm_conn.go:137

180: 483840 [1333: 3583104] @ 0xcd2266 0xcd3d10 0xcd7025 0x1c257ec 0x1c257f0 0x1c549fc 0x1c58ade 0x1c58ffa 0x1c6df45 0x1c39d31 0x1b0a0df 0x1b09108 0x1b32687 0xd32fe1
#	0x1c257eb	github.com/ipfs/boxo/bitswap/client/internal/blockpresencemanager.(*BlockPresenceManager).updateBlockPresence+0x22b	github.com/ipfs/boxo@v0.15.0/bitswap/client/internal/blockpresencemanager/blockpresencemanager.go:48
#	0x1c257ef	github.com/ipfs/boxo/bitswap/client/internal/blockpresencemanager.(*BlockPresenceManager).ReceiveFrom+0x22f		github.com/ipfs/boxo@v0.15.0/bitswap/client/internal/blockpresencemanager/blockpresencemanager.go:30
#	0x1c549fb	github.com/ipfs/boxo/bitswap/client/internal/sessionmanager.(*SessionManager).ReceiveFrom+0x9b				github.com/ipfs/boxo@v0.15.0/bitswap/client/internal/sessionmanager/sessionmanager.go:158
#	0x1c58add	github.com/ipfs/boxo/bitswap/client.(*Client).receiveBlocksFrom+0x5fd							github.com/ipfs/boxo@v0.15.0/bitswap/client/client.go:336
#	0x1c58ff9	github.com/ipfs/boxo/bitswap/client.(*Client).ReceiveMessage+0x279							github.com/ipfs/boxo@v0.15.0/bitswap/client/client.go:380
#	0x1c6df44	github.com/ipfs/boxo/bitswap.(*Bitswap).ReceiveMessage+0x84								github.com/ipfs/boxo@v0.15.0/bitswap/bitswap.go:181
#	0x1c39d30	github.com/ipfs/boxo/bitswap/network.(*impl).handleNewStream+0x530							github.com/ipfs/boxo@v0.15.0/bitswap/network/ipfs_impl.go:431
#	0x1b0a0de	github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).SetStreamHandler.func1+0x3e					github.com/libp2p/go-libp2p@v0.32.1/p2p/host/basic/basic_host.go:598
#	0x1b09107	github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).newStreamHandler+0x6a7						github.com/libp2p/go-libp2p@v0.32.1/p2p/host/basic/basic_host.go:440
#	0x1b32686	github.com/libp2p/go-libp2p/p2p/net/swarm.(*Conn).start.func1.1+0xa6							github.com/libp2p/go-libp2p@v0.32.1/p2p/net/swarm/swarm_conn.go:137

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hacdias hacdias

Labels
kind/bug A bug in existing code (including security flaws) need/analysis Needs further analysis before proceeding P1 High: Likely tackled by core team if no one steps up
Projects
No open projects
IPFS Shipyard Team
Status: 🥞 Todo
Milestone
No milestone
Development

No branches or pull requests
3 participants
@aschmahmann @hacdias @Dreamacro