MFS/IPFS Pin performance

[RubenKelevra]

Additional information:

• OS: Linux
• IPFS Cluster version: 1.0.1
• Installation method: dist.ipfs.io
• IPFS version:

go-ipfs version: 0.14.0-dev-a72753bad-dirty
Repo version: 12
System version: amd64/linux
Golang version: go1.18.2

Describe the bug:

I'm using the MFS to construct my file/folder structure, which then gets added to the cluster-pinset.

I'm currently dealing with performance issues, as the pinning operations after having the full file/folder-strucuture already inside the MFS takes an s**tload of IO.

It kinda needs to be pinned I guess, as I will modify the files/folders before pinning the next ones (and otherwise the GC run might clean things up inadvertently), but isn't there any way to tell ipfs that this stuff is already fully locally available and not traverse the full ~500GB for every change?

Edit: Ah and yes, I already do ipfs-cluster-ctl pin update --no-status $oldcid $newcid followed by an ipfs-cluster-ctl pin rm --no-status $oldcid.

[RubenKelevra]

This is how this looks like in iotop:

And these are some stats from the machine:

[RubenKelevra]

Here's the IPFS bugreport for this:

ipfs/kubo#8988

[RubenKelevra]

And just for context, that's my setting for pinning concurrences on odin:

[ipfs@odin ~]$ cat .ipfs-cluster/service.json | grep pin_tracker -A5
  "pin_tracker": {
    "stateless": {
      "max_pin_queue_size": 1000000,
      "concurrent_pins": 1
    }
  },

I now tried to reduce the concurrent bitswap tasks, hope this helps (without hurting the performance too much):

[ipfs@odin ~]$ ipfs config show | grep Internal -A7
  "Internal": {
    "Bitswap": {
      "EngineBlockstoreWorkerCount": 4,
      "EngineTaskWorkerCount": 2,
      "TaskWorkerCount": 4
    },
    "UnixFSShardingSizeThreshold": "1MB"
  },

A nice workaround would actually be to not pin the folder at all on this node, as I run the GC on demand anyway, when I don't modify the MFS. So all data should be always available until the new CID is put into the cluster-pinset.

Blocker for this is that I cannot exclude the local peer (or all trusted peers for that matter) from beeing part of the pinset. Blocker is tracked here: #997

@hsanjuan maybe you can have another look at this? I think it's fairly easy to implement #997, as it's just a filter for selecting cluster-members as part of the pinset if not -1, -1 replication is set.