Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rpc: provide per message stats for global traffic via new RPC 'getnetmsgstats' #29418

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

vasild
Copy link
Contributor

@vasild vasild commented Feb 11, 2024

Introduce a new RPC, getnetmsgstats to retrieve traffic bytes and count of messages possibly broken down by:

  • direction (sent or received)
  • network (ipv4, tor, etc)
  • connection type (outbound-full-relay, block-relay-only, etc)
  • message type (verack, ping, etc)

Implements: #26337 Add per message stats to getnettotals rpc


Examples:

bitcoin-cli getnetmsgstats '["direction", "network", "connection_type", "message_type"]'
{
  "bytes": 5392756,
  "count": 32683
}
bitcoin-cli getnetmsgstats '["direction", "network", "connection_type"]'
{
  "sendcmpct": {
    "bytes": 975,
    "count": 31
  },
  "sendaddrv2": {
    "bytes": 816,
    "count": 28
  },
  "headers": {
    "bytes": 1786,
    "count": 28
  },
  "feefilter": {
    "bytes": 816,
    "count": 27
  },
  "version": {
    "bytes": 4843,
    "count": 37
  },
  "ping": {
    "bytes": 1104,
    "count": 36
  },
  "getaddr": {
    "bytes": 384,
    "count": 13
  },
  "block": {
    "bytes": 5128467,
    "count": 20365
  },
  "getheaders": {
    "bytes": 29436,
    "count": 28
  },
  "pong": {
    "bytes": 1040,
    "count": 34
  },
  "getdata": {
    "bytes": 1218925,
    "count": 20008
  },
  "sendheaders": {
    "bytes": 447,
    "count": 16
  },
  "wtxidrelay": {
    "bytes": 816,
    "count": 28
  },
  "addrv2": {
    "bytes": 301161,
    "count": 25
  },
  "verack": {
    "bytes": 816,
    "count": 28
  }
}
bitcoin-cli getnetmsgstats '["direction", "connection_type", "message_type"]'
{
  "i2p": {
    "bytes": 1311861,
    "count": 8226
  },
  "onion": {
    "bytes": 3932760,
    "count": 24272
  },
  "ipv6": {
    "bytes": 631265,
    "count": 3362
  },
  "ipv4": {
    "bytes": 1300106,
    "count": 7773
  }
}
bitcoin-cli getnetmsgstats
{
  "recv": {
    "i2p": {
      "manual": {
        "sendcmpct": {
          "bytes": 90,
          "count": 3
        },
        "sendaddrv2": {
          "bytes": 99,
          "count": 3
        },
        "headers": {
          "bytes": 309,
          "count": 3
        },
        "feefilter": {
          "bytes": 87,
          "count": 3
        },
        "version": {
          "bytes": 408,
          "count": 3
        },
        "ping": {
          "bytes": 87,
          "count": 3
        },
        "block": {
          "bytes": 432950,
          "count": 1794
        },
        "getheaders": {
          "bytes": 3150,
          "count": 3
        },
        "pong": {
          "bytes": 87,
          "count": 3
        },
        "wtxidrelay": {
          "bytes": 99,
          "count": 3
        },
        "addrv2": {
          "bytes": 68508,
          "count": 3
        },
        "verack": {
          "bytes": 99,
          "count": 3
        }
      }
    },
    "onion": {
      "block-relay-only": {
        "sendcmpct": {
          "bytes": 66,
          "count": 2
        },
        "sendaddrv2": {
          "bytes": 24,
          "count": 1
        },
        "headers": {
          "bytes": 106,
          "count": 1
        },
        "feefilter": {
          "bytes": 32,
          "count": 1
        },
        "version": {
          "bytes": 126,
          "count": 1
        },
        "ping": {
          "bytes": 64,
          "count": 2
        },
        "block": {
          "bytes": 478479,
          "count": 1896
        },
        "getheaders": {
          "bytes": 1053,
          "count": 1
        },
        "pong": {
          "bytes": 64,
          "count": 2
        },
        "sendheaders": {
          "bytes": 24,
          "count": 1
        },
        "wtxidrelay": {
          "bytes": 24,
          "count": 1
        },
        "verack": {
          "bytes": 24,
          "count": 1
        }
      },
      "outbound-full-relay": {
        "sendcmpct": {
          "bytes": 195,
          "count": 6
        },
        "sendaddrv2": {
          "bytes": 129,
          "count": 5
        },
        "headers": {
          "bytes": 527,
          "count": 5
        },
        "feefilter": {
          "bytes": 157,
          "count": 5
        },
        "version": {
          "bytes": 639,
          "count": 5
        },
        "ping": {
          "bytes": 189,
          "count": 6
        },
        "block": {
          "bytes": 1465688,
          "count": 5957
        },
        "getheaders": {
          "bytes": 5262,
          "count": 5
        },
        "pong": {
          "bytes": 189,
          "count": 6
        },
        "sendheaders": {
          "bytes": 24,
          "count": 1
        },
        "wtxidrelay": {
          "bytes": 129,
          "count": 5
        },
        "addrv2": {
          "bytes": 120693,
          "count": 9
        },
        "verack": {
          "bytes": 129,
          "count": 5
        }
      }
    },
    "ipv6": {
      "manual": {
        "sendcmpct": {
          "bytes": 120,
          "count": 4
        },
        "sendaddrv2": {
          "bytes": 132,
          "count": 4
        },
        "headers": {
          "bytes": 412,
          "count": 4
        },
        "feefilter": {
          "bytes": 116,
          "count": 4
        },
        "version": {
          "bytes": 544,
          "count": 4
        },
        "ping": {
          "bytes": 116,
          "count": 4
        },
        "block": {
          "bytes": 138074,
          "count": 583
        },
        "getheaders": {
          "bytes": 4200,
          "count": 4
        },
        "pong": {
          "bytes": 116,
          "count": 4
        },
        "wtxidrelay": {
          "bytes": 132,
          "count": 4
        },
        "addrv2": {
          "bytes": 93436,
          "count": 4
        },
        "verack": {
          "bytes": 132,
          "count": 4
        }
      },
      "outbound-full-relay": {
        "version": {
          "bytes": 126,
          "count": 1
        }
      }
    },
    "ipv4": {
      "outbound-full-relay": {
        "sendcmpct": {
          "bytes": 66,
          "count": 2
        },
        "sendaddrv2": {
          "bytes": 24,
          "count": 1
        },
        "headers": {
          "bytes": 106,
          "count": 1
        },
        "feefilter": {
          "bytes": 32,
          "count": 1
        },
        "version": {
          "bytes": 252,
          "count": 2
        },
        "ping": {
          "bytes": 32,
          "count": 1
        },
        "block": {
          "bytes": 313988,
          "count": 1255
        },
        "getheaders": {
          "bytes": 1053,
          "count": 1
        },
        "pong": {
          "bytes": 32,
          "count": 1
        },
        "sendheaders": {
          "bytes": 24,
          "count": 1
        },
        "wtxidrelay": {
          "bytes": 24,
          "count": 1
        },
        "addrv2": {
          "bytes": 40,
          "count": 1
        },
        "verack": {
          "bytes": 24,
          "count": 1
        }
      }
    }
  },
  "sent": {
    "i2p": {
      "manual": {
        "sendcmpct": {
          "bytes": 90,
          "count": 3
        },
        "sendaddrv2": {
          "bytes": 99,
          "count": 3
        },
        "headers": {
          "bytes": 66,
          "count": 3
        },
        "feefilter": {
          "bytes": 87,
          "count": 3
        },
        "version": {
          "bytes": 544,
          "count": 4
        },
        "ping": {
          "bytes": 87,
          "count": 3
        },
        "getaddr": {
          "bytes": 99,
          "count": 3
        },
        "getheaders": {
          "bytes": 3150,
          "count": 3
        },
        "pong": {
          "bytes": 87,
          "count": 3
        },
        "getdata": {
          "bytes": 105382,
          "count": 1789
        },
        "sendheaders": {
          "bytes": 99,
          "count": 3
        },
        "wtxidrelay": {
          "bytes": 99,
          "count": 3
        },
        "verack": {
          "bytes": 99,
          "count": 3
        }
      }
    },
    "onion": {
      "block-relay-only": {
        "sendcmpct": {
          "bytes": 33,
          "count": 1
        },
        "sendaddrv2": {
          "bytes": 24,
          "count": 1
        },
        "headers": {
          "bytes": 25,
          "count": 1
        },
        "version": {
          "bytes": 127,
          "count": 1
        },
        "ping": {
          "bytes": 64,
          "count": 2
        },
        "getheaders": {
          "bytes": 1053,
          "count": 1
        },
        "pong": {
          "bytes": 64,
          "count": 2
        },
        "getdata": {
          "bytes": 115932,
          "count": 1884
        },
        "sendheaders": {
          "bytes": 24,
          "count": 1
        },
        "wtxidrelay": {
          "bytes": 24,
          "count": 1
        },
        "verack": {
          "bytes": 24,
          "count": 1
        }
      },
      "outbound-full-relay": {
        "sendcmpct": {
          "bytes": 162,
          "count": 5
        },
        "sendaddrv2": {
          "bytes": 129,
          "count": 5
        },
        "headers": {
          "bytes": 122,
          "count": 5
        },
        "feefilter": {
          "bytes": 157,
          "count": 5
        },
        "version": {
          "bytes": 771,
          "count": 6
        },
        "ping": {
          "bytes": 221,
          "count": 7
        },
        "getaddr": {
          "bytes": 129,
          "count": 5
        },
        "getheaders": {
          "bytes": 5262,
          "count": 5
        },
        "pong": {
          "bytes": 189,
          "count": 6
        },
        "getdata": {
          "bytes": 363478,
          "count": 5932
        },
        "sendheaders": {
          "bytes": 129,
          "count": 5
        },
        "wtxidrelay": {
          "bytes": 129,
          "count": 5
        },
        "addrv2": {
          "bytes": 231,
          "count": 4
        },
        "verack": {
          "bytes": 129,
          "count": 5
        }
      }
    },
    "ipv6": {
      "manual": {
        "sendcmpct": {
          "bytes": 120,
          "count": 4
        },
        "sendaddrv2": {
          "bytes": 132,
          "count": 4
        },
        "headers": {
          "bytes": 88,
          "count": 4
        },
        "feefilter": {
          "bytes": 116,
          "count": 4
        },
        "version": {
          "bytes": 544,
          "count": 4
        },
        "ping": {
          "bytes": 116,
          "count": 4
        },
        "getaddr": {
          "bytes": 132,
          "count": 4
        },
        "getheaders": {
          "bytes": 4200,
          "count": 4
        },
        "pong": {
          "bytes": 116,
          "count": 4
        },
        "getdata": {
          "bytes": 35492,
          "count": 584
        },
        "sendheaders": {
          "bytes": 99,
          "count": 3
        },
        "wtxidrelay": {
          "bytes": 132,
          "count": 4
        },
        "verack": {
          "bytes": 132,
          "count": 4
        }
      },
      "outbound-full-relay": {
        "version": {
          "bytes": 254,
          "count": 2
        }
      }
    },
    "ipv4": {
      "outbound-full-relay": {
        "sendcmpct": {
          "bytes": 33,
          "count": 1
        },
        "sendaddrv2": {
          "bytes": 24,
          "count": 1
        },
        "headers": {
          "bytes": 25,
          "count": 1
        },
        "feefilter": {
          "bytes": 32,
          "count": 1
        },
        "version": {
          "bytes": 508,
          "count": 4
        },
        "ping": {
          "bytes": 32,
          "count": 1
        },
        "getaddr": {
          "bytes": 24,
          "count": 1
        },
        "getheaders": {
          "bytes": 1053,
          "count": 1
        },
        "pong": {
          "bytes": 32,
          "count": 1
        },
        "getdata": {
          "bytes": 76531,
          "count": 1231
        },
        "sendheaders": {
          "bytes": 24,
          "count": 1
        },
        "wtxidrelay": {
          "bytes": 24,
          "count": 1
        },
        "verack": {
          "bytes": 24,
          "count": 1
        }
      }
    }
  }
}

Previous incarnations of this:
#27534 rpc: add 'getnetmsgstats', new rpc to view network message statistics (Thank you, @satsie!)
#28926 rpc: add 'getnetmsgstats' RPC (Thank you, @willcl-ark!)

@DrahtBot
Copy link
Contributor

DrahtBot commented Feb 11, 2024

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type Reviewers
Concept ACK epiccurious

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #29256 (Improve new LogDebug/Trace/Info/Warning/Error Macros by ryanofsky)
  • #28521 (net: additional disconnect logging by Sjors)
  • #19463 (Prune locks by luke-jr)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@DrahtBot
Copy link
Contributor

🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the
documentation.

Possibly this is due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.

Leave a comment here, if you need help tracking down a confusing failure.

Debug: https://github.com/bitcoin/bitcoin/runs/21448984768

@epiccurious
Copy link

Concept ACK 91c90d7.

@DrahtBot
Copy link
Contributor

DrahtBot commented Apr 4, 2024

Needs rebase if still relevant

@achow101 achow101 requested a review from mzumsande April 9, 2024 15:52
@0xB10C
Copy link
Contributor

0xB10C commented Apr 9, 2024

just fyi as this came up: You could count up message stats with the inbound and outbound message tracepoints. However, this does only gives you stats from the points on where you start hooking into the tracepoints, might not be as user friendly (RPC is easier to use than the tracepoints), and the tracepoints currently only work on Linux.

While I personally don't need it, I see why some node operators need traffic statistics. E.g. How much of my (limited monthly) traffic is used up by inbound connections?

@DrahtBot DrahtBot marked this pull request as draft April 23, 2024 06:28
@DrahtBot
Copy link
Contributor

Converted to draft due to failing CI and inactivity

@vasild
Copy link
Contributor Author

vasild commented May 9, 2024

91c90d7595...6bfda84b92: rebase, pick changes from #29421 and adjust the newly added test.

@DrahtBot DrahtBot removed the CI failed label May 9, 2024
achow101 added a commit that referenced this pull request May 21, 2024
…e constant

b3efb48 protocol: make message types constexpr (Vasil Dimov)
2fa9de0 net: make the list of known message types a compile time constant (Vasil Dimov)

Pull request description:

  Turn the `std::vector` to `std::array` because it is cheaper and allows us to have the number of the messages as a compile time constant: `ALL_NET_MESSAGE_TYPES.size()` which can be used in future code to build other `std::array`s with that size.

  ---

  This change is part of #29418 but it makes sense on its own and would be good to have it, regardless of the fate of #29418. Also, if this is merged, that would reduce the size of #29418, thus the current standalone PR.

ACKs for top commit:
  achow101:
    ACK b3efb48
  jonatack:
    ACK b3efb48
  maflcko:
    utACK b3efb48 🎊
  willcl-ark:
    ACK b3efb48

Tree-SHA512: 6d3860c138c64514ebab13d97ea67893e2d346dfac30a48c3d9bc769a1970407375ea4170afdb522411ced306a14a9af4eede99e964d1fb1ea3efff5d5eb57af
vasild and others added 3 commits May 23, 2024 11:47
Before this change only per-peer stats were gathered. They
are lost when the peer disconnects.

So, collect the traffic stats globally in `CConnman`, broken down by
* direction (sent or received, (2))
* network of the peer (IPv4, IPv6, Tor, I2P, CJDNS (5))
* connection type (inbound, full outbound, feeler, etc, (6))
* message type (verack, ping, etc, (36))
Introduce a new RPC `getnetmsgstats` that renders the global traffic
stats from `CConnman`.

Co-authored-by: Vasil Dimov <vd@FreeBSD.org>
Aggregation can be by either one of direction, network, connection type
or message type. For example if the following

```
{
    "ipv4": { "ping": 3 },
    "ipv6": { "ping": 4 }
}
```

is aggregated by network, then the result will be

```
{
    "ping": 7
}
```
@vasild
Copy link
Contributor Author

vasild commented May 23, 2024

6bfda84b92...b5189f543c: rebase due to conflicts (part of this PR was merged via #29421)

@vasild vasild marked this pull request as ready for review May 23, 2024 09:51
Co-authored-by: Vasil Dimov <vd@FreeBSD.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants