[spike] measure Holochain's performance under real-world circumstances using HoloFuel

[jost-s]

I think of it as a spike of exploring Holochain networking performance in general

motivation (why)

• measure Holochain's performance under real-world circumstances using HoloFuel as an example app

unclear

• are we going to mimic or actually use real-world circumstances?

[steveej]

conductor metrics:

• holochain's currently available influxdb metric
• data source for the Launcher's progress bar
• data source for the holochain-terminal CLI

choose frontend-agent execution framework and metrics:

• drive real UI with puppeteer?
  • consider: existing frontend metrics the real UI
• headless version of a holofuel user?
  • sketch a rust CLI that drives HoloFuel and collects metrics we want
• write the logic in a TryCP test

metrics collection

• ensure the influxdb metrics are enabled
• choose a central point that can store all the metrics for the test
• ensure that the test runs are distinguishable, if the central storage persists over test runs

---

how could this look like using TryCP

focus on: performance metrics

metrics collection
either

• set up a static central influxdb instance and configure holochain to report metrics there
• extend the conductor with a admin API call to get the influxdb metrics

conductor logs?

• does TryCP pipe conductor logs to its own stdout/stderr? this is not a requirement just want to know whether the logs are streamed to grafana on vanilla HPOS configurations

---

concrete steps

1. define an initial set of metrics that we want to measure
   • meta: the one's that are most significant for HoloFuel
   • Art: countersigning performance, writing an entry + link
   • countersigning
     • scheduling countersigning requests across many agents is challenging because they all lock their chain and then mutually reject their requests
     • staggered countersigning requests are likely the bigger challenge than how long a single countersigning session takes
   • look at the specification document for Wind-Tunnel, specifically the ones marked with * were regarded as relevant. @artbrock and @ThetaSinner to revisit this list together.
2. use existing TryCP/HoloFuel tests as a reference or starting point and add metric collection
   • measure countersigning metrics specifically
   • for comparison create a HoloFuel-agnostic reproduction of the countersigning measurement
3. ensure we have HoloPorts running the desired version of Holochain for the tests
   • figure out whether we need more specific configuration for the HPs

[steveej]

as discussed in a recent meeting a preliminary decision has been made to reuse TryCD in the short-term for this effort.

Holochain team concerns:

• what other services and infrastructure do we need for HoloFuel tests?
• what happens if we drive up the load on the conductor? this won't be explicitly tested if we were to re-use the UI automation (puppeteer) tests
• ideally we have a tight feedback loop between the UI and conductor testing efforts. specifically it could happen that the UI tests surface unexpected behavior that we can then encode as a more specific test on the conductor side.