Beacon chain kubernetes based multinet cluster

[sinkingsugar]

Based on the work done here: https://github.com/eth2-clients/multinet it would be nice to open a discussion about setting up our own multinet cluster in order to execute a variety of tests in the presence of other clients.
The requirements are simple in terms of tech, just a standard kubernetes (+helm) cluster and access to it.
I personally have experience with gcloud but aws should be ok too (or azure).

The key question is how much can we spend in terms of CPU cores and memory (also storage but that's the cheap bit)?

Assuming that each node should receive at least 0.5 core and 2GB memory.
If we could have 64 cores we could run 128 nodes of 64 (could try 128 even) validators each.
That would make 4096 (8192) validators. Should we run more validators per node btw?
Any input is welcome as I don't have too much knowledge in that ratio.

cc @arnetheduck @dryajov @mratsim @stefantalpalaru

[mratsim]

Is 2GB enough at the moment? I thought we still had resource leaks?

Regarding validator count:

1. 4096 validators is the minimum to have committees achieve their target size of 111 validators per committee. Below that, the chain is not secure.
2. For launch-like tests, the minimum number of validators at genesis is 16384.
3. The maximum number of validators is 4.2M

To understand more about those figures I recommend reading the annotated specs of:

• Vitalik: https://github.com/ethereum/annotated-spec/blob/master/phase0/beacon-chain.md
• Benjamin: https://benjaminion.xyz/eth2-annotated-spec/phase0/beacon-chain/
• Danny: https://notes.ethereum.org/@djrtwo/Bkn3zpwxB

in particular the notes around MAX_COMMITTEES_PER_SLOT MAX_VALIDATORS_PER_COMMITTEE and MIN_GENESIS_ACTIVE_VALIDATOR_COUNT

Regarding the validators per node, 64 per node is 2048ETH staked, at a $300/ETH rate that's $0.6M ($1.2M), that seemed like a healthy amount of money per node.

[sinkingsugar]

Thanks @mratsim , well in terms of memory I never really saw much usage in my tests, for all clients, I suspect one of the reasons is we do not sync, all clients start together at genesis.
Also leaks happen and I saw leak issues in every clients (check teku discord for example), that is part of the test and with kubernetes if one pod goes over quota gets killed and will be restarted (also a good metric).
That said of course we can set 4gb as limit.. and dynamically will be allocated (in cloud resource numbers).

Nice info on validators.. I suppose for our own testnet we don't need to be as strict tho? Ideally an EF testnet should follow such parameters tho I guess.
I will check those specs thanks!