Upgrade storage for mainnet fleet

[jakubgs]

It's about time we increase the storage available for both Docker containers(Geth) and Systemd services(Beacon Nodes):

jakubgs@linux-01.ih-eu-mda1.nimbus.mainnet:~ % df -h / /docker /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       366G   62G  286G  18% /
/dev/sdc        1.5T  1.2T  174G  88% /docker
/dev/sdb        1.5T  1.2T  259G  82% /data

The current layout involves single logical volume per single physical volume(SSD) configured in the controller.

The migration to RAID0 logical volumes using two SSDs using a HPE Smart Array utility is documented here:
https://docs.infra.status.im/general/hp_smart_array_raid.html

The steps for the migration of each host will look like this:

1. Request attachment of a temporary 1.5 TB(or bigger) SSDs on the host for migrations.
2. Migrate /data files to temporary migration SSD.
3. Destroy /data logical volume and re-create it with two physical volumes(SSDs) as one RAID0 logical volume.
4. Migrate from temporary SSD back to new RAID0 /data volume.
5. Repeat steps 2, 3, & 4 for the /docker volume.
6. Inform support they can move the migration SSD to another host, and repeat for that host.

I would recommend creating a single support ticket to order 2 extra SSDs of the same type for all nimbus.mainnet hosts, and then manage migration of each host in the comments of that ticket.

[jakubgs]

You can find more example of me using ssacli to configure volumes here:

• Research alternatives to Hetzner for testnet hosting #132

Just need to click the Load more... button:

[yakimant]

Ticket Created #351756

[yakimant]

We are able to connect today only 9 SSD disks of 1.6TB capacity (4 servers). If you need all with 1.6TB capacity, then it will be possible to connect the remaining in the next 2 weeks or if you want, we may connect a 3.84TB drive on each of the remaining servers.

The cost for the single additional disk is 20 euro per 1.6TB SSD drive.

Asked about 4TB price and 3TB (if they have)

[yakimant]

4TB will cost twice of that, will go ahead.

Pros:

• No RAID0 needed for that disk
• No need for temporary drive

Cons:

• RAID0 would provide better speed, but we don't need it

[yakimant]

ssacli installation:

echo "deb http://downloads.linux.hpe.com/SDR/repo/mcp jammy/current non-free" | sudo tee /etc/apt/sources.list.d/hp-mcp.list
wget -qO- http://downloads.linux.hpe.com/SDR/hpPublicKey1024.pub | sudo tee -a /etc/apt/trusted.gpg.d/hp-mcp.asc
wget -qO- http://downloads.linux.hpe.com/SDR/hpPublicKey2048.pub | sudo tee -a /etc/apt/trusted.gpg.d/hp-mcp.asc
wget -qO- http://downloads.linux.hpe.com/SDR/hpPublicKey2048_key1.pub | sudo tee -a /etc/apt/trusted.gpg.d/hp-mcp.asc
wget -qO- http://downloads.linux.hpe.com/SDR/hpePublicKey2048_key1.pub | sudo tee -a /etc/apt/trusted.gpg.d/hp-mcp.asc
apt update
apt install ssacli

[yakimant]

Disk are installed:

❯ ansible nimbus-mainnet-metal -i ansible/inventory/test -a 'sudo ssacli ctrl slot=0 pd allunassigned show'
linux-06.ih-eu-mda1.nimbus.mainnet | CHANGED | rc=0 >>

Smart Array P420i in Slot 0 (Embedded)

   Unassigned

      physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SAS SSD, 3.8 TB, OK)
linux-07.ih-eu-mda1.nimbus.mainnet | CHANGED | rc=0 >>

Smart Array P420i in Slot 0 (Embedded)

   Unassigned

      physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SAS SSD, 3.8 TB, OK)
linux-02.ih-eu-mda1.nimbus.mainnet | CHANGED | rc=0 >>

Smart Array P420i in Slot 0 (Embedded)

   Unassigned

      physicaldrive 2I:2:6 (port 2I:box 2:bay 6, SAS SSD, 3.8 TB, OK)
linux-04.ih-eu-mda1.nimbus.mainnet | FAILED | rc=1 >>

Error: The controller identified by "slot=0" was not detected.non-zero return code
linux-01.ih-eu-mda1.nimbus.mainnet | CHANGED | rc=0 >>

Smart Array P420i in Slot 0 (Embedded)

   Unassigned

      physicaldrive 2I:2:6 (port 2I:box 2:bay 6, SAS SSD, 3.8 TB, OK)
linux-03.ih-eu-mda1.nimbus.mainnet | CHANGED | rc=0 >>

Smart Array P420i in Slot 0 (Embedded)

   Unassigned

      physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS SSD, 3.8 TB, OK)
linux-05.ih-eu-mda1.nimbus.mainnet | CHANGED | rc=0 >>

Smart Array P420i in Slot 0 (Embedded)

   Unassigned

      physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SAS SSD, 3.8 TB, OK)

linux-04 has a different slot:

❯ sudo ssacli ctrl slot=1 pd allunassigned show

Smart Array P222 in Slot 1

   Unassigned

      physicaldrive 2I:1:3 (port 2I:box 1:bay 3, SAS SSD, 3.8 TB, OK)