minibank

bitcoin lightning node

For older models see History.

Table of contents

• About
• Hardware
• Operating System
• Heat
• Remove Extras
• Network
• Storage
• Software
• Monitoring
• Operations

About

Minibank is a HOWTO for building a Lightning Network node from scratch

• Limited set of features: bitcoind, lnd, tor, electrs
• Components are built from source code
• Mirror Raid with 2 external storage devices
• Development environment (CLI only)
• Monitoring (time series, dashboards, alerts)

Comparison to similar projects, it the order of difficulty of use:

1. Minibank - for those who like to cook from scratch, aims to increase security by limiting surface area / no add-ons and building everything from source (paranoid mode) - [CLI Setup]
2. RaspiBlitz - for technically interested / "geeky" users, building projects, trying stuff out, tinkering - [CLI Setup]
3. RoninDojo - no lightning, for privacy focused users, maximally focused on Samourai Wallet's Dojo - [CLI Setup]
4. nodl - for less technical users that still want improved security, e.g. merchants - [CLI Setup]
5. nodl Dojo - for privacy focused users, Lightning lnd is available as a 1-click install - [GUI Setup]
6. myNode - for non-technical users, works out of the box, web interface prioritized - [GUI Setup]
7. Embassy - for grandma to run Lighting and other self-hosted apps. An operating system by start9labs, they ship a device that you plug into home router and use a phone app to install self-hosted apps - [GUI Setup]
8. Umbrel - OS and friendly UI with similar goals as Embassy

Hardware

Model 4B :: Node at Home

Pi 4 Model B. Two high-speed SSDs for Raid-1 mirroring.

Total 317 USD as of Oct 2023

• Pi 4b kit (8GB RAM): Raspberry Pi 4 8GB Model B
• Power supply
• FLIRC Passive cooling case Flirc Raspberry Pi 4 Case
• Micro SD card 32G (for operating system) SanDisk-Extreme-microSD-UHS-I-Adapter
• Card Reader (for 1 time setup) Transcend-microSDHC-Reader-TS-RDF5K-Black
• SAMSUNG 1 TB SSD (for data; Raid-1 mirror): SAMSUNG T7 Portable SSD 1TB
• SanDisk 1 TB SSD (for data; Raid-1 mirror): SanDisk 1TB Extreme Portable External SSD
  • Pros: Different manufacturers so they don't fail at the same time. Cons: SanDisk failed first after I used this setup for several year. SanDisk company only tests Win and Mac. It does not show having cache on Linux so this many be a result of degraded performance. No indicator light. Shipping took much longer than SAMSUNG. Conclusion: in the future I might just get two SAMSUNG drives instead

Hardware with known issues:

• [Not sure this is still true. Now I know the data corruption is caused by UAS. So later in this article I disable UAS. This may actually fix the perceived Seagate issue] WARNING: The following Seagate device [may] caused data corruption when plugging into USB, other storage connected to UBS also got affected. DO NOT USE Seagate 500 GB SSD (for Raid-1 mirror): Seagate-Barracuda-500GB-External-Portable

Operating System

1. Download the image the Raspberry Pi Foundation’s official supported operating system Raspberry Pi OS (64-bit) Lite from official raspberrypi link
2. Uncompress the file: xz -d Downloads/2023-05-03-raspios-bullseye-arm64-lite.img.xz
3. Transfer the contents on the ".img" file to your SD card (I use dd, Raspberry Pi has installers and instructions for doing this from Linux, Mac, and Windows) Here is how I do it (avoiding using Raspberry Pi Installer):

sudo dmesg --follow  # first run the command then insert your SD card and verify that it's sdb
# Press Ctrl-c to exist out of dmesg or run in a different terminal / tab
sudo dd if=Downloads/2023-05-03-raspios-bullseye-arm64-lite.img of=/dev/sdb  # careful, sdb may be some other drive, check dmesg for correc block device

First Login (create your username in Raspberry Pi)

Now that you have the SD card, put it in. Don't connect to network. Connect monitor and keyboard. Power-up Pi.

On first boot, the Pi will ask you to create an account. Give it your special username and a strong password.

Once logged in, check to make sure you have a 64-bin linux OS, type:

arch

if you get "arch64" you're good to go (continue with this manual). Otherwise, this manual will not work (maybe you have have older hardware that's 32-bit only or you downloaded the wrong SD card image).

Heat

I recommend using FLIRC passive cooling:

• Pi temp under 50C
• No more worries of airflow obstruction
• Fan won't fail because there is not fan

If you still want to go with a fan, follow this howto. Tip: Connect the fan to GPIO pins with quiet cooling mode works best for me https://www.raspberrypi.org/forums/viewtopic.php?t=248918#p1519636

To measure the temperature, run:

while :; do /opt/vc/bin/vcgencmd measure_temp; sleep 1; done

Anything bellow 70C is good. The throttling kicks in at 80 C.

Remove Extras

Raspberry Pi comes with a lot of extras that we probably don't want running (exposing greater surface area to security issues, slowing down boot, pollution logs) so let's remove them like this:

sudo systemctl disable bluetooth.service
sudo systemctl disable avahi-daemon.service
sudo systemctl disable dphys-swapfile.service
sudo systemctl disable wpa_supplicant.service  # if your not using Wi-Fi
sudo systemctl disable triggerhappy.service
sudo systemctl disable triggerhappy.socket

# NOTE: the services will keep running until you do sudo systemctl stop or shutdown/reboot the pi

sudo apt-get purge bluez -y
sudo apt-get autoremove -y

sudo vi /boot/config.txt

1. press "G" to go to the end of file
2. press "A" and Enter to start typing
3. add the following

# Disable Bluetooth
dtoverlay=disable-bt

Network

Don't connect to network yet.

Connect via monitor and keyboard.

The following ~20 steps will need to be typed (not copy and pasted) because we are connected directly without network (not thru another computer). Setting up the firewall this way provides a higher level of security.

Tip: take a photo of these instructions and open it on your phone next to your keyboard

1. Setup no-incoming-connections firewall before connecting to the network! If you don't add a firewall you're at risk of getting hacked:

Run:

sudo mkdir /etc/iptables

2. Edit /etc/iptables/rules.v4 with your favourite command-line text editor, e.g. vi (if your not familiar with vi type "nano" instead of "vi" - nano is less advanced yet easier to use)

sudo vi /etc/iptables/rules.v4

3. nothing

4. Now type the following in the editor, save and exit.

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
COMMIT

5. Edit IPv6 rules

sudo vi /etc/iptables/rules.v6

6. Now type the following in the editor, save and exit.

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
COMMIT

7. Reboot:

sudo reboot

-- start of critical section (this section contains the step of connecting to the network, complete until the end of critical section or remove from network before rebooting) ---

8. Now run:

sudo iptables -F
sudo ip6tables -F
cat /etc/iptables/rules.v4 | sudo iptables-restore
cat /etc/iptables/rules.v6 | sudo ip6tables-restore

9. Carefully check for duplicate lines or extra lines in the output of:

sudo iptables-save  # should look exactly like the lines in step 4
sudo ip6tables-save  # should look exactly like the lines in step 6

• numbers at the end of the line may be different, those are your network statistics (if you accidentally connected to the network then the dropped/accepted packets will get counted)
• if you see entries that don't match what you head in step 4 and 6, go back and check that rules were typed in correctly

10. Connect Ethernet cable or (Optionally) setup Wi-Fi

NOTE: steps 9 (check firewall) and 10 (connect to network) need to be done back to back. E.g. if you reboot firewall rules will be lost and you'll need to go back to step 8 before connecting to network.

11. Update the system: sudo apt update && sudo apt upgrade;. If you don't upgrade you may get hacked. Some keyboards stop working after upgrade so be ready to find a different keyboard (DAS Keyboard works well, yet Pi needs to be rebooted while it's plugged in).
12. Make firewall persistent, if you don't persist firewall you may get hacked:

sudo apt install iptables-persistent  # when asked "Save currrent rules?" say "Yes" for both IPv4 and IPv6

sudo /etc/init.d/netfilter-persistent restart

sudo iptables-save  # show current v4 rules: check if this just like before
sudo iptables-save -6  # show current v6 rules: check that it is drop-everything 

12. Disconnect from network (Ethernet cable or Wi-Fi)
13. Reboot Pi

sudo reboot

14. Again check firewall after reboot:

sudo iptables-save  # show current v4 rules: check if this just like before
sudo ip6tables-save  # show current v6 rules: check that it is drop-everything 

15. Connect to network (Ethernet cable or Wi-Fi)

-- end of critical section ---

16. SSH over Tor

We first setup a management connection over Tor which will be slow. Later you will be able to add a fast management connection on your local network.

Tor:

• slow
• do not need to know any IPs
• accessible from anywhere on the internet with a Tor client

Local network:

• fast
• need to know local IP of the Raspberry Pi
• accessible only when you're connected to your local network

sudo apt install tor

17. Enable remote login over SSH. Run sudo raspi-config and select Interface Options -> SSH -> SSH server to be enabled

18. Test ssh locally (ssh to yourself while in Keyboard-Monitor mode):

ssh 127.0.0.1

19. Configure Tor

sudo vi /etc/tor/torrc

Find and uncomment lines with:

HiddenServiceDir
HiddenServicePort

and change

HiddenServicePort 80 127.0.0.1:8080

to

HiddenServicePort 22 127.0.0.1:22

and add another line

HiddenServiceVersion 3

20. Restart Tor

sudo systemctl restart tor@default.service

21. Reveal the hidden hostname

cat /var/lib/tor/hidden_service/hostname

write it down in a safe place

22. From your laptop run: torify ssh <PI_USER_NAME>@<TOR_HOSTNAME_HERE>.onion (replace <PI_USER_NAME> from "First Login" section, and <TOR_HOSTNAME_HERE> from step 21). When prompted enter your Raspberry Pi password from "First Login" section.

23. I recommend that you also setup a fast SSH over the local network (without Tor) you can do this by following https://github.com/alevchuk/minibank/blob/first/other-notes/no-tor-ssh.md

24. Follow Authorized Keys section

Authorized keys

So you don't have to type the password every time you need to log-in to the pi, setup authorized_keys.

On your laptop run:

ssh-keygen -f ~/.ssh/minibank_id_rsa

Hit enter twice when prompted for password.

Print you're new public key:

cat  ~/.ssh/minibank_id_rsa.pub

Copy the output to clipboard.

SSH into your Pi and run:

cat >> ~/.ssh/authorized_keys

paste the pubkey from clipboard, press Enter, and then press Ctrl-d.

Now run:

chmod o=,g= ~/.ssh/authorized_keys

Now log out, press Ctrl-d.

Now try logging back in like this:

torify ssh -i ~/.ssh/minibank_id_rsa <PI_USER_NAME>@<TOR_HOSTNAME_HERE>.onion

You should not need to re-enter password.

Once you can login without a password, disable login with password: edit /etc/ssh/sshd_config and find PasswordAuthentication. Uncommented and set to no. Restart ssh server sudo systemctl restart ssh

Finally, back on your laptop, add an alias

echo 'alias mb4="torify ssh -i ~/.ssh/minibank_id_rsa <PI_USER_NAME>@<TOR_HOSTNAME_HERE>.onion"' >> ~/.bash_profile
. ~/.bash_profile

Now type mb4 and that should log you into the Pi.

Storage

In this section will setup a Raid-1 Mirror from your two new SSD drives.

WARNING: any data in the SSD drives will be deleted.

Lookup storage device info

In this sections were going to look up the following for each of the SSDs:

• Block device name (e.g. "sda")
• idVendor (a hex number, e.g. "04e8")
• idProduct (a hex number, e.g. "61f5")

Steps:

1. Run sudo dmesg --follow
2. Unplug and re-plug one of the external SSD drives
3. Look for the block device name, starting with "sd" followed by a lowercase English letter. Write that down.
4. Look for idVendor and idProduct. Write those down
5. Repeat from step 2 for the other SSD

NOTE: it's important to label the storage devices with their idVendor and idProduct in case one of them fails and you'll need to know which one to replace. Block device names change depending in which order you plug in the drives. For that reason, do not write the block device name (the "sd" followed by a letter) on the label.

From now I will refer to the block device names as:

• YOUR_SSD_BLOCK_DEVICE_1 ("sd" followed by a letter)
• YOUR_VENDOR_ID_FOR_DEVICE_1 (hex number)
• YOUR_PRODUCT_ID_FOR_DEVICE_1 (hex number)
• and same for DEVICE_2

The relevant output of dmesg --follow would look like this:

[22341.090044] usb 2-1: new SuperSpeed Gen 1 USB device number 3 using xhci_hcd
[22341.118242] usb 2-1: New USB device found, idVendor=04e8, idProduct=61f5, bcdDevice= 1.00
[22341.118261] usb 2-1: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[22341.118273] usb 2-1: Product: Portable SSD T5
[22341.118284] usb 2-1: Manufacturer: Samsung
[22341.118296] usb 2-1: SerialNumber: 1234567DAFFD
[22341.136371] scsi host3: uas
[22341.139726] scsi 3:0:0:0: Direct-Access     Samsung  Portable SSD T5  0    PQ: 0 ANSI: 6
[22341.143092] sd 3:0:0:0: [sdd] 976773168 512-byte logical blocks: (500 GB/466 GiB)
[22341.143406] sd 3:0:0:0: [sdd] Write Protect is off
[22341.143420] sd 3:0:0:0: [sdd] Mode Sense: 43 00 00 00
[22341.144983] sd 3:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[22341.145925] sd 3:0:0:0: Attached scsi generic sg2 type 0
[22341.147078] sd 3:0:0:0: [sdd] Optimal transfer size 33553920 bytes
[22341.174323] sd 3:0:0:0: [sdd] Attached SCSI disk

• Notice that [sdd] is the block device name in the above example

Disable UAS

To prevent occasional freezing of your Pi, disable UAS. UAS (USB Attached SCSI) is a protocol that adds marginal performance improvement yet it's not reliable (at least for the current USB 3.0 hardware of the RaspberryPi). I suspect that the freezes get triggered by radio interference of USB 3.0 with 2.4 GHz Wi-Fi or Bluetooth. However, the old mass storage device protocol is resilient to this issue. So we simply need to follow the Raspberry Pi team's recommendation and disable UAS.

From my tests, the following were the advantages of disabling UAS:

• The freezes stopped.
  • By freeze I mean: getting /mnt/btrfs mount point failure with "uas_eh_abort_handler" for "CMD OUT" and "CMD IN" errors in dmesg
  • The failure does not cause data loss/corruption, yet brings down the whole system
  • A repro is to try to run btrfs balance start /mnt/btrfs and you'll get the failure within a minute
• For the first time I'm now able to run and complete sudo btrfs balance start -v --full-balance /mnt/btrfs/
• There is no significant performance degradations from disabling UAS

For more details on this issue see https://github.com/alevchuk/minibank/blob/first/incidents/i5-ssd-disconnect.md

To check if you have UAS enabled:

1. run dmesg | grep -v registered | grep uas soon after rebooting the Pi
2. You should see "scsi host0: uas"

To disable UAS:

1. From previous section you'll need the idVendor/idProduct pairs for both SSD devices
2. If you already setup /mnt/btrfs then stop all services using it and run umount /mnt/btrfs
3. Make a backup sudo cp /boot/cmdline.txt /cmdline.txt-old-backup
4. Edit the boot command by running sudo vi /boot/cmdline.txt
5. Add usb-storage.quirks=YOUR_VENDOR_ID_FOR_DEVICE_1:YOUR_PRODUCT_ID_FOR_DEVICE_1:u,YOUR_VENDOR_ID_FOR_DEVICE_2:YOUR_PRODUCT_ID_FOR_DEVICE_2:u in front of the command

• it's "usb-storage.quirks=" followed a comma separated list of "idVendor:idProduct:u"
• The part that you add needs to be followed by a space " " (e.g. usb-storage.quirks=0781:558c:u,04e8:61f5:u dwc_otg.lpm_enable=0 console=serial0,115200 ...)
• replace YOUR_...
• don't miss the ":u" at the end
• The whole line should look similar to this usb-storage.quirks=0781:558c:u,04e8:61f5:u dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=3acd0083-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

6. Reboot
7. Check dmesg | less. Search for "UAS", by typing "/UAS" and pressing "n" to go to next one. There should be 2 "UAS is ignored" messages for each USB device. They look like this:

[    1.617927] usb 2-1: UAS is ignored for this device, using usb-storage instead
[    1.618064] usb 2-1: UAS is ignored for this device, using usb-storage instead
...
[    2.049009] usb 2-2: UAS is ignored for this device, using usb-storage instead
[    2.049152] usb 2-2: UAS is ignored for this device, using usb-storage instead
...
[    2.624504] sd 0:0:0:0: [sda] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
...
[    3.071842] sd 1:0:0:0: [sdb] 976773168 512-byte logical blocks: (500 GB/466 GiB)

BTRFS RAID-1 Mirror

Install BTRFS progs:

sudo apt update
sudo apt install btrfs-progs

WARNING: any data in the SSD drives will be deleted. If you don't know what your doing, try running the command without --force first.

Create file-systems Bitcoin and LND nodes

sudo mkfs.btrfs --force /dev/YOUR_SSD_BLOCK_DEVICE_1_NAME_HERE
sudo mkfs.btrfs --force /dev/YOUR_SSD_BLOCK_DEVICE_2_NAME_HERE

Mount

sudo mkdir /mnt/btrfs
sudo mount /dev/YOUR_SSD_BLOCK_DEVICE_1_NAME_HERE /mnt/btrfs

Label it

sudo btrfs fi label /mnt/btrfs minibank4

Add it to systemd (in the past we used fstab):

sudo vi /dev/disk/by-label/minibank4

Paste:

[Unit]
Description=Minibank Directory (/mnt/btrfs)
DefaultDependencies=no
Conflicts=umount.target
After=network.target

[Mount]
What=/dev/disk/by-label/minibank4
Where=/mnt/btrfs
Type=btrfs
Options=defaults

[Install]
WantedBy=multi-user.target

and type ":wq" to save and quit vim

Now you can mount it like this (even if block device names change):

sudo systemctl stop mnt-btrfs.mount
sudo systemctl start mnt-btrfs.mount

To have the mount during boot time:

sudo systemctl enable mnt-btrfs.mount

Check BTRFS sizes like this (--si makes numbers compatible with numbers in parted):

sudo btrfs fi show --si

To setup Raid1 mirror you can do it at the time of running mkfs.btrfs or add a new device later, like this:

sudo btrfs dev add -f /dev/YOUR_SSD_BLOCK_DEVICE_2_NAME_HERE /mnt/btrfs

# check current Raid setup
sudo btrfs fi df /mnt/btrfs

# convert to Raid1 mirror
sudo btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/btrfs/

At any time, check for errors:

sudo btrfs dev stats /mnt/btrfs/

Software

Build Bitcoind

Follow instruction to build bitcoin: alevchuk/minibank/bitcoin

Start Bitcoind

Prerequisites:

• Build Bitcoind

Log-in as bitcoin

sudo su -l bitcoin

Note: bitcoinclients direcetory was created when bitcoin was built https://github.com/alevchuk/minibank/tree/first/bitcoin#bitcoin-cookie

Edit ~/.bitcoin/bitcoin.conf

server=1
disablewallet=0

# Bind to given address to listen for JSON-RPC connections. Use [host]:port notation for IPv6.
# This option can be specified multiple times (default: bind to all interfaces)
####rpcbind=<addr>:<port>
####rpcbind=192.168.0.17:8332
rpcbind=127.0.0.1:8332


# By default, only RPC connections from localhost are allowed.
# You can speficy multiple rpcallowip lines to allow different IPs
####rpcallowip=<addr>
####rpcallowip=192.168.0.17
rpcallowip=127.0.0.1

rpccookiefile=/home/bitcoin/bitcoinclients/cookie
startupnotify=chmod g+r /home/bitcoin/bitcoinclients/cookie



# Listen for RPC connections on this TCP port:
####rpcport=8332

onlynet=ipv4
zmqpubrawblock=tcp://0.0.0.0:29000
zmqpubrawtx=tcp://0.0.0.0:29001

prune=0  # No prune if you have 1 TB drive(s)
## prune=476000  # if you have 500 TB of storage space (raid-1 of 2 drives 500 TB each) you'll need to prune but you will need to disable txindex and blockfilterindex

txindex=1  # Maintain a full transaction index, LND uses this, otherwise there will be a lot of disk scans
blockfilterindex=1 #  takes a few GB of storage and helps to speed-up blockchain rescanning

## # tunning (not needed on the new Pi3 with SSDs)
## dbcache=200  # Maximum database cache size <n> MiB
## maxorphantx=10  # Keep at most <n> unconnectable transactions in memory (default: 100)
## maxmempool=50  # Keep the transaction memory pool below <n> megabytes
## maxconnections=20  # Maintain at most <n> connections to peers
## maxuploadtarget=50  # MiB/day for the community
## whitelist=download@127.0.0.1  # disable the limit for local p2p connections

# Detailed logging
####debug=bench
####debug=db
####debug=reindex
####debug=cmpctblock
####debug=coindb
####debug=leveldb

You'll need to set things like PASSWORD_1_HERE and PASSWORD_2_HERE with unique passwords. Generate random strings (of 30 alphanumeric characters) for each password. First character should be a letter. rpcuser should also look like a password. You can use: openssl rand -base64 32 | grep -o '[a-z0-9]' | xargs | tr -d ' ' to generate random strings.

Start

bitcoind

Once we are happy with how bitcoind works we can add it to systemd so it start up on it's own at boot time:

sudo vi /etc/systemd/system/minibank-bitcoin.service

Paste (this is a frok of https://github.com/bitcoin/bitcoin/blob/master/contrib/init/bitcoind.service which is different only in that it starts the service after the btrfs mount):

[Unit]
Description=Bitcoin daemon
Documentation=https://github.com/bitcoin/bitcoin/blob/master/doc/init.md

# https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/usr/bin/bitcoind -pid=/run/bitcoind/bitcoind.pid \
                            -conf=/etc/bitcoin/bitcoin.conf \
                            -datadir=/var/lib/bitcoind \
                            -startupnotify='systemd-notify --ready' \
                            -shutdownnotify='systemd-notify --stopping'

# Make sure the config directory is readable by the service user
PermissionsStartOnly=true
ExecStartPre=/bin/chgrp bitcoin /etc/bitcoin

# Process management
####################

Type=notify
NotifyAccess=all
PIDFile=/run/bitcoind/bitcoind.pid

Restart=on-failure
TimeoutStartSec=infinity
TimeoutStopSec=600

# Directory creation and permissions
####################################

# Run as bitcoin:bitcoin
User=bitcoin
Group=bitcoin

# /run/bitcoind
RuntimeDirectory=bitcoind
RuntimeDirectoryMode=0710

# /etc/bitcoin
ConfigurationDirectory=bitcoin
ConfigurationDirectoryMode=0710

# /var/lib/bitcoind
StateDirectory=bitcoind
StateDirectoryMode=0710

# Hardening measures
####################

# Provide a private /tmp and /var/tmp.
PrivateTmp=true

# Mount /usr, /boot/ and /etc read-only for the process.
ProtectSystem=full

## TODO: move out of /home
## Deny access to /home, /root and /run/user
# ProtectHome=true

# Disallow the process and all of its children to gain
# new privileges through execve().
NoNewPrivileges=true

# Use a new /dev namespace only populated with API pseudo devices
# such as /dev/null, /dev/zero and /dev/random.
PrivateDevices=true

# Deny the creation of writable and executable memory mappings.
MemoryDenyWriteExecute=true

[Install]
WantedBy=multi-user.target

Convenience stuff

In following sections you will:

• Name your Pi
• Set your Time-zone
• Expand bash history
• Customize Vim
• Customize GNU Screen

Name your Pi

Edit 2 files replacing "raspberrypi" with the name you came up with.

sudo vi /etc/hostname
sudo vi /etc/hosts  # edit the line with 127.0.0.1 adding a space and your new hostname at the end of that line

If you want to give your host a name without rebooting. No spaces or punctuation. (Put what you want instead of "minibank1".)

sudo hostname minibank1

You'll see the change after rebooting, run sudo reboot, re-SSH back in.

Time-zone

Run sudo raspi-config and select Localization Options --> Change Timezone --> Other --> UTC to make your system clock right. Check time by running date

I recommend setting your timezone to UTC because that's better for privacy and that's what bitcoind logs use.

Vim

More text editing features.

sudo apt install vim

This will replace "vi" as well.

Vi has a very inconvenient feature of making mouse actions not native to the OS that your SSHing from. E.g. to make middle-click paste work will paste, run:

sudo su -c "echo set mouse= >> /usr/share/vim/vim82/defaults.vim"

Make vim default (e.g. whenever git needs to invoke an editor):

echo 'export EDITOR="vim"' >> ~/.bashrc

Vim as Development Environment (for Python and stuff like that):

sudo su -c "cat <<EOF >> /etc/vim/vimrc
set paste
syntax on
set tabstop=4 shiftwidth=4 expandtab
EOF
"

Bash

To the end of the bash rc file add the following lines after running vi ~/.bashrc

# https://unix.stackexchange.com/a/48113/4058
export HISTCONTROL=ignoredups:erasedups  # no duplicate entries
export HISTSIZE=100000                   # big big history
export HISTFILESIZE=100000               # big big history
shopt -s histappend                      # append to history, don't overwrite it
# Save and reload the history after each command finishes
export PROMPT_COMMAND="history -a; history -c; history -r; $PROMPT_COMMAND"

Now do the same /etc/skel by running sudo vi /etc/skel/.bashrc

• so that every new accounts gets this

GNU Screen

sudo apt install screen

To the end of default screen config add the following lines by running sudo vi /etc/screenrc

startup_message off

escape ^Bb
defscrollback 6000
maptimeout 0
defhstatus "^EH"
hardstatus alwayslastline '%{= G}[ %{G} %h %{g} ][%= %{= w}%?%-Lw%?%{= B}%n*%f %t%?%{= B}(%u)%?%{= w}%+Lw%?%= %{= g}][%{B} %Y-%m-%d %{W}%c %{g}]'

After setting up Convenience stuff restart Bitcoind in Screen

Screen allows you to run things printing logs to your virtual screen while your away / logged out / diconnected.

If you rebooted, run sudo mount /mnt/btrfs/ to connect to external storage devices.

Now you can re-start bitcoin in screen, log-out, and it will continue running. To do that:

1. Find where bticoind is currently running, click on that, and press Ctrl-c
2. Wait for bitcoin to exit
3. Run screen
4. Start Bitcoin by running:

sudo su -l bitcoin
bitcoind

Now you don't have to worry about loosing SSH connection or logging out.

To deatch from screen press Ctrl-b and then press "d"

To re-attach, run screen -r

Install LND

Follow instructions under alevchuk/minibank/lightning

Start LND

Prerequisites:

• Start Bitcoin
• Install LND

Login as lightning:

sudo su -l lightning

Edit ~/.lnd/lnd.conf

[Application Options]
listen=0.0.0.0:9735
rpclisten=localhost:10009

[Bitcoin]
bitcoin.active=1
bitcoin.mainnet=1
bitcoin.node=bitcoind

[Bitcoind]
bitcoind.rpchost=localhost
bitcoind.rpccookie=/home/bitcoin/bitcoinclients/cookie
bitcoind.zmqpubrawblock=tcp://localhost:29000
bitcoind.zmqpubrawtx=tcp://localhost:29001

[tor]
; The port that Tor's exposed SOCKS5 proxy is listening on. Using Tor allows
; outbound-only connections (listening will be disabled) -- NOTE port must be
; between 1024 and 65535
tor.socks=9050
tor.active=1
tor.v3=1

When setting up bitcoind we explain how /home/bitcoin/bitcoinclients/cookie is created.

However, you will need to add the lightning user to bitcoinclients group like this:

sudo /usr/sbin/adduser lightning bitcoinclients

Enable bash completion for lncli:

sudo cp /home/lightning/gocode/src/github.com/lightningnetwork/lnd/contrib/lncli.bash-completion /etc/bash_completion.d/lncli

Start:

lnd

Create your Lightning wallet

Create a wallet

lncli create

This will create:

1. Your bitcoin private key stored on disk
2. A mnemonic phrase that you can backup to paper and use to restore the bitcoin funds
3. A password that will need to be entered every time LND starts

Fund your LND wallet and open channels

1. Create a one-time-use address and transfer some bitcoin to it

lncli newaddress np2wkh  # Nested SegWit address

2. Send the funds from an external bitcoin wallet.

3. Check that the funds arrived

lncli walletbalance  # will show unconfirmed balance within a few seconds. One confirmation will happen roughly every 10 minutes

4. Wait for 6 confirmations. About 1 hour.

5. Restart LND

6. Open Channels, follow: https://docs.lightning.engineering/lightning-network-tools/lnd/first-steps-with-lnd

7. Check activity in 1 hour:

lncli walletbalance
lncli channelbalance
lncli listchannels  | grep active | sort | uniq -c  # number of open channels

Install LND operations scripts

Change into Lighting account:

sudo su -l lightning

Checkout scripts and copy to lnd-e2e-testing:

git clone https://github.com/alevchuk/minibank.git
cp -r ~/minibank/scripts/* ~/lnd-e2e-testing/

• close_channel_custom.py
• pay_or_get_paid.py
• rebalance_channels.py
• treasury_report.py

Most of those scripts are short/readable and have internal documentation.

Treasury: Keep track of your total balance

Use treasury_report.py script

# One-time setup:
~/lnd-e2e-testing/treasury_report.py >> ~/balance_history.tab

# Track balance
while :; do echo; (cat ~/balance_history.tab; ~/lnd-e2e-testing/treasury_report.py ) | column -t; date; sleep 60; done

# Record balance
~/lnd-e2e-testing/treasury_report.py | grep -v Time  >> ~/balance_history.tab

As channels open and close you may see total balance go down but should it recover eventually. That's because LND overestimates the fees for the channel closing transactions.

Monitor channels

while :; do echo; date; ~/lnd-e2e-testing/rebalance_channels.py; sleep 1m; done

Example, output:

Mon 25 Mar 21:14:04 UTC 2019
Incative channels:
           chan_id      pubkey       local          remote      remote-pct      mini-id
--------------------------------------------------------------------------------

Active channels:
           chan_id      pubkey       local          remote      remote-pct      mini-id
--------------------------------------------------------------------------------
625373626745421824      0360f95      15789               3          33.33%      1
625357134040268800      02d58ee      15513               6          66.67%      0

Suggested new remote balance percentage --dst-pct 50.00

Monitor channels

while :; do echo; date; ~/lnd-e2e-testing/rebalance_channels.py; sleep 1m; done

Example, output:

Mon 25 Mar 21:14:04 UTC 2019
Incative channels:
           chan_id      pubkey       local          remote      remote-pct      mini-id
--------------------------------------------------------------------------------

Active channels:
           chan_id      pubkey       local          remote      remote-pct      mini-id
--------------------------------------------------------------------------------
625373626745421824      0360f95      15789               3          33.33%      1
625357134040268800      02d58ee      15513               6          66.67%      0

Suggested new remote balance percentage --dst-pct 50.00

Monitoring

Prometheus exporters

Prerequisites:

• Storage
• Build Go

Citations:

• This section is based on github.com/prometheus

Install on all nodes.

sudo adduser --disabled-password monitoring

cd /mnt/btrfs

sudo mkdir ./monitoring
sudo mkdir ./monitoring/gocode
sudo mkdir ./monitoring/src

sudo chown -R monitoring ./monitoring

Allow everyone to read lighting source code:

sudo chmod o+rx /mnt/btrfs/lightning         # x means "execute (or search for directories)"
sudo chmod -R o+rX /mnt/btrfs/lightning/src  # X means "execute/search only if the file is a directory or already has execute permission for some user"

Loging as "monitoring" user

sudo su -l monitoring
ln -s /mnt/btrfs/monitoring/src
ln -s /mnt/btrfs/monitoring/gocode
ln -s /mnt/btrfs/lightning/src/ ~/src_readonly

Add exports to ~/.profile by running:

echo 'export GOROOT=~/src_readonly/go' >> ~/.profile
echo 'export GOPATH=~/gocode' >> ~/.profile
echo 'export PATH=$GOROOT/bin:$GOPATH/bin:$PATH' >> ~/.profile
echo 'export PATH=$HOME/bin/bin:$PATH' >> ~/.profile

Load new profile

. ~/.profile

Node Exporter: Host metrics

Node Exporter is used to export system metrics to Prometheus

go install github.com/prometheus/node_exporter@latest

cd ${GOPATH-$HOME/go}/src/github.com/prometheus/node_exporter

git pull

make

Run node_exporter

${GOPATH-$HOME/go}/src/github.com/prometheus/node_exporter/node_exporter --no-collector.mdadm --no-collector.infiniband

Once confimed that node_exporter start up, kill it (by pressing Ctrl-c) and configure systemd:

sudo vi /etc/systemd/system/minibank-mon.service

Press "i" and paste:

[Unit]
Description=Minibank Node Exporter for Prometheus monitoring
After=mnt-btrfs.mount

[Service]
ExecStart=/home/monitoring/gocode/src/github.com/prometheus/node_exporter/node_exporter
User=monitoring
Group=monitoring
ExecReload=/bin/true
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=simple
RuntimeDirectory=minibank-mon
RuntimeDirectoryMode=0755

[Install]
WantedBy=default.target

type ":wq" to save and quit out of Vim

Test starting and stopping:

sudo systemctl start minibank-mon.service
sudo systemctl status minibank-mon.service

sudo systemctl stop minibank-mon.service
sudo systemctl status minibank-mon.service

Configure to start at boot time:

sudo systemctl enable minibank-mon.service

Test:

sudo reboot
# after system comes back:
sudo systemctl status minibank-mon.service

Prometheus

Requierments

• Lightning (because we re-use Go build)

If you have multiple nodes, install this on the base station to pull in all metrics into a single place.

Setup accounts:

sudo adduser --disabled-password prometheus
sudo mkdir /mnt/btrfs/prometheus
sudo mkdir /mnt/btrfs/prometheus/gocode
sudo mkdir /mnt/btrfs/prometheus/data
sudo mkdir /mnt/btrfs/prometheus/src
sudo mkdir /mnt/btrfs/prometheus/bin

sudo chown -R prometheus /mnt/btrfs/prometheus/

sudo su -l prometheus
ln -s /mnt/btrfs/lightning/src ~/lightning_readonly # symlink to read-only go installation
ln -s /mnt/btrfs/prometheus/src ~/src
ln -s /mnt/btrfs/prometheus/bin ~/bin
ln -s /mnt/btrfs/prometheus/gocode ~/gocode

Build node.js (includes NPM)

• unfortunatly, the make step here will take many hours (a whole day)
  • installing node.js some other way maybe possible yet comes with other issues (e.g. the official debian build is out-dated, likely incompatible and has security vulnerabilities)

git clone https://github.com/nodejs/node.git ~/src/node
cd ~/src/node
git fetch
git checkout $(git tag | grep ^v | sort -V | tail -n1)  # lastes release
./configure --prefix $HOME/bin
make
make install

Enable Go. Add exports to ~/.profile by running:

echo 'export GOROOT=~/src_readonly/go' >> ~/.profile
echo 'export GOPATH=~/gocode' >> ~/.profile
echo 'export PATH=$GOROOT/bin:$GOPATH/bin:$PATH' >> ~/.profile
echo 'export PATH=$HOME/bin/bin:$PATH' >> ~/.profile

Load new profile

. ~/.profile

Install Yarn:

npm install -g yarn

Fetch source code and build prometheus:

go get github.com/prometheus/prometheus/cmd/...
cd /home/prometheus/gocode/src/github.com/prometheus/prometheus/
make build

Configure:

ln -s /mnt/btrfs/prometheus/data ~/.prometheus
vi ~/.prometheus/prometheus.yml

Configure to collect from node exporters from all managed hosts, including self, e.g.:

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

scrape_configs:
- job_name: 'node3'
  static_configs:
  - targets: ['bl3:9100', 'bl4:9100', 'bl5:9100', 'bl3:8334', 'bl4:8334', 'bl5:8334']

Run prometheus:

cd ~prometheus/.prometheus && ~/gocode/src/github.com/prometheus/prometheus/prometheus --storage.tsdb.retention 5y

Grafana

Grafana is a monitoring/analytics web interface.

Warning: This is a web server, so be especially careful with security. E.g. you may want to run grafana on a different Pi or inside a container (e.g. Docker)

To install and run Grafana follow alevchuk/minibank/grafana

Operations

• To update OS and utilities with latest security patches, regularly run: sudo apt update && sudo apt upgrade -y
• To replace a bad OS SD Card, run thru this manual again but skip building software (i.e. make commands) and don't reformat your external SSD drives (they contain your data, configuration, and software that you've built in this manual)
• To replace a bad external SSD drive BTRFS Raid wiki