This document was written in response to Challenge 005 of the Stakewars III (an event regarding the NEAR Protocol, which is a Blockchain project / ecosystem)
I started by registering a new hosting account from one of the well known providers. I already have a node running on Hetzner, for the purpose of this guide, I create another one with Digital Ocean.
Note that, when creating a brand new Digital Ocean account, with a new payment method, we're entitled to $100 credit to kick start our project. Nice, itsn't it?
Each server instance is called a "droplet" in Digital Ocean terminology, so here I'm creating a new droplet. Please refer to the recommended hardware guide and choose the server specs that are best to run a NEAR network validator node.
Note that, in the screenshot above I chose the "$48/month" option with "8 GB / 4 CPU", but later on I realized it was not enough. Probably the cloud vCPU is not as powerful as a real CPU? Either way, if possible, you're recommended to choose a spec higher than shown in the screenshot above.
During the server provisioning process, I deliberately set up an ssh key authorization, so that later I could securely log into my server without any password prompt. This step is optional, but I always prefer to have it for all my servers.
As shown above, I simply generated a new key pair on my local computer (local worksation), then add the public key to the server. Digital Ocean provides an interface for us to do this. If you use other cloud providers, there should be similar interfaces for this process.
Once all setup is filled in, I double check everything one last time before confirming to create the server (aka "droplet"):
Now I just need to log into the server with the correct key. The first time logging in, you'll be prompted e.g. "are you sure…." - so go ahead and proceed. There won't be any prompt during subsequent logins.
It's a good practice to NOT operate as the root user. Root users have too much power, and a small mistake can easily render your server less secured, or completely broken. So, the 1st thing we do after successfully log in as root, is to create a non-root (normal) user. Here's an example of how one might do this:
Once you've got a normal user created, please feel free to set up the ssh login for that user (just like you did earlier with the root user). The only difference is that this time you don't have the Digital Ocean interface for this. How to do that is beyond the scope of this document, but here's an example tutorial. Just like last time, this step is optional. If you do it correctly, you'll be able to log into the server as the non-root user, like shown in this example screenshot:
All tools like node, npm, near-cli can be installed exactly as instructed in the Stakewars guide. Below, however, is how I typically install the tools for myself. Note that you don't have to do exactly as shown below. As long as you have all the tools at the end of the day, then you're good.
Note that above, we explicitly set NEAR_ENV to shardnet for all login sessions. If you intend to use this user, on this server, for only shardnet (for the purpose of the Stakewars), then you should do this. Alternatively, you don't have to do this, but you must set NEAR_ENV for each and every near-cli command you invoke, otherwise it'll default to testnet, which is completely different from shardnet.
With near-cli installed, we can now investigate the blockchain network info, such as the validator proposals:
or the performance of current validators:
or the state of the validator nodes in the near future:
Heads up: earlier, we created a non-root user to handle non-system operations (e.g. interacting with the NEAR node). However, the next step requires us to be logged in as root (since we're installing core system dependencies).
First, as instructed, we'll verify that the server CPU architecture supports NEAR node:
Next, all system dependencies are to be installed:
When I first did this, the Stakewars guide had a typo, and as a result, python3-pip was not installed on my server. Therefore, I needed to install it explicitly afterwards:
Again, just follow the official guideline and set up the rest of the dependencies & system env vars:
Next up is to install the rust toolchain. To my knowledge, rust itself can be installed per-user, so for this I can log in as a non-root user. Your mileage here might vary, but in my case, I got a small hiccup, as the system rust was interfering with the user rust. This was how the hiccup was:
To resolve the issue, I had to log in as root again, and remove the cargo and rustc packages from the system:
Once those are done, as the non-root user, I can finally install rust:
With rust installed, let's clone the NEAR sourcecode and compile nearcore ourselves!
Once nearcore is compiled, we'll pull the system configuration files as needed (e.g. genesis.json, config.json and the other node specific config files):
As shown above, I saved the original genesis.json file into genesis.json.original, then download the new genesis (as instructed by the Stakewars guide) as my official genesis.json file. Again, you don't have to do exactly as I did; as long as you have the correct genesis.json file, then you're good.
With the correct genesis file in place, we can actually run the node and start participating in the chain (although not as a validator node - yet)!
I'd suggest that you leave the node running (like in the screenshot above), as it'll take some time for it to fully sync anyways. The plan is, while it syncs, we'll take care of something else, then later when it's fully synced, we'll finish the rest of the server-side setup.
- NEAR validator key implements the public-private key mechanism. Using the
near generate-keycommand, a file will be written to a designated destination, that file contains the key pair that can then be used as your node "validator key". - The key generation operation can happen anywhere (on any computer, not necessarily the server that hosts your NEAR validator node), so it's recommended that you generate it on your local computer, then copy the file over to the server where nearcore runs.
HEADS UP: in the screenshot below, I can call near-cli commands without specifying NEAR_ENV=shardnet because I already set it as a default env var for my session (on my local machine). If you don't do so, you'll need to specify the env var for each and every command.
First, you'll need to authenticate yourself with the NEAR blockchain. Executing near login should trigger this process:
In the example screenshots above, I followed the process that near login triggered and logged myself in. Note that, at the end of this process, there should exist a file in the ~/.near-credentials
Now, let's check if there's no file existing at the destination:
If, for any reason, a file already exists at that exact location, then you might wanna back it up somewhere, so that you're not accidentally overwriting something that you don't even know about ;) - but of course the call is yours!
Then let's generate a key pair. You can enter the name of the staking pool you want to use, so it's pre-filled in the output file for you:
As per the instruction, you'll now need to manually edit the newly created file, and replace private_key with secret_key
This file should exist in the NEAR workdir on your server machine (next to other config files such as config.json)
Now, if your node has fully synced with the latest blockchain state, we can move on to the last part: making neard an autonomous system service.
What we mean by "autonomous system service" is a service that always runs on the background, and will automatically start every time after the server restarts (either on purpose or by accident).
All this is also rather easy if you just follow the instruction as-is. My /etc/systemd/system/neard.service file looks like so:
Once that file is in place, you can go ahead and start neard like any standard service (e.g. systemctl start neard). You'll also want to make it start automatically on server restart (e.g. systemctl enable neard)
Once neard is started as a service, you can follow its log like so: journalctl -n 100 -f -u neard | ccze -A (the ccze part is to support colors in log output, it is optional, but can be helpful at times)
Now that the node is running, you need a Staking Pool contract to officially register yourself as a validator node. This was my command & its result:
NEAR_ENV=shardnet near call factory.shardnet.near create_staking_pool '{"staking_pool_id": "dklabco", "owner_id": "zur.shardnet.near", "stake_public_key": "ed25519:5Hg5KdmNy5X4K6uCZq686YgPrHk9gVZ6d48JtFXPtmQc", "reward_fee_fraction": {"numerator": 7, "denominator": 100}, "code_hash":"DD428g9eqLL8fWUxv8QSpVFzyHi1Qd16P8ephYCTmMSZ"}' --accountId="zur.shardnet.near" --amount=30 --gas=300000000000000
Scheduling a call: factory.shardnet.near.create_staking_pool({"staking_pool_id": "dklabco", "owner_id": "zur.shardnet.near", "stake_public_key": "ed25519:5Hg5KdmNy5X4K6uCZq686YgPrHk9gVZ6d48JtFXPtmQc", "reward_fee_fraction": {"numerator": 7, "denominator": 100}, "code_hash":"DD428g9eqLL8fWUxv8QSpVFzyHi1Qd16P8ephYCTmMSZ"}) with attached 30 NEAR
Doing account.functionCall()
Retrying request to broadcast_tx_commit as it has timed out [
'EQAAAHp1ci5zaGFyZG5ldC5uZWFyAKPx0+Cfn9MhH7zBjbaU/ZBvVaMS+8S36Ybn2VIVB2Q8Qy8joQIBAAAVAAAAZmFjdG9yeS5zaGFyZG5ldC5uZWFyXRpZ0zv5rLt/4W4BujQfWuiB9bBEXK6Q+c4uyDn8skcBAAAAAhMAAABjcmVhdGVfc3Rha2luZ19wb29s+QAAAHsic3Rha2luZ19wb29sX2lkIjoiZGtsYWJjbyIsIm93bmVyX2lkIjoienVyLnNoYXJkbmV0Lm5lYXIiLCJzdGFrZV9wdWJsaWNfa2V5IjoiZWQyNTUxOTo1SGc1S2RtTnk1WDRLNnVDWnE2ODZZZ1BySGs5Z1ZaNmQ0OEp0RlhQdG1RYyIsInJld2FyZF9mZWVfZnJhY3Rpb24iOnsibnVtZXJhdG9yIjo3LCJkZW5vbWluYXRvciI6MTAwfSwiY29kZV9oYXNoIjoiREQ0MjhnOWVxTEw4ZldVeHY4UVNwVkZ6eUhpMVFkMTZQOGVwaFlDVG1NU1oifQDAbjHZEAEAAAAA3tgDPEK/0BgAAAAAAACIKI0kWW8sz/KmjnZQOnZ+Du1ko1xolmbHv+TJHLmx5BBwKTMCcxXD0cua6R18yoMU8aSiV+JE5HRGeDdcSUII'
]
Retrying request to broadcast_tx_commit as it has timed out [
'EQAAAHp1ci5zaGFyZG5ldC5uZWFyAKPx0+Cfn9MhH7zBjbaU/ZBvVaMS+8S36Ybn2VIVB2Q8Qy8joQIBAAAVAAAAZmFjdG9yeS5zaGFyZG5ldC5uZWFyXRpZ0zv5rLt/4W4BujQfWuiB9bBEXK6Q+c4uyDn8skcBAAAAAhMAAABjcmVhdGVfc3Rha2luZ19wb29s+QAAAHsic3Rha2luZ19wb29sX2lkIjoiZGtsYWJjbyIsIm93bmVyX2lkIjoienVyLnNoYXJkbmV0Lm5lYXIiLCJzdGFrZV9wdWJsaWNfa2V5IjoiZWQyNTUxOTo1SGc1S2RtTnk1WDRLNnVDWnE2ODZZZ1BySGs5Z1ZaNmQ0OEp0RlhQdG1RYyIsInJld2FyZF9mZWVfZnJhY3Rpb24iOnsibnVtZXJhdG9yIjo3LCJkZW5vbWluYXRvciI6MTAwfSwiY29kZV9oYXNoIjoiREQ0MjhnOWVxTEw4ZldVeHY4UVNwVkZ6eUhpMVFkMTZQOGVwaFlDVG1NU1oifQDAbjHZEAEAAAAA3tgDPEK/0BgAAAAAAACIKI0kWW8sz/KmjnZQOnZ+Du1ko1xolmbHv+TJHLmx5BBwKTMCcxXD0cua6R18yoMU8aSiV+JE5HRGeDdcSUII'
]
Retrying request to broadcast_tx_commit as it has timed out [
'EQAAAHp1ci5zaGFyZG5ldC5uZWFyAKPx0+Cfn9MhH7zBjbaU/ZBvVaMS+8S36Ybn2VIVB2Q8Qy8joQIBAAAVAAAAZmFjdG9yeS5zaGFyZG5ldC5uZWFyXRpZ0zv5rLt/4W4BujQfWuiB9bBEXK6Q+c4uyDn8skcBAAAAAhMAAABjcmVhdGVfc3Rha2luZ19wb29s+QAAAHsic3Rha2luZ19wb29sX2lkIjoiZGtsYWJjbyIsIm93bmVyX2lkIjoienVyLnNoYXJkbmV0Lm5lYXIiLCJzdGFrZV9wdWJsaWNfa2V5IjoiZWQyNTUxOTo1SGc1S2RtTnk1WDRLNnVDWnE2ODZZZ1BySGs5Z1ZaNmQ0OEp0RlhQdG1RYyIsInJld2FyZF9mZWVfZnJhY3Rpb24iOnsibnVtZXJhdG9yIjo3LCJkZW5vbWluYXRvciI6MTAwfSwiY29kZV9oYXNoIjoiREQ0MjhnOWVxTEw4ZldVeHY4UVNwVkZ6eUhpMVFkMTZQOGVwaFlDVG1NU1oifQDAbjHZEAEAAAAA3tgDPEK/0BgAAAAAAACIKI0kWW8sz/KmjnZQOnZ+Du1ko1xolmbHv+TJHLmx5BBwKTMCcxXD0cua6R18yoMU8aSiV+JE5HRGeDdcSUII'
]
Retrying request to broadcast_tx_commit as it has timed out [
'EQAAAHp1ci5zaGFyZG5ldC5uZWFyAKPx0+Cfn9MhH7zBjbaU/ZBvVaMS+8S36Ybn2VIVB2Q8Qy8joQIBAAAVAAAAZmFjdG9yeS5zaGFyZG5ldC5uZWFyXRpZ0zv5rLt/4W4BujQfWuiB9bBEXK6Q+c4uyDn8skcBAAAAAhMAAABjcmVhdGVfc3Rha2luZ19wb29s+QAAAHsic3Rha2luZ19wb29sX2lkIjoiZGtsYWJjbyIsIm93bmVyX2lkIjoienVyLnNoYXJkbmV0Lm5lYXIiLCJzdGFrZV9wdWJsaWNfa2V5IjoiZWQyNTUxOTo1SGc1S2RtTnk1WDRLNnVDWnE2ODZZZ1BySGs5Z1ZaNmQ0OEp0RlhQdG1RYyIsInJld2FyZF9mZWVfZnJhY3Rpb24iOnsibnVtZXJhdG9yIjo3LCJkZW5vbWluYXRvciI6MTAwfSwiY29kZV9oYXNoIjoiREQ0MjhnOWVxTEw4ZldVeHY4UVNwVkZ6eUhpMVFkMTZQOGVwaFlDVG1NU1oifQDAbjHZEAEAAAAA3tgDPEK/0BgAAAAAAACIKI0kWW8sz/KmjnZQOnZ+Du1ko1xolmbHv+TJHLmx5BBwKTMCcxXD0cua6R18yoMU8aSiV+JE5HRGeDdcSUII'
]
Retrying request to broadcast_tx_commit as it has timed out [
'EQAAAHp1ci5zaGFyZG5ldC5uZWFyAKPx0+Cfn9MhH7zBjbaU/ZBvVaMS+8S36Ybn2VIVB2Q8Qy8joQIBAAAVAAAAZmFjdG9yeS5zaGFyZG5ldC5uZWFyXRpZ0zv5rLt/4W4BujQfWuiB9bBEXK6Q+c4uyDn8skcBAAAAAhMAAABjcmVhdGVfc3Rha2luZ19wb29s+QAAAHsic3Rha2luZ19wb29sX2lkIjoiZGtsYWJjbyIsIm93bmVyX2lkIjoienVyLnNoYXJkbmV0Lm5lYXIiLCJzdGFrZV9wdWJsaWNfa2V5IjoiZWQyNTUxOTo1SGc1S2RtTnk1WDRLNnVDWnE2ODZZZ1BySGs5Z1ZaNmQ0OEp0RlhQdG1RYyIsInJld2FyZF9mZWVfZnJhY3Rpb24iOnsibnVtZXJhdG9yIjo3LCJkZW5vbWluYXRvciI6MTAwfSwiY29kZV9oYXNoIjoiREQ0MjhnOWVxTEw4ZldVeHY4UVNwVkZ6eUhpMVFkMTZQOGVwaFlDVG1NU1oifQDAbjHZEAEAAAAA3tgDPEK/0BgAAAAAAACIKI0kWW8sz/KmjnZQOnZ+Du1ko1xolmbHv+TJHLmx5BBwKTMCcxXD0cua6R18yoMU8aSiV+JE5HRGeDdcSUII'
]
Receipts: 3SmSL8dK9jjfFkDjSNKMAmyYtgLKaZCv5dzeskebBTof, 6iW39NYeZEneV34VoveBrV3LWzLtMsETc1TqJ1tutrRe
Log [factory.shardnet.near]: The staking pool @dklabco.factory.shardnet.near was successfully created. Whitelisting...
Transaction Id 4SJJjXk7SKpa37FtNE7keYePEVr6EdjXpUDVGHjg3TSS
To see the transaction in the transaction explorer, please open this url in your browser
https://explorer.shardnet.near.org/transactions/4SJJjXk7SKpa37FtNE7keYePEVr6EdjXpUDVGHjg3TSS
Once the Staking Pool contract was created, anyone can stake money to your pool (including yourself aka self-stake). The command & the result for me was like so:
NEAR_ENV=shardnet near call dklabco.factory.shardnet.near deposit_and_stake --amount 502 --accountId zur.shardnet.near --gas=300000000000000
Scheduling a call: dklabco.factory.shardnet.near.deposit_and_stake() with attached 502 NEAR
Doing account.functionCall()
Retrying request to broadcast_tx_commit as it has timed out [
'EQAAAHp1ci5zaGFyZG5ldC5uZWFyAJK6q9Nv36VvBvjnUgQ0zS1RHF6Nn1jpkyO78HPJi3j9gV0qousAAAAdAAAAZGtsYWJjby5mYWN0b3J5LnNoYXJkbmV0Lm5lYXJKU+0RONpS+G8xWWLCjEDxBMtfKAmFk2khQ48cidYqcQEAAAACEQAAAGRlcG9zaXRfYW5kX3N0YWtlAgAAAHt9AMBuMdkQAQAAAAC2+dmFh6I+nwEAAAAAAL7P5+dxJT+ts9dFAL6jOHS3yxkiF8Di+OC1wJRxdCB0zY99sywqBQitdcezw5GxRa1XZapQ/2hLKZygUmVtggE='
]
Receipts: 4tjeD9VE7bU5Rbk9cCtRMfAyujTwv8qKnH8nDFfP73Jx, 5TwKKhvcAoZ2ZV4hHC1qARKW2obw2tmvX1Z1Tgiu7TPM, CD8Y1AvEnTtjvaWbhFw6QR3xbbtmoRuaWLCeEofbLux5
Log [dklabco.factory.shardnet.near]: @zur.shardnet.near deposited 502000000000000000000000000. New unstaked balance is 502000000000000000000000000
Log [dklabco.factory.shardnet.near]: @zur.shardnet.near staking 502000000000000000000000000. Received 502000000000000000000000000 new staking shares. Total 0 unstaked balance and 502000000000000000000000000 staking shares
Log [dklabco.factory.shardnet.near]: Contract total staked balance is 531999999999999000000000000. Total number of shares 531999999999999000000000000
Transaction Id AGAjfmc3fFRAZ5dZeWGuYiNU4YtfS49XCsGwxiYfPNmR
To see the transaction in the transaction explorer, please open this url in your browser
https://explorer.shardnet.near.org/transactions/AGAjfmc3fFRAZ5dZeWGuYiNU4YtfS49XCsGwxiYfPNmR
''
The above suggests that I've successfully self-staked (self-delegated) my first 502 NEAR tokens to my shardnet validator node.
Please feel free to explore the remaining contract calls as mentioned in the stakewars challenge guideline.
An example of calling the method is_account_unstaked_balance_available on my staking pool for one of my delegator accounts:
There are various things to care about when it comes to reading validator node status. The RPC API is one widely used approach. Here's an example of reading the neard build version:
Note that the utilities curl and jq are used in these example operations, so you might want to install them first with apt install curl jq.
Using the near-cli itself is also a popular way to extract very useful data out of the node.
Here's how to check on the delegators to your validator node:
And here's an example call to check why your node will be kicked out of the active validator pool:
Note that the call above returned "empty" because the node will not be kicked out due to any malperformance during the current epoch. If the node performs poorly, the "kickout reason" shall be displayed like in this example:
curl -s -d '{"jsonrpc": "2.0", "method": "validators", "id": "dontcare", "params": [null]}' -H 'Content-Type: application/json' 176.9.84.61:3030 | jq -c '.result.prev_epoch_kickout[] | select(.account_id | contains ("dklabco.factory.shardnet.near"))' | jq .reason
{
"NotEnoughChunks": {
"expected": 3,
"produced": 1
}
}
Besides the cli tools, you can always look around for UI tools that might be more convenient (at times / to some people). For example, we at DKLAB developed this tool to check on a validator nodes' delegators (and their status).