Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

blobber cleanup #1331

Open
wants to merge 7 commits into
base: staging
Choose a base branch
from
+107 −63
Open

blobber cleanup #1331

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
332d97b
blobber cleanup
hm90121 Nov 19, 2023
3cd2b55
additions
hm90121 Nov 19, 2023
bb6c3b3
add links to standalone guides
hm90121 Nov 23, 2023
8d5dc3a
fixes
hm90121 Nov 24, 2023
a713ca2
minor additions
hm90121 Nov 24, 2023
0e4694a
sudo fix for directories setup
hm90121 Nov 24, 2023
3df2130
typos
hm90121 Nov 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
170 changes: 107 additions & 63 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

# Blobber - A storage provider in Züs network

This readme provides instructions on how to setup and register blobber to the Züs network .
A blobber serves as a storage provider within the Züs network, comprising decentralized servers scattered across the globe, all interconnected to the Züs network to cater to our users storage requirements. This readme provides instructions on how to setup and register blobber to the Züs network.

## Table of Contents

Expand All @@ -15,7 +15,8 @@ This readme provides instructions on how to setup and register blobber to the Z
- [Directory Setup for Blobbers](#directory-setup-for-blobbers)
- [Building and Starting the Blobber](#building-and-starting-the-nodes)
- [Building on Standard Hardware](#building-on-standard-hardware)
- [Building on apple silicon](#building-on-apple-silicon)
- [Building on apple silicon](#building-on-apple-silicon)
- [Creating Allocation on Blobbers](#creating-allocation-on-blobbers)
- [Connect to other network](#connect-to-other-network)
- [Miscellaneous](#miscellaneous)
- [Cleanup](#cleanup)
Expand Down Expand Up @@ -45,100 +46,173 @@ Other apps are [Bolt](https://bolt.holdings/), a wallet that is very secure with

- Linux (Ubuntu Preferred) Version: 20.04 and Above
- Mac(Apple Silicon or Intel) Version: Big Sur and Above
- Windows(Requires WSL ) Version: Windows 11 or 10 version 2004 and above
- Docker is available for Linux, macOS and Windows platforms. Find instructions for the preferred operating system [here](https://docs.docker.com/engine/install/#supported-platforms).
- Windows Version: Windows 11 or 10 version 2004 and later requires WSL2. Instructions for installing WSL with docker can be found [here](https://github.com/0chain/0chain/blob/hm90121-patch-1/standalone_guides.md#install-wsl-with-docker).
- Docker is required to run blobber containers. Instructions for installing Docker can be found [here](https://github.com/0chain/0chain/blob/hm90121-patch-1/standalone_guides.md#install-docker-desktop).

### Directory Setup for Blobbers

1. Clone the Blobber repository using the command
1. Clone the Blobber repository and go to blobber directory.
```
git clone https://github.com/0chain/blobber.git
cd blobber
```
2. In the git/blobber run the following command
2. In the blobber directory run the following command for linux/wsl :

```
chmod +x ./docker.local/bin/blobber.init.setup.sh
./docker.local/bin/blobber.init.setup.sh
sudo ./docker.local/bin/blobber.init.setup.sh
```
**NOTE**: For mac user please run below:
**NOTE**: For mac user please run the command below:
```
chmod +x ./docker.local/bin/blobber.init.setup-mac.sh
./docker.local/bin/blobber.init.setup-mac.sh
sudo ./docker.local/bin/blobber.init.setup-mac.sh
```
## Building and Starting the Nodes

1. Setup a network called testnet0 for each of these node containers to talk to each other.
1. In case network is not configured setup a network called testnet0 for each of these node containers to talk to each other.

```
```
docker network create --driver=bridge --subnet=198.18.0.0/15 --gateway=198.18.0.255 testnet0
```
Note: Run all scripts as sudo
Note: Run all scripts as sudo.

2. Set up the block_worker URL

A block worker URL is a field in the `blobber/config/0chain_validator.yaml` and `blobber/config/0chain_blobber.yaml` configuration files that require the URL of blockchain network you want to connect to. For testing purposes we will connect to the beta 0chain network and replace the default URL in blobber/config/0chain_validator.yaml and 0chain_blobber.yaml with the below-mentioned URL.
A block worker URL is a field in the `blobber/config/0chain_validator.yaml` and `blobber/config/0chain_blobber.yaml` configuration files that require the URL of blockchain network you want to connect to. For testing purposes we will connect to the demo Züs network and replace the default URL in `blobber/config/0chain_validator.yaml` and `0chain_blobber.yaml` with the below-mentioned URL.

```
block_worker: https://demo.zus.network/dns
```
**Note:** Change the default value of block_worker field with the following: `http://198.18.0.98:9091/` for the local testnet.

### Building on standard hardware

3. Go back to the blobber directory and build blobber containers using the scripts below
3. Go back to the blobber directory in terminal(`cd -`) and build blobber containers using the scripts below:

```
./docker.local/bin/build.base.sh
./docker.local/bin/build.blobber.sh
./docker.local/bin/build.validator.sh
```
Note: Run all scripts as sudo.
This would take few minutes.
Note: Run all scripts as sudo. This would take few minutes.

### Building on apple silicon

Sometimes in Apple Silicon devices (m1/m2 macbooks), buildx will not work for build scripts. To force a regular blobber build, run the following instead of the build commands mentioned above:
4. Sometimes in Apple Silicon devices (m1/m2 macbooks), build might fail using the scripts above. To force a regular blobber build, run the following instead of the build commands mentioned above:
```
DOCKER_BUILD=build ./docker.local/bin/build.base.sh
DOCKER_BUILD=build ./docker.local/bin/build.blobber.sh
DOCKER_BUILD=build ./docker.local/bin/build.validator.sh.
```
If the above doesnot work you can follow steps below:

4. To link to local gosdk so that the changes are reflected on the blobber build please use the below command(optional)
- Click on the Finder icon in your dock, or open a new Finder window.

```
./docker.local/bin/build.blobber.dev.sh
- Navigate to the "Applications" folder. You can usually find this on the left sidebar of a Finder window.

- Open the "Utilities" folder within "Applications."

- Look for the "Terminal" application. Right-Click Terminal > Get Info > Check Open using Rosetta.

- Double-click on the Terminal application to open it under Rosetta.

- Now to go to blobber directory and build blobber containers using the commands below:

```
cd blobber
./docker.local/bin/build.base.sh
./docker.local/bin/build.blobber.sh
./docker.local/bin/build.validator.sh
```

5. To link to local gosdk so that the changes are reflected on the blobber build please use the command(optional)
```
./docker.local/bin/build.blobber.dev.sh
```
6. Now install [zwalletcli](https://github.com/0chain/zwalletcli/tree/digismash-patch-2#1-installation), then proceed to configure the network as outlined [here](https://github.com/0chain/zwalletcli/tree/digismash-patch-2#2-configure-network), and create a wallet using zwalletcli as detailed [here](https://github.com/0chain/zwalletcli#creating-wallet---any-command).

5. Now create a wallet using zwalletcli and install zboxcli to perform storage operations on blobbers.Instructions for [creating wallet](https://github.com/0chain/zwalletcli#creating-wallet---any-command) and installing zboxcli are available [here](https://github.com/0chain/zboxcli#installation-guides)
7. Next, install zboxcli to execute storage operations on blobber. Detailed instructions for installation can be found [here](https://github.com/0chain/zboxcli/tree/hm90121-patch-1-1#1-installation).

6. Once the wallet is created, the wallet information will be stored in wallet.json located in the .zcn folder of the Linux home directory. Now navigate to the .zcn folder (this is created during zbox build)
8. Once the wallet is created, the wallet information will be stored in `wallet.json` located in the .zcn folder of the linux or mac `$HOME` directory. Now navigate to the .zcn folder
```
cd $HOME/.zcn/
```
7. Open the wallet.json file. It should be similar to the similar to the output below:
9. Open the wallet.json file. It should be similar to the output below:
```
{"client_id":"4af719e1fdb6244159f17922382f162387bae3708250cab6bc1c20cd85fb594c",
"client_key":"da1769bd0203b9c84dc19846ed94155b58d1ffeb3bbe35d38db5bf2fddf5a91c91b22bc7c89dd87e1f1fecbb17ef0db93517dd3886a64274997ea46824d2c119","keys":[{"public_key":"da1769bd0203b9c84dc19846ed94155b58d1ffeb3bbe35d38db5bf2fddf5a91c91b22bc7c89dd87e1f1fecbb17ef0db93517dd3886a64274997ea46824d2c1>
"private_key":"542f6be49108f52203ce75222601397aad32e554451371581ba0eca56b093d19"}],"mnemonics":"butter whisper wheat hope duck mention bird half wedding aim good regret maximum illegal much inch immune unlock resource congress drift>
"version":"1.0","date_created":"2021-09-09T20:22:56+05:30"}
```
8. Copy the client_id value and paste it into blobbers and validators settings. The files can be found in `blobber/config` directory.
10. Copy the client_id value and paste it into blobbers and validators settings. These files can be found in `blobber/config` directory.

9. Open both the `blobber/config/0chain_validator.yaml` and `blobber/config/0chain_blobber.yaml` and edit the `delegate_wallet` value with your `client_id` value.
11. Open both the `blobber/config/0chain_validator.yaml` and `blobber/config/0chain_blobber.yaml` and edit the `delegate_wallet` value with your `client_id` value.

10. Now run the blobbers by navigating into blobber directories for Blobber1 (git/blobber/docker.local/blobber1) and run the container using
12. Now run the blobbers by navigating into blobber directories for Blobber1 (`cd blobber/docker.local/blobber1`) and run the containers using

```
# For locally build images
../bin/blobber.start_bls.sh

# For remote images
../bin/p0blobber.start.sh

```

Note: Run atleast 4 blobbers for a minimal working setup. For running more blobbers repeat the process in more blobber directories (blobber/docker.local/blobber|i)(blobber1/2/3 etc.).

**_Note: Replace the localhost form `docker.local/p0docker-compose.yml` to your public IP if you are trying to connect to another network ._**

If you are facing `insufficient balance to pay fee` errors when starting blobbers, you can turn
off fees in [0chain.yaml.server_chain.smart_contract.miner](https://github.com/0chain/0chain/blob/3c38dfd0920675d86876a5b8895272cb66ded9ad/docker.local/config/0chain.yaml#LL96C3-L96C16)
by adjusting true to false.
If you are facing `insufficient balance to pay fee` errors when starting blobbers, you can turn off fees in [0chain.yaml.server_chain.smart_contract.miner](https://github.com/0chain/0chain/blob/3c38dfd0920675d86876a5b8895272cb66ded9ad/docker.local/config/0chain.yaml#LL96C3-L96C16) by adjusting true to false.

## Creating Allocation on Blobbers

1. Now you can create allocations on blobber and store files. For creating allocations you need tokens into your wallet, Running the command below in zwallet will give 1 token to wallet.

```sh
./zwallet faucet --methodName pour --input "need token"
```

You can specify the number of tokens required using the following command for adding 5 tokens:

```sh
./zwallet faucet --methodName pour --input "need token" --tokens 5
```

Sample output from `faucet` prints the transaction.

```
Execute faucet smart contract success with txn: d25acd4a339f38a9ce4d1fa91b287302fab713ef4385522e16d18fd147b2ebaf
```
To check wallet balance run `./zwallet getbalance` command

Response:
```
Balance: 5 ZCN (4.2299999999999995 USD)
```

2. Then open zbox in another terminal window and create new allocation using the command below:

```
./zbox newallocation --lock 0.5
```
Note: Atleast have 1 ZCN balance in your wallet before running the command above.

Now, you can store files in allocated space and execute a variety of operations using zboxcli. For a comprehensive list of zbox commands and their respective functionalities, please refer to the documentation [here](https://github.com/0chain/zboxcli/tree/hm90121-patch-1-1#commands-table).

Note: If unable to create new allocations as shown below.

```
./zbox newallocation --lock 0.5
Error creating allocation: transaction_not_found: Transaction was not found on any of the sharders
```
To fix this issue you must lock some tokens on the blobber stake pool. Get the blobber id using the `./zbox ls-blobbers` and use the command below to lock tokens into stake pool.

```
./zbox sp-lock --blobber_id $BLOBBER_ID --tokens 1
```
Note: At least have 1 ZCN token balance in your wallet before locking tokens into stake pool.

Still facing any issues refer to troubleshooting section [here](#troubleshooting).

## Troubleshooting

Expand All @@ -150,7 +224,7 @@ docker ps
```
This should display the container image blobber_blobber and should have the ports mapped like "0.0.0.0:5050->5050/tcp"

2. Now check whether the blobber has registered to the blockchain by running the following zbox command
2. To check whether the blobber has registered to the blockchain by running the following zbox command

```
./zbox ls-blobbers
Expand All @@ -160,7 +234,7 @@ In the response you should see the local blobbers mentioned with their urls for
Sample Response:
```
- id: 0bf5ae461d6474ca1bebba028ea57d646043bbfb6a4188348fd649f0deec5df2
url: http://beta.0chain.net:31304
url: http://demo.zus.network:31304
used / total capacity: 14.0 GiB / 100.0 GiB
last_health_check: 1635347306
terms:
Expand Down Expand Up @@ -190,7 +264,7 @@ Sample Response:
cct: 2m0s
max_offer_duration: 744h0m0s
- id: f8dc4aaf3bb32ae0f4ed575dd6931a42b75e546e07cb37a6e1c6aaf1225891c5
url: http://beta.0chain.net:31305
url: http://demo.zus.network:31305
used / total capacity: 13.3 GiB / 100.0 GiB
last_health_check: 1635347346
terms:
Expand All @@ -205,24 +279,8 @@ Note: When starting multiple blobbers, it could happen that blobbers are not bei

Blobber registration takes some time and adding at least 5 second wait before starting the next blobber usually avoids the issue.

3. Now you can create allocations on blobber and store files.

Note: If unable to create new allocations as shown below.

```
./zbox newallocation --lock 0.5
Error creating allocation: transaction_not_found: Transaction was not found on any of the sharders
```

To fix this issue you must lock some tokens on the blobber.Get the local blobber id using the `./zbox ls-blobbers` and use the following command

```
zbox sp-lock --blobber_id f65af5d64000c7cd2883f4910eb69086f9d6e6635c744e62afcfab58b938ee25 --tokens 0.5
```

## Connect to other network


- Your network connection depends on the block_worker url you give in the `config/0chain_blobber/validator.yaml` and `0chain_blobber.yaml` config file.

```
Expand All @@ -248,41 +306,27 @@ block_worker: https://demo.zus.network/dns

1. Get rid of old unused docker resources:



```

docker system prune

```


2. To get rid of all the docker resources and start afresh:



```

docker system prune -a

```


3. Stop All Containers


```

docker stop $(docker ps -a -q)

```



4. Remove All Containers



```

docker rm $(docker ps -a -q)
Expand Down