0chain_inegration_tests

Integration tests

Integration testing combines individual 0chain modules and tests them as a group. Integration testing evaluates the compliance of a system for specific functional requirements and usually occurs after unit testing.

For integration testing, A conductor which is an RPC(Remote Procedure Call) server is implemented to control the behavior of nodes.To know more about the conductor refer to the conductor documentation

Architecture

A conductor requires the nodes to be built in a certain order to control them during the tests. A config file is defined in conductor.config.yaml which contains important details such as details of all nodes used and custom commands used in integration testing.

For running multiple test cases, the conductor uses a test suite that contains multiple sets of tests.Test suites can be categorized into 3 types of tests

standard tests - Checks whether chain continue to function properly despite bad miner and sharder participants

view-change tests - Checks whether addition and removal of nodes is working

blobber tests - Checks whether storage functions continue to work properly despite bad or lost blobber, and confirms expected storage function failures

Below is an example of the conductor test suite.

# Under `enable` is the list of sets that will be run.
enable: 
  - "Miner down/up"
  - "Blobber tests"

# Test sets defines the test cases it covers.
sets: 
  - name: "Miner down/up" 
    tests:
      - "Miner: 50 (switch to contribute)"
      - "Miner: 100 (switch to share)"
  - name: "Blobber tests"
    tests:
      - "All blobber tests"

# Test cases defines the execution flow for the tests.
tests: 
  - name: "Miner: 50 (switch to contribute)"
    flow: 
    # Flow is a series of directives.
    # The directive can either be built-in in the conductor 
    # or custom command defined in "conductor.config.yaml"
      - set_monitor: "sharder-1" # Most directive refer to node by name, these are defined in `conductor.config.yaml` 
      - cleanup_bc: {} # A sample built-in command that triggers stop on all nodes and clean up.
      - start: ['sharder-1']
      - start: ['miner-1', 'miner-2', 'miner-3']
      - wait_phase: 
          phase: 'contribute'
      - stop: ['miner-1']
      - start: ['miner-1']
      - wait_view_change:
          timeout: '5m'
          expect_magic_block:
            miners: ['miner-1', 'miner-2', 'miner-3']
            sharders: ['sharder-1']
  - name: "Miner: 100 (switch to share)"
    flow:
    ...
  - name: "All blobber tests"
    flow:
      - command:
          name: 'build_test_blobbers' # Sample custom command that executes `build_test_blobbers`
    ...
...

Getting Started

Prerequisites

Docker and Git must be installed to run the tests.

Install Git using the following command:

sudo apt install git

Docker installation instructions can be found here.

Cloning the repository and Building Base Image

Clone the 0chain repository:

git clone https://github.com/0chain/0chain.git

Build miner docker image for integration test

(cd 0chain && ./docker.local/bin/build.miners-integration-tests.sh)

Build sharder docker image for integration test

(cd 0chain && ./docker.local/bin/build.sharders-integration-tests.sh)

NOTE: The miner and sharder images are designed for integration tests only. If wanted to run the chain normally, rebuild the original images.

(cd 0chain && ./docker.local/bin/build.sharders.sh && ./docker.local/bin/build.miners.sh)

Confirm that view change rounds are set to 50 on 0chain/docker.local/config.yaml

    start_rounds: 50
    contribute_rounds: 50
    share_rounds: 50
    publish_rounds: 50
    wait_rounds: 50

Running standard tests

Run miners test

(cd 0chain && ./docker.local/bin/start.conductor.sh miners)

Run sharders test

(cd 0chain && ./docker.local/bin/start.conductor.sh sharders)

Running view-change tests

1. Set view_change: true on 0chain/docker.local/config.yaml
2. Run view-change tests

(cd 0chain && ./docker.local/bin/start.conductor.sh view-change-1)
(cd 0chain && ./docker.local/bin/start.conductor.sh view-change-2)
(cd 0chain && ./docker.local/bin/start.conductor.sh view-change-3)

Running blobber tests

Cloning required repos

blobber

git clone https://github.com/0chain/blobber.git

zboxcli

git clone https://github.com/0chain/zboxcli.git

zwalletcli

git clone https://github.com/0chain/zwalletcli.git

0dns

git clone https://github.com/0chain/0dns.git

Confirm whether all the cloned directories exist.

0chain/
blobber/
zboxcli/
zwalletcli/
0dns/

Install zboxcli

(cd zboxcli && make install)

Install zwalletcli

(cd zwalletcli && make install)

Patch 0dns for the latest 0chain network configuration.

(cd 0dns && git apply --check ../0chain/docker.local/bin/conductor/0dns-local.patch)
(cd 0dns && git apply ../0chain/docker.local/bin/conductor/0dns-local.patch)

Patch blobbers for the latest blobber tests

(cd blobber && git apply --check ../0chain/docker.local/bin/conductor/blobber-tests.patch)
(cd blobber && git apply ../0chain/docker.local/bin/conductor/blobber-tests.patch)

Add ~/.zcn/config.yaml as follows

block_worker: http://127.0.0.1:9091
signature_scheme: bls0chain
min_submit: 50
min_confirmation: 50
confirmation_chain_length: 3
max_txn_query: 5
query_sleep_time: 5

Apply if on Ubuntu 18.04

https://github.com/docker/for-linux/issues/563#issuecomment-547962928

The bug relates to docker-credential-secretservice package required by docker-compose and used by docker. A docker process (a build, for example) can sometimes fail due to the bug. Some tests have internal docker builds and can fail due to this bug.

Run blobber tests

(cd 0chain && ./docker.local/bin/start.conductor.sh blobber-1)
(cd 0chain && ./docker.local/bin/start.conductor.sh blobber-2) 

Adding new Tests

New tests can be easily added to the conductor check Updating conductor tests in the conductor documentation for more information.

Enabling or Disabling Tests

Check Temporarily disabling tests in the conductor documentation for more information

Supported Conductor Commands

Check the supported directives in the conductor documentation for more information.

Creating Custom Conductor Commands

Check Custom Commands in the conductor documentation for more information