Merge pull request #223 from dragonchain/master

Release 4.1.0

.dockerignore

@@ -32,6 +32,9 @@ pip-delete-this-directory.txt
 .vscode
 *.swp
 
+# Built helm charts
+dragonchain-k8s-*.tgz
+
 # OS Files
 .DS_Store
 desktop.ini

.gitignore

@@ -24,6 +24,9 @@ pip-delete-this-directory.txt
 .build
 docs/static/chart/
 
+# Built helm charts
+dragonchain-k8s-*.tgz
+
 # OS Files
 .DS_Store
 desktop.ini

.version

@@ -1 +1 @@
-4.0.1
+4.1.0

CHANGELOG.md

@@ -1,5 +1,46 @@
 # Changelog
 
+## 4.1.0
+
+Note this update adds the invoker tag field for indexing smart contract
+transactions. This will only be added for all new smart contracts after
+updating to this version (or newer). If you want/need to retroactively index
+this field for existing transactions,
+[check these docs](https://dragonchain-core-docs.dragonchain.com/latest/deployment/migrating_v4.html#manually-triggering-a-reindex)
+for info on manually triggering a re-index which will populate the invoker
+field where necessary.
+
+- **Feature:**
+  - Default error reporting to save to disk, so that crash logs/tracebacks can be automatically saved
+  - Provide better error message when bad input to api doesn't match required schemas
+  - Adds verification-notification callback in the reciept endpoint
+  - Add indexed redisearch tag field "invoker" by default when indexing smart contract transactions
+  - Remove max limit for custom indexes on a transaction type/smart contract
+- **Bugs:**
+  - Fix a bug where getting the cached list of verifications for a block would always fail
+  - Fix a bug where existing interchain networks could be overwritten if trying to create a new network with the same blockchain and name
+  - Fix a bug where the chain could not parse very old legacy transactions from storage
+  - Fix a bug where L5s returned a 404 for the status endpoint when a default interchain was not yet set
+  - Fix a bug where very large payloads could attempt to be cached and crash the caching redis
+  - Fix a bug so that non-existant routes in the api now properly return a 404
+  - Fix a bug where accepting the disable_schedule parameter when updating a contract didn't do anything
+  - Fix a bug where the deadline key for l2+ nodes skipping unneeded blocks was generated incorrectly
+  - Fix a bug where custom indexes text fields couldn't have both sortable and nostem at the same time
+- **Documentation:**
+  - Change dragonchain deployment docs to reflect helm install changes from helm repository with pinned version
+  - Stop posting helm chart directly to docs
+- **Packaging:**
+  - Update aioredis, docker, boto3, web3, redis, fastjsonschema, and aiohttp dependencies
+  - Update installed version of redisearch to 1.4.17
+  - Update fwatchdog to 0.18.2 for OpenFaaS smart contracts
+  - Update helm chart to use a pinned container version by default
+  - Use a helm chart repository for helm distribution
+  - Add a README for the helm chart itself which will be rendered [on helm hub](https://hub.helm.sh/charts/dragonchain/dragonchain-k8s)
+- **Development:**
+  - Enforce `appVersion` in Chart.yaml and image version tags to be always up to date (and add associated version bump helper function in `tools.sh`)
+  - Add strict helm lint checking
+  - Add a public docker container for the current build of the master (development) branch (`dragonchain/dragonchain_core:edge`)
+
 ## 4.0.1
 
 - **Bugs:**

README.md

@@ -65,8 +65,8 @@ In order to develop locally you should be able to run `./tools.sh full-test` and
 1. Install the python requirements: `./tools.sh pip-install`
    (Note this will install the python packages to the current user's site-packages.
    For a python venv, follow the steps below)
-1. [Helm](https://helm.sh/) is required to be installed for the docs to be built properly,
-   as it is used to package the helm chart when the docs build
+1. [Helm](https://helm.sh/) is required to be installed for linting the helm chart.
+1. [yq](https://github.com/mikefarah/yq) is required to be installed for building the docs, as it parses the helm chart's version to update the docs dynamically.
 
 ### Using a Python Virtual Environment
 

cicd/CICD.cft.yml

@@ -1,4 +1,4 @@
-Description: CI/CD Pipeline for Dragonchain Docker Builds
+Description: CI/CD Pipeline for Dragonchain Builds
 
 Parameters:
   GitHubSecret:
@@ -82,6 +82,8 @@ Resources:
                   - !GetAtt Bucket.Arn
                   - !Join ["", [!GetAtt DocsWebsiteBucket.Arn, /*]]
                   - !GetAtt DocsWebsiteBucket.Arn
+                  - !Join ["", [!GetAtt HelmChartRepoBucket.Arn, /*]]
+                  - !GetAtt HelmChartRepoBucket.Arn
               - Action:
                   - "secretsmanager:GetSecretValue"
                 Effect: Allow
@@ -185,6 +187,11 @@ Resources:
       WebsiteConfiguration:
         IndexDocument: index.html
 
+  HelmChartRepoBucket:
+    Type: AWS::S3::Bucket
+    Properties:
+      BucketName: dragonchain-charts
+
   CloudFrontDistribution:
     Type: AWS::CloudFront::Distribution
     Properties:
@@ -213,8 +220,7 @@ Resources:
     Properties:
       AliasTarget:
         HostedZoneId: Z2FDTNDATAQYW2
-        DNSName:
-          Fn::GetAtt: [CloudFrontDistribution, DomainName]
+        DNSName: !GetAtt CloudFrontDistribution.DomainName
       HostedZoneId: Z2ADV4KXMZWAVJ
       Name: dragonchain-core-docs.dragonchain.com
       Type: A

cicd/Dockerfile.dependencies

@@ -1,10 +1,12 @@
 # This container is used as a base by Dockerfile.test in order to speed up dependency install for testing purposes only
 FROM python:3.7-alpine
 
-# Install helm for building docs
-RUN wget 'https://get.helm.sh/helm-v2.14.3-linux-amd64.tar.gz' && \
+# Install helm for linting chart, and yq for building docs
+RUN wget -O helm-v2.14.3-linux-amd64.tar.gz 'https://get.helm.sh/helm-v2.14.3-linux-amd64.tar.gz' && \
     tar xzf helm-v2.14.3-linux-amd64.tar.gz && mv linux-amd64/helm /usr/local/bin/helm && \
-    rm -rf helm-v2.14.3-linux-amd64.tar.gz linux-amd64
+    rm -rf helm-v2.14.3-linux-amd64.tar.gz linux-amd64 && \
+    wget -O yq 'https://github.com/mikefarah/yq/releases/download/2.4.0/yq_linux_amd64' && \
+    chmod +x yq && mv yq /usr/local/bin/
 
 # Install dev build dependencies
 RUN apk upgrade && apk add g++ make gmp-dev libffi-dev automake autoconf libtool && echo "UTC" > /etc/timezone

cicd/Dockerfile.test

@@ -1,12 +1,15 @@
 # Change FROM to python:3.7-alpine to test without the dependencies container
 FROM dragonchain/dragonchain_core_dependencies:latest as base
 
-# Install Helm for doc builds if it doesn't exist
+# Install Helm for chart linting and/or yq for doc builds if it doesn't exist
 RUN if ! command -v helm; then \
-    wget 'https://get.helm.sh/helm-v2.14.3-linux-amd64.tar.gz' && \
+    wget -O helm-v2.14.3-linux-amd64.tar.gz 'https://get.helm.sh/helm-v2.14.3-linux-amd64.tar.gz' && \
     tar xzf helm-v2.14.3-linux-amd64.tar.gz && mv linux-amd64/helm /usr/local/bin/helm && \
     rm -rf helm-v2.14.3-linux-amd64.tar.gz linux-amd64; fi && \
-    helm init --client-only
+    helm init --client-only && \
+    if ! command -v yq; then \
+    wget -O yq 'https://github.com/mikefarah/yq/releases/download/2.4.0/yq_linux_amd64' && \
+    chmod +x yq && mv yq /usr/local/bin/; fi
 
 WORKDIR /usr/src/core
 # Install necessary base dependencies and set UTC timezone for apscheduler

cicd/buildspec.deploy.yml

@@ -7,29 +7,45 @@ phases:
     commands:
       - if [ -z "$STAGE" ]; then echo "STAGE env var is missing"; exit 1; fi
       - if [ -z "$AWS_DEFAULT_REGION" ]; then echo "AWS_DEFAULT_VERSION env var is missing"; exit 1; fi
+      # Install helm and s3repo plugin
+      - curl -LO https://git.io/get_helm.sh
+      - bash get_helm.sh --version v2.14.3
+      - helm init --client-only
+      - helm plugin install https://github.com/cheeseandcereal/s3-public-helm-repo --version 0.1.0
+      # Codebuild doesn't set correct execute bit permission when cloning
       - chmod +x ./dragonchain/job_processor/bin/fwatchdog
       # For some reason codebuild doesn't handle these symlinks correctly when cloning, so we just copy the relevant source files instead
       - cp CHANGELOG.md docs/overview/changelog.md && cp CONTRIBUTING.md docs/meta/contributing.md
   pre_build:
     commands:
+      # Run tests before building
       - echo Building and running tests
       - docker build . -f cicd/Dockerfile.test -t built
       - docker run -v $(pwd)/docs:/usr/src/core/docs built
       - if [ ! -d "./docs/.build/html" ]; then echo "Docs did not build correctly!"; exit 1; fi
   build:
     commands:
+      # Package/upload helm chart (if necessary)
+      - sh cicd/deploy_helm.sh
+      # Set docker tags
       - export VERSION=$(cat .version)
       - TAG="381978683274.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/dragonchain_core:$STAGE-$VERSION"
+      - PUB_TAG_EDGE="dragonchain/dragonchain_core:edge"
       - PUB_TAG_LATEST="dragonchain/dragonchain_core:latest"
       - PUB_TAG_VERSION="dragonchain/dragonchain_core:$VERSION"
+      # Login to docker repositories
       - echo Logging into docker
       - $(aws ecr get-login --no-include-email --region us-west-2)
       - aws secretsmanager get-secret-value --secret-id dockerHubPassword --query SecretString --output text | docker login -u dragonchain --password-stdin
+      # Build/tag container
       - echo Building and pushing docker containers
       - docker build . -t $TAG
       - docker tag $TAG $PUB_TAG_LATEST
       - docker tag $TAG $PUB_TAG_VERSION
+      - docker tag $TAG $PUB_TAG_EDGE
+      # Upload built containers and docs
       - docker push $TAG
+      - docker push $PUB_TAG_EDGE
       - sh cicd/deploy_docs.sh
       - if [ "$STAGE" = dev ]; then echo Generating Banana Index && jq -c ".message |= \"$(grep -ir banana . | wc -l)\"" cicd/banana-shield.json > shield.json && aws s3 cp shield.json s3://dragonchain-core-docs/banana-shield.json; fi
       - if [ "$STAGE" = prod ]; then docker push $PUB_TAG_LATEST && docker push $PUB_TAG_VERSION; fi

cicd/deploy_helm.sh

@@ -0,0 +1,10 @@
+#!/bin/sh
+set -eu
+
+if [ "$STAGE" = "prod" ]; then
+    echo Packaging and uploading helm chart to public repository
+    CHART_FILE="$(helm package helm/dragonchain-k8s/ | grep -o '/.*\.tgz')"
+    helm s3repo add dragonchain-charts "$CHART_FILE" -n
+elif [ "$STAGE" = "dev" ]; then
+    echo Not uploading development helm chart
+fi

docs/components/broadcast_processor.md

@@ -8,13 +8,21 @@ The broadcast processor is responsible for retrieving verifications for
 completed L1 blocks. The microservice uses Dragon Net to find validating nodes
 that match criteria and request verifications from them.
 
-The service's responsibilities include managing discovered Dragon Net chains
-and re-broadcasting blocks if verifications aren't returned in time.
+The service's responsibilities include:
+- Managing discovered Dragon Net chains.
+- Re-broadcasting blocks if verifications aren't returned in time.
+- Optionally notifying a preconfigured URL via HTTP POST when block verifications
+are received.
 
 Turning off or removing the broadcast processor removes the chain from Dragon
 Net (although the `BROADCAST` environment variable should also be set to False
 so that unnecessary state isn't loaded when creating blocks).
 
+HTTP verification notifications may also be broadcast to a preconfigured URL by
+changing the value of `dragonchain.verificationNotification` within the helm
+template's relevant `values.yaml` file i.e.:
+`/docs/static/chart/opensource-config.yaml`.
+
 ### Entrypoint
 
 In order to run the broadcast processor,

docs/deployment/deploying.md

@@ -56,11 +56,9 @@ by the chain. Once doing this, simply remember to set
 ## Helm Chart
 
 **Please Note**: The helm chart is subject to significant changes.
-A helm chart is provided here, but will have better support in the
-future.
 
-Both the helm chart and a template for the necessary values can be downloaded
-[HERE](links).
+A documented template for the necessary configurable helm chart values can be
+downloaded [HERE](links).
 
 ### Deploying Helm Chart
 
@@ -72,7 +70,9 @@ are in the first section).
 Once the values are set, install the helm chart with:
 
 ```sh
-helm upgrade --install my-dragonchain dragonchain-k8s-1.0.0.tgz --values opensource-config.yaml --namespace dragonchain
+helm repo add dragonchain https://dragonchain-charts.s3.amazonaws.com
+helm repo update
+helm upgrade --install my-dragonchain --values opensource-config.yaml --namespace dragonchain dragonchain/dragonchain-k8s --version 1.0.2
 ```
 
 If you need to change any values AFTER the helm chart has already been

docs/deployment/links.rst

@@ -1,15 +1,10 @@
 Deployment Links
 ================
 
-Here are download links for the deployment of a Dragonchain to a
-local kubernetes cluster.
-
-Helm Chart
-""""""""""
-
-:download:`Download Link For Helm Chart </static/chart/dragonchain-k8s-1.0.0.tgz>`
+Here is the download link for a documented example of some common Helm chart
+values for the current version of the dragonchain-k8s helm chart.
 
 Helm Values
 """""""""""
 
-:download:`Download Link For Helm Values Template </static/chart/opensource-config.yaml>`
+:download:`opensource-config.yaml </static/chart/opensource-config.yaml>`

docs/deployment/migrating_v4.md

@@ -49,6 +49,19 @@ be automatically deleted. You can manually delete the old ElasticSearch
 persistent volume from kubernetes and its corresponding data to clear up space
 after upgrading if desired.
 
+#### Manually Triggering a Reindex
+
+If you want to re-index all your blockchain data in the future for any reason,
+you can do so by running the following command in the redisearch pod and
+subsequently restarting the webserver:
+
+```sh
+# Make sure to run this on the redisearch and NOT the persistent redis, or else you will permantently break the chain
+redis-cli flushall
+```
+
+All the notes above still applying when manually triggering a re-index like this.
+
 ### Custom Indexing Changes
 
 As redisearch uses a different schema for indexing, any custom indexes that
@@ -88,6 +101,7 @@ can be queried by default:
 - `timestamp` - Sortable numeric field
 - `block_id` - Sortable numeric field
 - `tag` - Text field
+- `invoker` - Tag field (Only exists on transactions created by smart contract output) (as of 4.1.0)
 
 In addition to these fields, any fields specified as custom index fields
 when creating the transaction type (or smart contract), can also be used when

dragonchain/broadcast_processor/broadcast_functions.py

@@ -17,6 +17,7 @@
 
 import re
 import time
+import os
 from typing import List, Tuple, Set
 
 from dragonchain.lib import dragonnet_config
@@ -28,7 +29,10 @@
 IN_FLIGHT_KEY = "broadcast:in-flight"
 BROADCAST_BLOCK_PREFIX = "broadcast:block"
 CLAIM_CHECK_KEY = "broadcast:claimcheck"
+NOTIFICATION_KEY = "broadcast:notifications"
+
 FAULT_TOLERATION = 10  # Number of times fetching verifications fails before rolling back block verification state
+HAS_VERIFICATION_NOTIFICATIONS = os.environ.get("VERIFICATION_NOTIFICATION") is not None
 
 
 def state_key(block_id: str) -> str:
@@ -102,13 +106,30 @@ def increment_storage_error_sync(block_id: str, current_level: int) -> None:
 
 async def get_blocks_to_process_for_broadcast_async() -> List[Tuple[str, int]]:
     """Get the blocks scheduled to be checked by the broadcast processor right now
-    Args:
     Returns:
         List of (block_id, score) tuples to be processed by the broadcast processor (limit of 1000)
     """
     return await redis.z_range_by_score_async(IN_FLIGHT_KEY, 0, int(time.time()), withscores=True, offset=0, count=1000)
 
 
+async def get_notification_verifications_for_broadcast_async() -> set:
+    """Get the notifications scheduled to be checked by the broadcast processor right now
+    Returns:
+        List of notifications to be re-transmitted by the broadcast processor
+    """
+    return await redis.smembers_async(NOTIFICATION_KEY)
+
+
+async def remove_notification_verification_for_broadcast_async(value: str) -> int:
+    """Removes the notification object
+    Args:
+        value: string to remove from the notification verification set
+    Returns:
+        Number of elements removed from the notification verification list
+    """
+    return await redis.srem_async(NOTIFICATION_KEY, value)
+
+
 async def get_current_block_level_async(block_id: str) -> int:
     """Get the current level of verifications that a particular block is accepting right now (async)
     Args:
@@ -176,6 +197,26 @@ async def schedule_block_for_broadcast_async(block_id: str, time: int = 0) -> No
     await redis.zadd_async(IN_FLIGHT_KEY, time, block_id)
 
 
+def schedule_notification_for_broadcast_sync(notification_location: str) -> None:
+    """Schedule a certain notification to be checked by the broadcast processor (sync)
+    Args:
+        notification_location: notification_location to schedule
+    """
+    redis.sadd_sync(NOTIFICATION_KEY, notification_location)
+
+
+def verification_storage_location(l1_block_id: str, level_received_from: int, chain_id: str) -> str:
+    """ Format the path for the storage of a verification object
+    Args:
+        l1_block_id: the id of the L1 block which this verification verifies
+        level_received_from: the level from which this verification was received
+        chain_id: the internal id of the chain who sent this verification
+    Returns:
+        path: the formatted path
+    """
+    return f"BLOCK/{l1_block_id}-l{level_received_from}-{chain_id}"
+
+
 def set_receieved_verification_for_block_from_chain_sync(block_id: str, level: int, chain_id: str) -> None:
     """Signify a successful receipt from a higher level node receipt for a certain block (sync)
     Args:
@@ -197,6 +238,10 @@ def set_receieved_verification_for_block_from_chain_sync(block_id: str, level: i
     verifications = p.execute()[1]  # Execute the commands and get the result of the scard operation (number of members in the set)
     required = dragonnet_config.DRAGONNET_CONFIG[f"l{level}"]["nodesRequired"]
 
+    if HAS_VERIFICATION_NOTIFICATIONS:
+        # Schedule the notification of this verification
+        schedule_notification_for_broadcast_sync(verification_storage_location(block_id, level, chain_id))
+
     # Check if this block needs to be promoted to the next level
     if verifications >= required:
         if level >= 5: