Merge pull request #194 from dragonchain/master

Production Release 4.0.0

.gitignore

@@ -22,6 +22,7 @@ pip-delete-this-directory.txt
 
 # Docs
 .build
+docs/static/chart/
 
 # OS Files
 .DS_Store

.version

@@ -1 +1 @@
-3.5.0
+4.0.0

CHANGELOG.md

@@ -1,5 +1,43 @@
 # Changelog
 
+## 4.0.0
+
+This update introduces a breaking change required by the switch from
+ElasticSearch to redisearch. These changes primarily exist with custom
+indexing, and how querying is preformed. Please read the
+[relevant docs page](https://dragonchain-core-docs.dragonchain.com/latest/deployment/migrating_v4.html)
+for more details on upgrading to v4.
+
+- **Feature:**
+  - Add new endpoint `GET /v1/contract/logs/<contract_id>` for getting logs of smart contracts
+  - Remove elasticsearch integrations and replace with redisearch
+  - Refactor all query endpoints, including changing query inputs (Breaking Change)
+  - Changed route for `GET /v1/contract` from a query to a list (Breaking Change)
+  - Add support to disable schedule when updating a smart contract
+- **Documentation:**
+  - Update documentation for redisearch
+  - Add new page for migration considerations when upgrading from v3 to v4
+  - Add RAM usage requirements
+  - Add info about bug/security/project bounty programs
+- **Development:**
+  - Add logs when getting storage errors
+  - Resolve all bandit errors and turn up bandit verbosity
+  - Move/extract entire helm chart for better control
+- **Packaging:**
+  - Update helm chart version, adding redisearch while removing elasticsearch
+  - Remove service links from mounting on pods in the helm chart
+  - Expose redis and redisearch image locations in opensource-config.yaml
+  - Add redisearch and remove elasticsearch from requirements.txt
+  - Update boto3, bit, and aiohttp packages
+  - Update fwatchdog to 0.18.0 for OpenFaaS smart contracts
+- **CICD:**
+  - Fixed an issue where the CICD wouldn't properly render changelog or contributing pages
+  - Make helm chart be packaged on demand when creating docs
+- **Bugs:**
+  - Fixed an issue where transaction types whose contracts no longer exist couldn't be deleted
+  - Fixed a bug with ethereum classic and ethereum classic testnet interchains which caused transactions to be signed improperly
+  - Fixed a bug which caused built smart contract images to not be deleted from the image repository when deleting a smart contract
+
 ## 3.5.0
 
 - **Feature:**
@@ -28,7 +66,7 @@
     - `POST /v1/public-blockchain-transaction`
   - Reduced initial delay checks for the webserver so kubernetes will mark the webserver as ready quicker
   - Add direct TLS support for the Dragonchain webserver (for NodePort deployed services)
-- **Bug:**
+- **Bugs:**
   - Fixed some bugs with the helm chart which caused the incorrect dockerhub image to be pulled
 - **Documentation:**
   - Add docs/update helm chart and values for added TLS support
@@ -47,7 +85,7 @@
 
 ## 3.4.46
 
-- **Bug:**
+- **Bugs:**
   - Don't require registration to bitcoin node on webserver boot
   - Properly handle 401-403 responses from matchmaking
 - **Documentation:**
@@ -61,7 +99,7 @@
 
 ## 3.4.45
 
-- **Bug:**
+- **Bugs:**
   - Fix bug with job processor consuming too much memory due to threads
 
 ## 3.4.43
@@ -92,6 +130,6 @@
   - Update fwatchdog to 0.15.2 for OpenFaaS smart contracts
 - **CICD:**
   - Updated cicd for new AWS buildspec runtimes
-- **Bug Fixes**
+- **Bugs:**
   - No longer send HTML on certain 500 responses, only JSON
   - Remove any possible existing entrypoints from built smart contract containers

CONTRIBUTING.md

@@ -163,3 +163,13 @@ dynamic typing.
 
 By contributing, you agree that your contributions will be licensed under the
 Apache License, Version 2.0.
+
+## Bounty Programs
+
+Want to make some money for helping the project?
+
+We have project, bug, and security bounty programs which we invite anyone to
+participate in. Details for these programs can be found here:
+
+- [Bug and Security Bounty Program](https://dragonchain.com/bug-and-security-bounty)
+- [Project Bounty Program](https://dragonchain.com/strategic-projects-bounty)

README.md

@@ -22,6 +22,8 @@ and public blockchain interoperability, Dragonchain shines a new and interesting
 - [Project Documentation](https://dragonchain-core-docs.dragonchain.com/latest/)
 - [Dragonchain Console](https://console.dragonchain.com/)
 - [Dragonchain Inc](https://dragonchain.com/)
+- [Bug and Security Bounty Program](https://dragonchain.com/bug-and-security-bounty)
+- [Project Bounty Program](https://dragonchain.com/strategic-projects-bounty)
 - [Dragonchain Academy](https://academy.dragonchain.org/)
 - [Dragonchain Blog](https://dragonchain.com/blog)
 - [Dragonchain Architecture Document](https://dragonchain.com/assets/Dragonchain-Architecture.pdf)
@@ -49,7 +51,7 @@ For interaction and using the Dragonchain, check out the SDKs (or CLI) and their
 Dragonchain is implemented in Python 3, packaged into container images with Docker, and intended to be run on Kubernetes (at the moment).
 
 The [tools.sh](/tools.sh) script is used to assist in various development functions such as installing dependencies,
-automatically formatting/linting code, runnint tests, etc. Simply run `./tools.sh` with no parameters to view what it
+automatically formatting/linting code, running tests, etc. Simply run `./tools.sh` with no parameters to view what it
 can do.
 
 In order to develop locally you should be able to run `./tools.sh full-test` and have all checks pass. For this, a few requirements should be met:
@@ -63,12 +65,14 @@ In order to develop locally you should be able to run `./tools.sh full-test` and
 1. Install the python requirements: `./tools.sh pip-install`
    (Note this will install the python packages to the current user's site-packages.
    For a python venv, follow the steps below)
+1. [Helm](https://helm.sh/) is required to be installed for the docs to be built properly,
+   as it is used to package the helm chart when the docs build
 
 ### Using a Python Virtual Environment
 
 It is highly recommended to use a python virtual environment rather than simply installing the python
 package requirements to your global environment. This allows the required packages for this project
-to be seperated from the rest of the (potentially conflicting) packages from the rest of the system.
+to be separated from the rest of the (potentially conflicting) packages from the rest of the system.
 
 In order to do this, instead of step 3 above, perform the following steps:
 
@@ -84,7 +88,14 @@ For more information, including a deeper dive on the architecture/code structure
 
 ## 🚀 Contributing
 
-For info on contributing, please read the [contributing](/CONTRIBUTING.md) document.
+Want to make some money for helping the project?
+We have project, bug, and security bounty programs which we invite anyone to participate in.
+Details for these programs can be found here:
+
+- [Bug and Security Bounty Program](https://dragonchain.com/bug-and-security-bounty)
+- [Project Bounty Program](https://dragonchain.com/strategic-projects-bounty)
+
+For more info on contributing, please read the [contributing](/CONTRIBUTING.md) document.
 
 ## ✔️ Support
 

SECURITY.md

@@ -0,0 +1,21 @@
+# Security Policy
+
+The Dragonchain project is dedicated towards finding and eliminating any possible security vulnerabilities that may arise.
+
+This includes this repository as well as all auxillary Dragonchain repositories.
+
+## Supported Versions
+
+Please not that this project currently only supports the latest major version.
+
+If you are running an old version of Dragonchain, we always advise updating, and in the case of breaking changes there will be provided documentation to help you migrate to a newer version if necessary.
+
+## Reporting a Vulnerability
+
+We have a bug and security bounty program which is [available here](https://dragonchain.com/bug-and-security-bounty).
+
+Please do not submit security vulnerabilities publicly, and instead use the link above to disclose any security vulnerabilities.
+
+Please follow reasonable [responsible disclosure](https://en.wikipedia.org/wiki/Responsible_disclosure). Failure to do so will disqualify the security vulnerability from the bounty program.
+
+When in doubt if a bug is a security vulnerability, please contact us first.

cicd/Dockerfile.dependencies

@@ -1,6 +1,11 @@
 # This container is used as a base by Dockerfile.test in order to speed up dependency install for testing purposes only
 FROM python:3.7-alpine
 
+# Install helm for building docs
+RUN wget 'https://get.helm.sh/helm-v2.14.3-linux-amd64.tar.gz' && \
+    tar xzf helm-v2.14.3-linux-amd64.tar.gz && mv linux-amd64/helm /usr/local/bin/helm && \
+    rm -rf helm-v2.14.3-linux-amd64.tar.gz linux-amd64
+
 # Install dev build dependencies
 RUN apk upgrade && apk add g++ make gmp-dev libffi-dev automake autoconf libtool && echo "UTC" > /etc/timezone
 # Install python dev dependencies

cicd/Dockerfile.test

@@ -1,6 +1,13 @@
 # Change FROM to python:3.7-alpine to test without the dependencies container
 FROM dragonchain/dragonchain_core_dependencies:latest as base
 
+# Install Helm for doc builds if it doesn't exist
+RUN if ! command -v helm; then \
+    wget 'https://get.helm.sh/helm-v2.14.3-linux-amd64.tar.gz' && \
+    tar xzf helm-v2.14.3-linux-amd64.tar.gz && mv linux-amd64/helm /usr/local/bin/helm && \
+    rm -rf helm-v2.14.3-linux-amd64.tar.gz linux-amd64; fi && \
+    helm init --client-only
+
 WORKDIR /usr/src/core
 # Install necessary base dependencies and set UTC timezone for apscheduler
 RUN apk --no-cache upgrade && apk --no-cache add libffi libstdc++ gmp && echo "UTC" > /etc/timezone

cicd/buildspec.deploy.yml

@@ -8,6 +8,8 @@ phases:
       - if [ -z "$STAGE" ]; then echo "STAGE env var is missing"; exit 1; fi
       - if [ -z "$AWS_DEFAULT_REGION" ]; then echo "AWS_DEFAULT_VERSION env var is missing"; exit 1; fi
       - chmod +x ./dragonchain/job_processor/bin/fwatchdog
+      # For some reason codebuild doesn't handle these symlinks correctly when cloning, so we just copy the relevant source files instead
+      - cp CHANGELOG.md docs/overview/changelog.md && cp CONTRIBUTING.md docs/meta/contributing.md
   pre_build:
     commands:
       - echo Building and running tests

docs/Makefile

@@ -16,4 +16,4 @@ help:
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %: Makefile
-	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) -W

docs/deployment/deploying.md

@@ -72,7 +72,7 @@ are in the first section).
 Once the values are set, install the helm chart with:
 
 ```sh
-helm upgrade --install my-dragonchain dragonchain-k8s-0.9.0.tgz --values opensource-config.yaml --namespace dragonchain
+helm upgrade --install my-dragonchain dragonchain-k8s-1.0.0.tgz --values opensource-config.yaml --namespace dragonchain
 ```
 
 If you need to change any values AFTER the helm chart has already been

docs/deployment/links.rst

@@ -7,7 +7,7 @@ local kubernetes cluster.
 Helm Chart
 """"""""""
 
-:download:`Download Link For Helm Chart </static/chart/dragonchain-k8s-0.9.0.tgz>`
+:download:`Download Link For Helm Chart </static/chart/dragonchain-k8s-1.0.0.tgz>`
 
 Helm Values
 """""""""""

docs/deployment/migrating_v4.md

@@ -0,0 +1,101 @@
+# Migrating From v3 to v4
+
+Dragonchain follows [Semantic Versioning](https://semver.org), which means that
+v4 introduced backwards-incompatible changes. If you are running a v3
+Dragonchain, then the following are some considerations before upgrading.
+
+## Redisearch Indexing Replacement
+
+The main change with the v4 update is that ElasticSearch has been discarded and
+replaced with redisearch as an indexing solution. This allows the footprint of
+a Dragonchain to be significantly reduced, although breaking changes are
+required.
+
+This means a few things, primarily that query endpoints and custom indexing are
+now completely different.
+
+### Updated SDKs Required For Querying
+
+Because there are breaking API changes with this new indexing system, in order
+to use any type of querying with the Dragonchain, updated SDKs will need to be
+used. These updated SDKs are already available, simply ensure that you have
+installed version 4.0.0 or later for the SDKs in order to work with a v4
+upgraded Dragonchain. Each relevant SDK will also have corresponding docs on
+upgrade considerations for exact API changes required for their particular SDK.
+
+Note that this also means that existing smart contracts which use the query
+functions of a Dragonchain SDK version <4.0.0 will be broken, and need to be
+updated to the newest SDK as well in order to continue working.
+
+### Existing Indexing Migrations
+
+With v4, ElasticSearch and all of its indexes are deleted, so indexes will need
+to be regenerated for redisearch. Fortunately, Dragonchain will do this
+dynamically on webserver boot. It will detect if migrations have ran before,
+and if not, will scan through all of your existing blockchain data, re-indexing
+everything from scratch.
+
+This means that the first boot of the webserver for an upgraded v4 chain may
+take a long time. In order to ensure that kubernetes does not try to reboot the
+webserver with a liveliness probe for being 'unresponsive' while migrations are
+running, set `webserverLiveliness` to `false` in the `opensource-config.yaml`
+values file.
+
+This value should be reset to `true` once the initial boot/migrations of from
+the webserver is complete.
+
+Also note that when upgrading, the persistent volume for ElasticSearch may not
+be automatically deleted. You can manually delete the old ElasticSearch
+persistent volume from kubernetes and its corresponding data to clear up space
+after upgrading if desired.
+
+### Custom Indexing Changes
+
+As redisearch uses a different schema for indexing, any custom indexes that
+exist in a v3 dragonchain are completely incompatible with v4 Dragonchains and
+will be deleted upon upgrading.
+
+Also note that indexes for transactions will now be deleted when their
+transaction type (or smart contract) is removed, rather than persisting like
+before. The transactions still exist, and can be retrieved directly if you have
+their transaction ids, but transactions from deleted transaction types will not
+be queryable. This saves on resources and allows a transaction type's name to
+be freed up and not influence future queries after being removed.
+
+### Querying Notes
+
+v4 Dragonchains support querying both transactions and blocks (querying smart
+contracts has been removed and replaced with a simple list which returns all
+smart contracts).
+
+It's worth noting that by default, all blocks have the following
+redisearch fields which can be used when querying:
+
+- `block_id` - Sortable numeric field
+- `timestamp` - Sortable numeric field
+- `prev_id` - Sortable numeric field
+
+Note `prev_id` is the number of the previous block id in the blockchain.
+This is useful if you want to step through the blockchain one block at a time
+on an L1 chain, as their block ids are not necessarily incremented by the same
+amount.
+
+Transaction querying now requires you to specify the particular transaction
+type to query, rather than querying every single transaction simultaneously.
+Similar to blocks, all transactions have the following redisearch fields which
+can be queried by default:
+
+- `timestamp` - Sortable numeric field
+- `block_id` - Sortable numeric field
+- `tag` - Text field
+
+In addition to these fields, any fields specified as custom index fields
+when creating the transaction type (or smart contract), can also be used when
+querying.
+
+## Dragon Net Communication
+
+Note that communication with Dragon Net remains unchanged, so v3 Dragonchains
+can still communicate with v4 Dragonchains, therefore updating to v4 is not
+required. Please keep in mind that v3 will no longer receive any updates, so
+it is highly recommended to upgrade.

docs/deployment/requirements.md

@@ -27,20 +27,23 @@ cluster with the intent to run dragonchain(s).
 - [Helm](https://helm.sh/) should be installed on the kubernetes cluster, as
   it is used to manage dragonchain deployments.
 
-- Access into the cluster from the greater internet is required
-  for participation in Dragon Net. We highly recommend using some sort of
-  [kubernetes ingress controller](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/)
-  with TLS termination for this. Ensure that if your cluster is behind a NAT
-  layer, the relevant ports to your running controller or chain(s) are
+- Access into the cluster from the greater internet is required for
+  participation in Dragon Net. Ensure that if your cluster is behind a NAT
+  layer, the relevant ports to your running chain(s) are forwarded
   appropriately.
 
+- ~1GB of RAM is required for L2+ chains to run, and ~1.5GB of RAM is
+  required for L1 chains. This means that the kubernetes node running the chain
+  (or the VM in the case of minikube) should have at least ~2GB of RAM total.
+
 ### Recommended
 
 - It is strongly encouraged to have a TLS certificate so that your chain can
   have TLS (HTTPS) support. Without this, your chain may be vulnerable to
   attacks. Running without a valid TLS certificate should be for non-production
-  use only, and TLS _may_ be enforced in the future. You can obtain a free TLS
-  certificate for a domain you own [from letsencrypt](https://letsencrypt.org/)
+  use only, and TLS _may_ be enforced by Dragon Net in the future. You can
+  obtain a free TLS certificate for a domain you own
+  [from letsencrypt](https://letsencrypt.org/).
 
 - It is _HIGHLY_ recommended to have RBAC enabled on the kubernetes cluster.
   See the relevant [kubernetes docs](https://kubernetes.io/docs/reference/access-authn-authz/rbac/)

docs/index.rst

@@ -40,6 +40,7 @@ All of the source code, as well as issue tracker can be viewed `on github <https
    deployment/dragonnet
    deployment/deploying
    deployment/links
+   deployment/migrating_v4
 
 .. toctree::
    :caption: Components

docs/meta/license.rst

@@ -1,4 +1,5 @@
 License
 =======
 
-.. include:: ../../LICENSE
+.. literalinclude:: ../../LICENSE
+    :language: text

docs/overview/architecture.md

@@ -30,8 +30,8 @@ Every chain requires a running instance of the following 3rd party components:
   [configured as an LRU cache](https://redis.io/topics/lru-cache) and is
   intended to be used as a write-thru cache for storage and other ephemeral
   data.
-- [Elasticsearch](https://www.elastic.co/products/elasticsearch) deployment,
-  accessible via the `ELASTICSEARCH_HOST` env var. This should be set up to
+- [Redisearch](https://oss.redislabs.com/redisearch/index.html), accessible via
+  the `REDISEARCH_ENDPOINT` and `REDIS_PORT` env vars. This should be set up to
   persist, as it is used for indexing.
 
 Additionally, L1 (business logic) chains also require a running instance of