Seahorse

Seahorse is an open-source visual framework allowing you to create Apache Spark applications in a fast, simple and interactive way.

Seahorse is distributed under the Apache 2.0 License.

Building Seahorse from source

Prerequisites:

• docker 1.30
• docker-compose 1.9
• JDK 8
• sbt 0.13
• python 2.7
  • PyYAML
• npm 4.6
• jekyll 3.2
  • pygments.rb
  • jekyll-sass-converter
  • jekyll-watch
• PhantomJS

Run

./build/build_all.sh

This will build all the needed docker images and create a docker-compose.yml file. You can now run it using docker-compose up. Seahorse will start at http://localhost:33321.

A good place to start using Seahorse is the basic examples section of the documentation.

Development

Note that in order to contribute to Seahorse you have to sign the Contributor License Agreement.

Before submitting a PR, please run the Scala style check:

sbt scalastylebackend && (cd ./seahorse-workflow-executor && sbt scalastyle)

Running tests

Initialize the submodules before running the tests:

git submodule init
git submodule update

Backend tests:

./build/build_and_run_tests.sh

Frontend tests:

./frontend/run_unit_tests.sh

End-to-end integration tests:

./build/e2e_tests.sh -a

In order for Seahorse to compile and run correctly on Mac OS, you need to increase memory for Docker engine to at least 6GB.

Bash completion for Python scripts

Some of our Python scripts used by devs support bash autocompletion using argcomplete.

pip install argcomplete
activate-global-python-argcomplete --user

See this for global completion support.

Mac OS

Note, that bash 4.2 is required. Installation instruction for Mac users

After the bash upgrade, you may have to rename .bash_profile to .bashrc. And maybe add /usr/local/bin to $PATH. Also, check if you're actually running the new bash with echo $BASH_VERSION - your terminal might still be using the old one.

Developing SDK operations on local repository

To compile and test SDK operations on local repository, you can use seahorse-sdk-example submodule

git submodule init
git submodule update
./build/prepare_sdk_dependencies.sh

Now it will compile and test against the local Seahorse repository:

cd seahorse-sdk-example
sbt test

Enterprise options and support

Seahorse was originally created at deepsense.ai. Technical support and customization options are available upon contact.