This Jenkins plugin enables you to deploy Apache Spark Applications to Spark Standalone clusters as a post build action. Keeps track of previous deployments and you can use the 'kill' feature to kill the previous submission before deploying a new version of your application.
- Support for custom Spark Configuration options.
- Kill previous submissions - Allows you to kill your last submission automatically before deploying the new version, if desired.
- Fail build on unsuccessful deployments (configurable).
- A verbose mode, which prints out requests / responses for easier debugging when things doesn't work as expected.
TBD
| Spark Master URL (*) | This is passed as a configuration to executors. Example: spark://master.spark.cluster.com:6066 |
| Spark Master REST HTTP URL | Optional. If your Spark Masters are behind a load balancer / proxy, you can use this to tell Jenkins which URL to use for communicating over REST. Also, if you have HTTPS enabled, you will have to use this option. If this is not provided, the Spark Master URL will be used to determine the default (that is spark:// will become http://). Example: http://master.spark.cluster.com:6066 |
| Spark Scala Version | Optional. Spark Scala Version. Default: 2.10. |
| Application Resource URL (*) | Application JAR URL. If the JAR file is in HDFS, this should be the HDFS URL. If it is available over HTTP or HTTPS, this should be the relevant URL. File URLs are also supported, but those URLs should be valid within executors for that to work. Example: http://some.file.server/spark-app.jar |
| Main Class (*) | The fully qualified main class name. For example, org.spark.demo.app.Main |
| Application Arguments | These are the command line arguments that will passed to the Spark application. Multiple arguments should be separated by a space. For example: arg1 arg2 arg3 |
| Spark Configuration Properties | Additional properties for Spark. These are typically passed in as --conf parameters for Spark Submit. The settings should be specified as key value pairs separated by commas. Example: spark.driver.supervise=false,spark.executor.memory=2G |
| Kill previous submissions before deploying | If enabled, Jenkins will try to kill the last submission done for this job before deploying latest version. |
| Fail the build if deployment fails | It does exactly what it says :) |
| Verbose | Prints the REST request / response in build output for debugging purposes. |
- Spark on YARN and Mesos are not supported yet.