Merge branch 'development'

.dockerignore

@@ -2,7 +2,11 @@
 .idea
 api_gateway/client/node_modules
 docs
-data
+data/registry/
+data/postgres/
+data/redis/
+data/portainer/
+data/minio/
 testing
 __pycache__
 **/__pycache__

.gitignore

@@ -37,7 +37,8 @@ data/registry/
 data/postgres/
 data/redis/
 data/portainer/
+data/minio/
 app_sdk/walkoff_app_sdk.egg-info
 venv/
 *.unison.*
-tmp-compose.yml
+tmp-compose.yml

CHANGELOG.md

@@ -2,6 +2,49 @@
 <!-- Use the tags Added, Changed, Deprecated, Removed, Fixed, Security, and
      Contributor to describe changes -->
 
+## [1.0.0-beta.1]
+
+This update introduces a number of new features, including an App Editor in the UI, more granular role-based permissions, 
+and a "bootloader" for automating deployment of WALKOFF. 
+
+### Added
+* App Editor for editing app files and building Docker images from said apps. You can use this to change apps while WALKOFF is running, without restarting the whole stack.
+* More granular RBAC where permissions on individual workflows and global variables can be restricted to specific roles.
+* Execution results display on individual nodes to aid in identifying results.
+* Bootloader container to automate deployment and teardown of the WALKOFF stack.
+* Autogenerated walkoff_client Python package for interfacing with WALKOFF API. Work in progress.
+* JSON editor GUI for editing arrays and objects in action parameters.
+* JSON editor GUI for editing Workflow and Global Variables
+
+### Changed
+* Dashboards renamed to Reports. Lots of work still to do here.
+* Condition and Transform exceptions now get passed up into workflow results for easier debugging.
+* Common config location to minimize number of locations where ports, service names, etc. need to change when configuring WALKOFF.
+* WALKOFF now runs only on Docker Swarm, no longer with plain Docker Compose.
+* All services now follow a walkoff_core, walkoff_app, or walkoff_resource naming scheme to disambiguate services from other stacks.
+* Exceptions in Apps are now propagated correctly to action results.
+* All WALKOFF services that need to be exposed now route through NGINX.
+
+### Security
+* WALKOFF now uses HTTPS behind NGINX using a self-signed certificate.
+
+### Fixed
+* Worker's workflow_types are now tagged with a _walkoff_type to prevent ambiguity with user provided data
+* SSH app updated to allow more conventional use of wildcards, relative/absolute paths, etc.
+* Conditions no longer cause entire subtrees to be cancelled, only the immediate successors.
+* walkoff_default Docker network is attachable by default for external services to attach to us.
+* Endpoints for PUT now correctly use resource IDs in path parameters.
+* Importing a workflow with the same name as an existing one should no longer overwrite the original.
+* Umpire scaling heartbeat slowed down to reduce race conditions - will be replaced with on-demand, resource aware scaling in future.
+* Queued actions are correctly cleaned up when aborting a workflow. 
+* Enforced startup order of all the services to avoid busywaiting when services aren't up.
+* Reduced intensity of "Server not responding" pop-up.
+* Database commit issues relating to Workflow errors resolved.
+* Parameters are correctly passed through to node status messages on the frontend.
+* Actions now enforce results being JSON serializable, preventing issues with serializing Python objects.
+* Action console logger reconnected to frontend.
+
+
 ## [1.0.0-alpha.2]
 
 This update includes numerous bugfixes and a number of reintroduced features. 
@@ -68,7 +111,6 @@ server in preparation for a future move to an asynchronous framework. The follow
 * Triggers have been temporarily removed, but are targeted for a near-future 1.1 release.
 
 
-
 ## [0.9.4]
 ###### 2018-12-11
 

ROADMAP.md

@@ -1,11 +1,10 @@
 WALKOFF 1.0.0 is intended to present a more robust and scalable implementation of Apps and Workers.
 
-This alpha is **not feature complete and is not production-ready**, but is intended for interested parties to test the 
-new architecture and provide feedback.
+The aim of this beta is to finish building out features intended to make running WALKOFF in production smoother.  
 
 ### 1.0 Intentions
 
-- Designed to run inside Docker Compose or Docker Swarm.
+- Designed to run with Docker Swarm.
     - This eliminates all installation dependencies other than Docker and Docker Compose 
     - Each component can also be run outside of a Docker container for development or debug, as long as it has 
     connectivity to the other pieces. However this is not the supported method of installation.
@@ -21,32 +20,37 @@ new architecture and provide feedback.
 
 ### Testing & Feedback Areas of Interest
 
-For the alpha release, here are some of the areas of interest we would like feedback on:
+For the beta release, here are some of the areas of interest we would like feedback on:
 
-- First-time app development.
-- Stress-testing of large/branching workflows.
-- Anything you can manage to break.
+- App development through the App Editor page.
+- Stress-testing of large/branching workflows and large numbers of concurrent workflows.
+- Testing of scaling WALKOFF across multiple Docker Swarm nodes.
 
 ### Roadmap:
 
-The following section gives a rough approximation of what we are looking to accomplish in upcoming releases.
+The following section gives a rough approximation of what we are looking to accomplish in the leadup to a full 1.0.0 release.
 
 #### 1.0.0-alpha.1
 
 - Core components functioning under new architecture.
-- Stable enough to begin writing usable apps and workflows
+- Stable enough to begin writing usable apps and workflows.
+- "Essentials" app suite, e.g. SSH, PowerShell, other common utilities.
+- All workflow node types (Actions, Parallel Actions, Conditions, Transforms, Triggers).
 
 #### 1.0.0-beta.1
 
-- Functional feature parity with pre-1.0 WALKOFF intentions
-    - Event-driven actions (removed in 0.5.0)
-    - Triggers (removed in 1.0.0-alpha.1)
-- Support for more complex workflow composition
-    - Nested workflows (running a workflow as a node in another workflow)
-    - Parallelized actions (dividing up work for a node into n subnodes)
+- App Editor for modifying apps on the UI.
+- Bootloader for automating deployment of WALKOFF.
+- Further stability improvements.
 
-#### 1.0.0-rc.1:
+#### 1.0.0-rc.1 **(Next Step)**:
 
-- Completed unit testing of components and end-to-end testing of a running cluster.
 - Security hardening.
-- "Essentials" app suite, e.g. SSH, PowerShell, other common utilities.
+- Complete unit testing of components and end-to-end testing of a running cluster.
+- Validation of WALKOFF running and scaling to large production workloads on multiple Docker Swarm nodes.
+- Dashboards for monitoring and working with WALKOFF outside of Docker CLI
+- Resource-aware scaling of apps and workflows.
+
+#### 1.1.0:
+
+- Future plans include features such as building App SDKs for other languages, building out more ways of executing workflows (e.g. pollers and listeners)

api_gateway/api/objects/global_variables.yaml

@@ -12,8 +12,11 @@ GlobalVariable:
     name:
       type: string
       description: The name of the global variable
+    permissions:
+      type: array
+      items:
+        type: object
     value:
-      type: string
 #      oneOf:
 #        - type: "string"
 #        - type: "number"

api_gateway/api/objects/roles.yaml

@@ -65,6 +65,11 @@ Resource:
         description: Name of the resource
         type: string
         example: playbooks, cases
+      needed_ids:
+        description: Temp string
+        type: array
+        items:
+          type: string
       permissions:
         description: A list of permissions associated with this resource
         example: [create, read, update, delete, execute]

api_gateway/api/objects/workflows.yaml

@@ -61,6 +61,11 @@ WorkflowJSON:
     _walkoff_type:
       type: string
       description: Workflow type for json decoder
+    permissions:
+      type: array
+      nullable: true
+      items:
+        type: object
 
 
 WorkflowMetaData:
@@ -335,7 +340,6 @@ WorkflowVariable:
      type: string
      description: The name of the workflow variable
    value:
-     type: string
      description: The value of the workflow variable
    description:
      type: string

api_gateway/api/umpire.yaml

@@ -31,7 +31,6 @@
           application/json:
             schema:
               $ref: '#/components/schemas/Error'
-    security: []
 
 /umpire/file/{app_name}/{app_version}:
   get:
@@ -71,7 +70,6 @@
           application/json:
             schema:
               $ref: '#/components/schemas/Error'
-    security: []
 
 /umpire/file_upload:
   post:
@@ -99,7 +97,6 @@
           application/json:
             schema:
               $ref: '#/components/schemas/Error'
-    security: []
 
 /umpire/build:
   get:
@@ -121,7 +118,7 @@
           application/json:
             schema:
               $ref: '#/components/schemas/Error'
-    security: []
+
 /umpire/build/{app_name}/{app_version}:
   post:
     tags:
@@ -155,7 +152,6 @@
           application/json:
             schema:
               $ref: '#/components/schemas/Error'
-    security: []
 
 /umpire/build/{build_id}:
   post:
@@ -184,6 +180,38 @@
           application/json:
             schema:
               $ref: '#/components/schemas/Error'
-    security: []
 
+/umpire/save/{app_name}/{app_version}:
+  post:
+      tags:
+        - Umpire
+      summary: Pushes image from minio to /apps and overwrites it.
+      description: ''
+      operationId: api_gateway.server.endpoints.umpire.save_umpire_file
+      parameters:
+        - name: app_name
+          in: path
+          description: 'The name of the app to list.'
+          required: true
+          schema:
+            type: string
+        - name: app_version
+          in: path
+          description: 'The version number of the app to list.'
+          required: true
+          schema:
+            type: string
+      responses:
+        200:
+          description: Success
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/WorkflowJSON'
+        404:
+          description: Workflow does not exist.
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/Error'
 

api_gateway/client/dist/walkoff/index.html

@@ -14,5 +14,5 @@
 <link rel="stylesheet" href="client/dist/walkoff/styles.a7145bf960e458a8099b.css"></head>
 <body>
     <main-component>Loading...</main-component>
-<script src="client/dist/walkoff/runtime.5978eebdd8411421365f.js"></script><script src="client/dist/walkoff/polyfills-es5.38ed56f75e7d87c23835.js" nomodule></script><script src="client/dist/walkoff/polyfills.425a354b82ebc224c491.js"></script><script src="client/dist/walkoff/scripts.dad7fdaf9a0a03a091ee.js"></script><script src="client/dist/walkoff/main.a0bfc3470eca0083aa57.js"></script></body>
+<script src="client/dist/walkoff/runtime.5978eebdd8411421365f.js"></script><script src="client/dist/walkoff/polyfills-es5.38ed56f75e7d87c23835.js" nomodule></script><script src="client/dist/walkoff/polyfills.425a354b82ebc224c491.js"></script><script src="client/dist/walkoff/scripts.dad7fdaf9a0a03a091ee.js"></script><script src="client/dist/walkoff/main.ed6e481dffde331533b4.js"></script></body>
 </html>

api_gateway/client/src/app/apps/app.service.ts

@@ -37,46 +37,27 @@ export class AppService {
 	}
 
 	getFile(appApi: AppApi, file_path: string): Promise<string> {
-		return this.http.post('/umpire/file', { app_name: appApi.name, app_version: appApi.app_version, file_path})
+		return this.http.get(`api/umpire/file/${ appApi.name }/${ appApi.app_version }`, { params: { file_path }})
 			.toPromise()
 			.then(data => data as string)
-
-		// const Key = `apps/${ appApi.name }/${ appApi.app_version }/${ file_path }`;
-		// return this.getS3()
-		// 	.getObject({ Bucket: 'apps-bucket', Key})
-		// 	.promise()
-		// 	.then(data => data.Body.toString())
 	}
 
 	putFile(appApi: AppApi, file_path: string, file_data: string): Promise<any> {
-		return this.http.post('/umpire/file-upload', { app_name: appApi.name, app_version: appApi.app_version, file_path, file_data})
+		return this.http.post('api/umpire/file_upload', { app_name: appApi.name, app_version: appApi.app_version, file_path, file_data})
 			.toPromise()
-
-		// const Key = `apps/${ appApi.name }/${ appApi.app_version }/${ path }`;
-		// return this.getS3()
-		// 	.putObject({ Bucket: 'apps-bucket', Key, Body })
-		// 	.promise()
-		// 	.catch(e => console.log(e))
 	}
 
 	buildImage(appApi: AppApi): Promise<string> {
-		return this.http.post('/umpire/build', { app_name: appApi.name, app_version: appApi.app_version })
+		return this.http.post(`api/umpire/build/${ appApi.name }/${ appApi.app_version }`, { app_name: appApi.name, app_version: appApi.app_version })
 			.toPromise()
 			.then((data: any) => data.build_id)
 	}
 
 	listFiles(appApi: AppApi) : Promise<any> {
-		return this.http.post('/umpire/files', { app_name: appApi.name, app_version: appApi.app_version })
+		return this.http.get(`api/umpire/files/${ appApi.name }/${ appApi.app_version }`)
 			.toPromise()
+			.then(files => (files as string[]).concat('').map(f => 'root/' + f))
 			.then(this.createTree)
-
-
-		// const Prefix = `apps/${ appApi.name }/${ appApi.app_version }/`;
-		// return this.getS3()
-		// 	.listObjectsV2({ Bucket: 'apps-bucket', Prefix})
-		// 	.promise()
-		// 	.then(data => data.Contents.map(c => c.Key.replace(Prefix, '')))
-		// 	.then(this.createTree);
 	}
 
 	createTree(files: string[]) {
@@ -92,13 +73,16 @@ export class AppService {
 			var parent = tree;
 			for (let i = 0; i < arr.length; i++) {
 				let node: any = { title: arr[i] };
+
+
 				if ( i != arr.length - 1 ) {
 					node.children = [];
 					node.folder = true;
 					node.expanded = true;
+					node.data = { path: arr.slice(1, i + 1).concat('').join('/') }
 				}
 				else {
-					node.data = { path: file }
+					node.data = { path: arr.slice(1).join('/') }
 				}
 
 				let curIndex = parent.findIndex(n => n.title == arr[i])
@@ -114,15 +98,4 @@ export class AppService {
 
 		return tree;
 	}
-
-	// getS3(): S3 {
-	// 	return new S3({
-	// 			accessKeyId: 'walkoff' ,
-	// 			secretAccessKey: 'walkoff123' ,
-	// 			endpoint: 'http://localhost:9001' ,
-	// 			s3ForcePathStyle: true, // needed with minio?
-	// 			signatureVersion: 'v4'
-	// 	});
-	// }
-
 }