v1.4.0

Looking for more beta testers:

• Shuffle can now run as a distributed Log Ingester with Sigma rules, meaning a SIEM may not be necessary in the future for detection engineering. This is through our collaboration with @tenzir

Workflow & App management:

• We implemented the "Integration Framework" app, which standardizes input/output, either through OCSF or otherwise. This makes categories "tool agnostic", meaning you can do a lot of the same actions, with the same input/output for ANY tool in the same category.
• The Workflow UI itself has been updated quite a bit, with a smoother way to work, as well as more suggestion systems to help you build faster.
• The Usecase Generator is further along, utilizing the Integration Framework app. Usecases are now also templates for multiple workflows, which we will expand upon further in the future with a proper Workflow Generator
• Apps made with the App Creator in Shuffle now get an additional action called "Custom action". This allows you to use the already pre-specified authentication & configuration, while making everything modifyable, including the URL path. This makes the use of the "HTTP" app obsolete for existing usecases.
• Workflows can now be started from a lot of new sources, such as Message queues & Kafka, as long as you have at least one on-premises environment running Orborus.
• The new App Search page is live!: https://shuffler.io/search
• Added workflow execution search by keyword in the Workflow Run Debugger (/workflows/debug)
• Standard workflows for Subflow & User Input categorization has been built. This will be distributed in the coming weeks & months after releasing 1.4.0
• Workflow Input Fields exist to allow you to ask questions when a workflow is being started. See them by clicking "Edit" on a workflow and scrolling to the bottom.

MSSP management:

• Added App Auth overrides for subflows! Different parent workflows can now call subflows with different app authentications.
• Workflows now have a sub-org "distribution" system in place, which allows you to share a workflow from a parent org to suborgs. This is the start of a system that allows you to keep different versions of the same workflow in multiple orgs, as to segregate data, but still manage it from a single workflow.
• A new admin dashboard is on the way (/admin2), which will be continuously updated and linked to when every page is finished.
• The Trigger UI has been updated to contain Pipelines, as well as Webhook & Triggers together, instead of just Schedules
• You can now select from any related tenant in the view /admin?tab=suborgs
• Required MFA as well as Billing management and other features for our MSSP power users is coming very soon!

Misc

• Documentation has a smooth new UI! This lets you traverse the documentation a lot easier #1386 #1382
• When you put data in the Datastore, each key now has a public interface that can return their values in formats such as JSON/CSV etc. This is built to integrate well into firewalls, EDR systems etc for easier Threat Intel management.

• File categories have been updated, with a few of them having a side-loading mechanism from Shuffle's standards library (e.g. for Sigma rules)
• Added user account deletion feature for onprem users!
• Fixed "No Organization found" bug after first admin registration for onprem from the last release #1394

What's Changed

• fixed the cancel button to work as intended by @satti-hari-krishna-reddy in #1339
• fixed overflow in conditions by @satti-hari-krishna-reddy in #1336
• added a way to get sub orgs of the current org by @satti-hari-krishna-reddy in #1342
• added an endpoint in backend that loads apps created by user by @satti-hari-krishna-reddy in #1344
• added webhooks to the trigger view by @satti-hari-krishna-reddy in #1372
• added code for deploying tenzir node by @satti-hari-krishna-reddy in #1375
• Implement lazy initialization for Tenzir node deployment by @satti-hari-krishna-reddy in #1381
• feat: Add caching for container status check by @satti-hari-krishna-reddy in #1383
• fresh look for docs by @yashsinghcodes in #1382
• added notification workflow stuff by @dhaval055 in #1331
• Docs performance improvement by @yashsinghcodes in #1386
• Fix for Docs taking code markdown as actual markdown. by @yashsinghcodes in #1393
• Fix for default org creation error for first user. by @yashsinghcodes in #1394
• A simple Modal for pipeline trigger and made improvements in tenzir deployement by @satti-hari-krishna-reddy in #1398
• 1.4.0 - Release time by @frikky in #1399
• discord image fix by @yashsinghcodes in #1401
• Very basic stack test by @0x0elliot in #1403
• fixing few things to make pipelines show up in the triggers ui by @satti-hari-krishna-reddy in #1402

New Contributors

• @satti-hari-krishna-reddy made their first contribution in #1339
• @yashsinghcodes made their first contribution in #1382

Full Changelog: v1.3.3...v1.4.0

v1.3.2

What's Changed

• Added management tools for stat management
• Proxy related bug fixes
• Fixed queue, variables and loop bugs
• Fixed input worker bugs
• Other bug fixes for onprem users

Full Changelog: v1.3.1...v1.3.2

v1.3.3

What's Changed

• Added a KMS (Key Management System) built into shuffle with secrets encrypted in the database with support for third party secret providers.
• Better workflow execution stats and enhanced workflow debugger
• Code editor quality of life updates (more coming soon)
• Third party data pipeline trigger (Tenzir)
• Made user inviting possible when user already exists.
• Minor UI and backend fixes

Full Changelog: 1.3.2...v1.3.3

1.3.1 - Runtime Debugger, Single Action Runs and Subflow Wait Fixes

This is an update to help fix some of the new issues 1.3.0 introduced.

• Made Notifications Org-Wide (per tenant) and happen and show up on more events
• Introduced Single Action executions (e.g. for Code Editor testing)
• Python can now be ran directly in the Code Editor if the Execute Python action is chosen (otherwise it's liquid)
• Added the Rerun execution API.
• Subflows with lists now do what they should again
• Added additional Proxy config to make it possible to split Proxy between External API's and Shuffle resources
• Conditions like Smaller Than and Larger Than now take type into account, meaning you don't need {{ ["list"] | size }} anymore.

What's Changed

• Syncing 1.4.0 with main by @frikky in #1266
• 1.3.1 by @frikky in #1283

Full Changelog: v1.3.0...1.3.1

1.3.0 - Stability, Statistics K8s & AI

🚀 Features 🚀

• Documentation -> OpenAPI Generator with Generative AI. This is an option on the /apps page for everyone.
• ShuffleGPT: We spent a lot of time testing out different usecases for generative AI throughout the year. The goal with Shuffle has always been to make programming more accessible in cybersecurity, and this is another step on our way there. As it's built on OpenAI (for now..), the goal is to release it on the open source side as well (with your own API key), and tightly building it into the features of Shuffle. First priority: Solve simple problems like "Get my last 10 last tickets and tell me about the most important one", and grow that into Workflow generation over time. More info in Q1 2024.
• Realtime Workflow Collaboration - Like Google Docs or Figma, Shuffle now has support for realtime workflow collaboration. This is still in Beta, and is disabled by default for now. We may use this feature to do realtime workflow update streaming to self-hosted versions to enhance support as well (your instance -> your browser -> shuffler.io -> our support in realtime).
• Health API! Adding a Status UI to Shuffle requires historic statistics for workflow performance to check for downtime. This runs checks regularly, both on cloud and onprem! /api/v1/health/stats + a feature that auto runs the health check on a regularly :)
• Statistics & Usage tracking: This temporarily lives under /admin?tab=billing until we find a nice home for it
• Orborus Utilization Monitoring: Orborus now checks properly whether the server has CPU and Memory left to run more workflows, and throttles if it doesn't.
• Added Workflow Run Explorer to make it easier to find workflow bugs over time. URL: /workflows/debug
• Added Workflow Versioning! Now you can see workflow revisions that you have made in the past.
• Oauth2 Application Permissions are now supported in the App Creator
• Added kubernetes support as a scaling option.
• Added automatic Docker image propagation between Shuffle workers if they are hosted on different machines
• We support ARM docker images from this version and forward for: 1.3.0+, latest and nightly (We have been supporting nightly for a while now)
• We fixed a liquid sanitisation related security vulnerability which led to network access to into the host machine through a poorly set up user workflow. Here is the PR for it for more details.
• And others!

What's Changed

• 1.2.0 by @dhaval055 in #1024
• Fixed compile time error. by @dhaval055 in #1018
• install-aws.md by @yogeshgurjar127 in #1042
• API for workflow actions recommendation by @dhaval055 in #1059
• Environment variable toggle support for liquidpy sanitization by @0x0elliot in #1127
• Fix swarm timeout between orborus and worker hosted on separate nodes by @0x0elliot in #1146
• 1.2.0 by @frikky in #1154
• 1.2.0 by @frikky in #1155
• Merging changes from 1.2.0 for cleanup by @frikky in #1153
• Bump golang.org/x/crypto from 0.0.0-20210921155107-089bfa567519 to 0.1.0 in /functions/onprem/worker by @dependabot in #1159
• Bump github.com/docker/docker from 23.0.0+incompatible to 23.0.3+incompatible in /functions/onprem/worker by @dependabot in #1158
• Bump github.com/docker/docker from 23.0.0+incompatible to 23.0.3+incompatible in /functions/onprem/orborus by @dependabot in #1157
• Bump golang.org/x/net from 0.0.0-20221014081412-f15817d10f9b to 0.7.0 in /functions/onprem/worker by @dependabot in #1156
• Bump golang.org/x/text from 0.3.7 to 0.3.8 in /functions/onprem/orborus by @dependabot in #1160
• Bump requests from 2.25.1 to 2.31.0 in /backend/app_sdk by @dependabot in #1163
• Bump golang.org/x/net from 0.0.0-20220722155237-a158d28d115b to 0.7.0 in /functions/onprem/orborus by @dependabot in #1161
• Bump github.com/docker/distribution from 2.8.1+incompatible to 2.8.2+incompatible in /functions/onprem/worker by @dependabot in #1164
• Bump github.com/docker/distribution from 2.8.1+incompatible to 2.8.2+incompatible in /functions/onprem/orborus by @dependabot in #1165
• Bump golang.org/x/crypto from 0.0.0-20210921155107-089bfa567519 to 0.1.0 in /functions/onprem/orborus by @dependabot in #1162
• feat: Ops Dashboard by @0x0elliot in #1181
• Verbose message added to improve UX for edit permission errors by @0x0elliot in #1189
• TypeError: "subloop" is read-only by @0x0elliot in #1200
• Bump urllib3 from 1.26.5 to 1.26.17 in /backend/app_sdk by @dependabot in #1209
• Bump gopkg.in/yaml.v3 from 3.0.0-20210107192922-496545a6307b to 3.0.0 in /functions/onprem/orborus by @dependabot in #1210
• Fix ICPL logo URL in README.md by @PeterDaveHello in #1217
• Quick ops dashboard fix by @0x0elliot in #1211
• Bump golang.org/x/net from 0.10.0 to 0.17.0 in /backend/go-app by @dependabot in #1212
• Bump golang.org/x/net from 0.7.0 to 0.17.0 in /functions/onprem/orborus by @dependabot in #1213
• Bump golang.org/x/net from 0.7.0 to 0.17.0 in /functions/onprem/worker by @dependabot in #1214
• Taking care of merge conflicts by @0x0elliot in #1221
• Shuffle on K8s by @dhaval055 in #1215
• Fix: making k8s set up feel better by @0x0elliot in #1222
• Bump urllib3 from 1.26.17 to 1.26.18 in /backend/app_sdk by @dependabot in #1218

New Contributors

• @yogeshgurjar127 made their first contribution in #1042
• @0x0elliot made their first contribution in #1127
• @dependabot made their first contribution in #1159
• @PeterDaveHello made their first contribution in #1217

Full Changelog: 1.1.0...v1.3.0

1.1.0 - Creators & Workflow Templates

📖 Resources 📖

• Loads of new Documentation about Scaling Shuffle
  Join the Discord and talk to other creators 💯

🚀 Features 🚀

• Added new welcome pages that helps you get set up by automatically downloading relevant apps and workflows (WAY less manual building)
• Added statistics for Workflows, Apps and total for Creators
• Added a Workflow templating system for generating workflows
• Added File Editing and Categories to handle new Detection methods
• Added and fixed Liquid filters
• New/Edited Apps now become version 1.1 and will return more about the request, including cookies
• Made Workflow sharing and discovery easier
• Added new Usecases that Workflows can now be mapped to
• Added distributed caching mechanism with Memcached
• Added statistics API that runs automatically in the background available on /api/v1/orgs/{orgid}/stats
• Added Documentation parser for Apps that loads from Github
• Added multiple redundancies for making sure Workflows execute in the right order
• Started development and Shuffle support for K8s to easily work in clusters and on AWS
• Added automatic app activation onprem for all released apps (search within a workflow)
• Added subflow export/import as part of parent workflow exports/imports
• And many more small changes :)

Additional changelog can be found in our Shuffle Shared Repository

What's Changed

• 0.6.1 by @frikky in #108
• 0.6.3 by @frikky in #113
• 0.6.5 by @frikky in #126
• 0.7.0 by @frikky in #136
• 0.7.1 by @frikky in #137
• 0.7.1 by @frikky in #147
• Stop using magic init function by @nicolaiskogheim in #148
• fix typo by @garanews in #166
• add .dockerignore file by @harduino in #154
• ability to select branch on workflows import by @harduino in #155
• improve checkout to custom branch on git clone by @harduino in #156
• recalculate authenticators stats on workflows import/de… by @harduino in #158
• dynamic app_sdk & worker image urls by @harduino in #157
• 0.7.5 by @frikky in #173
• Update README.md by @fakepop in #190
• 0.7.6 by @frikky in #191
• Import OpenAPI as file by @setheal in #182
• New app sdk images by @peterclemenko in #186
• 0.8.0 release by @frikky in #200
• Printf should have one argument per directive by @nicolaiskogheim in #201
• 0.8.3 by @frikky in #214
• Fix larger/smaller condition by @felipee07 in #233
• Shuffle v0.8.56 - Scalability and file fixing by @frikky in #234
• Fix should go on launch branch. by @amitkinfopercept in #255
• 0.8.60 by @frikky in #268
• Create CONTRIBUTING.md by @dgabriel123 in #265
• enhance .gitignore file by @harduino in #270
• pass SHUFFLE_DOWNLOAD_AUTH_BRANCH to backend service by @harduino in #271
• fix build with using golang:latest by @harduino in #272
• implement environment variable SHUFFLE_APP_FORCE_UPDATE by @harduino in #273
• frontend: fix Webpack issue on building Docker image by @harduino in #274
• fix workflows order on import by @harduino in #277
• improve work with Kubernetes by @harduino in #275
• App icon changes and new package for app icon crop by @amitkinfopercept in #282
• Grid view by @amitkinfopercept in #315
• 0.8.71 by @frikky in #322
• [Snyk] Security upgrade python from 3.9.1-alpine to 3.9.4-alpine by @snyk-bot in #338
• Improve for recursive casting by @felipee07 in #332
• Cortex responder by @azgaviperr in #366
• 0.9.0 RC1 by @frikky in #425
• Anonymous function point is the same value, so all the job params wil… by @helloannali in #450
• Update docker-compose.yml to use opensearch:1.0.0 instead of beta by @azgaviperr in #464
• Updated Code Editor and windows open-search guide by @gohil-jay in #500
• Fix #496: compatibility with cgroup v2 by @gc-advens in #501
• Launch by @frikky in #538
• Tried some linting by @Isoporhode in #545
• Ran eslint on Angular workflow by @Isoporhode in #551
• Fix comment for kali by @peterclemenko in #558
• Launch by @Isoporhode in #567
• Revert "Launch" by @frikky in #568
• Autoformating in fronted + bugfix by @Isoporhode in #570
• Updated textfield with modal based text-highlighting code editor by @gohil-jay in #580
• Update of Opensearch version by @azgaviperr in #608
• Fix 5 security vulns in deps by @gyrospectre in #638
• Update install-guide.md by @kp625544 in #645
• Updated the popup editor with indentation and variable detection feature by @gohil-jay in #623
• Added validation feature to expanded editor by @gohil-jay in #650
• Remove duplicate by @LaurenceJJones in #685
• Added line wrap feature in expanded editor by @gohil-jay in #681
• Added cursor tracking, tabs visuals; variable focus popup, highlighting and replacement by @gohil-jay in #727
• Added sub-variable detection and added scroll to editor popup by @gohil-jay in #768
• Updated text selection highlight to be visible over shuffle variables by @gohil-jay in #782
• Add GRPC Alpine variant SDK image by @weslambert in #812
• 1.0.0 by @frikky in #816
• Update install-guide.md by @dhaval055 in #845
• added temp dashboard page by @dhaval055 in #879
• added file editor by @dhaval055 in #896
• frontend file added by @jaydeepkotak in #951
• use official blackarch docker image by @peterclemenko in #955
• Remove some un-used Vars for SSR improving by @ayush0033 in #960
• chore: Fixed repo url to point to Shuffle org by @joostdecock in #984
• unused var removed by @ayush0033 in #976
• v1.1.0 - Creators & Workflow Templates by @frikky in #996

New Contributors

• @nicolaiskogheim made their first contribution in #148
• @garanews made their first contribution in #166
• @fakepop made their first contribution in #190
• @setheal made their first contribution in #182
• @peterclemenko made their first contribution in #186
• @felipee07 made their first contribution in #233
• @amitkinfopercept made their first contribution in #255
• @dgabriel123 made their first contribution in #265
• @snyk-bot made their first contribution in #338
• @azgaviperr made their first contribution in #366
• @helloannali made their first contribution in #450
• @gohil-jay made their first contribution in https://github.c...

1.0.0 - Launch!

📖 Resources 📖
See the release blogpost for Shuffle v1.0
See our support team in action
Join the Discord and talk to other creators 💯

🚀 Features 🚀
Added initial chat system to make it easier to reach support
Added the possibility of becoming a Creator, and search
Added new “Getting Started” section
Added App Framework to discover apps easily
Added Usecases to discover relevant workflows easier
Added Liquid Formatting to apps
Added SSO with SAML and OpenID PKCE
Added Swarm configuration possibility for Orborus
Added text/code/json/liquid editor to help making workflow easier
Added Ubuntu and GRPC images for app creation
Added OAuth2 possibility for Apps, whether from app creator or python
Added a lot more multi-tenancy usability features
Added new UI as documentation frontend
Added priority for when workflows execute to not clog up queues, and a button to clean the queue
Added multi-node copy-pasting of workflows

🐛 Bugs 🐛
Fixed a lot of bugs related to executions, UI bugs, missing buttons, API’s causing weird effects, bad JSON magic parsing, and much, much more.

What's Changed

• Tried some linting by @Isoporhode in #545
• Ran eslint on Angular workflow by @Isoporhode in #551
• Fix comment for kali by @peterclemenko in #558
• Launch by @Isoporhode in #567
• Revert "Launch" by @frikky in #568
• Autoformating in fronted + bugfix by @Isoporhode in #570
• Updated textfield with modal based text-highlighting code editor by @gohil-jay in #580
• Update of Opensearch version by @azgaviperr in #608
• Fix 5 security vulns in deps by @gyrospectre in #638
• Update install-guide.md by @kp625544 in #645
• Updated the popup editor with indentation and variable detection feature by @gohil-jay in #623
• Added validation feature to expanded editor by @gohil-jay in #650
• Remove duplicate by @LaurenceJJones in #685
• Added line wrap feature in expanded editor by @gohil-jay in #681
• Added cursor tracking, tabs visuals; variable focus popup, highlighting and replacement by @gohil-jay in #727
• Added sub-variable detection and added scroll to editor popup by @gohil-jay in #768
• Updated text selection highlight to be visible over shuffle variables by @gohil-jay in #782
• Add GRPC Alpine variant SDK image by @weslambert in #812
• 1.0.0 by @frikky in #816

New Contributors

• @Isoporhode made their first contribution in #545
• @gyrospectre made their first contribution in #638
• @LaurenceJJones made their first contribution in #685
• @weslambert made their first contribution in #812

Full Changelog: 0.9.25...1.0.0

0.9.25 - MSSP introduction

Join the Discord to see our updates in real time!

🚀 Features 🚀

Added initial CI/CD handlers
Fixed workflow view issues, and added workflow listing
Updated Opensearch to 1.1.0
Removed necessity for max_memory setting with Opensearch
Updated App SDK to handle LiquidPy with latest version
Added Oauth2 auth, both with and without App Creator
Added Gmail and Outlook triggers
Merged cgroup v2 compatibility issues
Added documentation popup for apps
Added file namespaces and download option for them (e.g. for Yara, Snort, other rule systems)
Added Webhook parser for Params to JSON to be used in workflows
Added start of a text editor within Workflows
Added resizer for right-side view in Workflow
Added startnode button for nodes in UI
Added suborgs and swapping (MSSP)
Added App Auth encryption possibility with environment variable (MSSP)
Added Single Signon possibility (MSSP)
Added basic Helm chart for K8s usage
Added notifications for failure of workflows (in UI, start workflow & email)
A new documentation UI

🕷️ Bugfixes 🕷️
LOOOADS - Too many to count, hidden inbetween new features 100

📖 Resources 📖
Loads of updated info in our documentation https://shuffler.io/docs/configuration

0.9.0

Features & Bugs TBD

0.8.71 release

TBD (looong list)