Merge pull request #3 from DecisiveAI/documentation-updates

task: update documentation

.github/workflows/mdbook.yml

@@ -30,6 +30,7 @@ jobs:
     runs-on: ubuntu-latest
     env:
       MDBOOK_VERSION: 0.4.21
+      MDBOOK_OUTPUT__HTML_GOOGLE_ANALYTICS: ${{ secrets.MDBOOK_GOOGLE_ANALYTICS }}
     steps:
       - uses: actions/checkout@v3
       - name: Install mdBook

.gitignore

@@ -1,4 +1,10 @@
 book
 dist
 .DS_Store
-.vscode
+.vscode
+.idea/workspace.xml
+.idea/vcs.xml
+.idea/mydecisive-docs.iml
+.idea/modules.xml
+.idea/shelf/Uncommitted_changes_before_rebase_\[Changes\]/shelved.patch
+.idea/shelf/Uncommitted_changes_before_rebase__Changes_.xml

NOT_COMPLETE/deployment.md

@@ -0,0 +1,4 @@
+## Deployment Guide
+- Step-by-step instructions for deploying EKS clusters locally or on-premises
+- Configuration steps for deploying OpenTelemetry components (collectors, exporters) on EKS
+- Configuration steps for deploying Prometheus for self-monitoring

NOT_COMPLETE/integration-instrumentation.md

@@ -0,0 +1,4 @@
+## Integration and Instrumentation
+- Guidance on instrumenting applications with OpenTelemetry SDKs for data collection
+- Configuration options for integrating OpenTelemetry with Prometheus for monitoring
+

OLD/ingress_old.md

@@ -0,0 +1,40 @@
+There are three entry points to the engine:
+
+1. **gRPC Load Balancer Endpoint** - enables gRPC telemetry to be sent to the engine.
+2. **http Load Balancer Endpoint** - enables http telemetry to be sent to the engine
+3. **console-ui Load Balancer Endpoint** - serves the MDAI Console for monitoring your engine instance
+
+In order to enable access allowed external access to the engine, you need to do the following:
+
+**Identify the Load Balancer DNS names**
+
+1. In the AWS Console, navigate to your [EC2 Load Balancers endpoints](https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#LoadBalancers) _Note: link above take you to region: us-east-1, you will need to change if you've deployed your engine in a different region_
+2. Locate the three load balancers required to run an MDAI Engine, there should be a gRPC, http, and console-ui load balancer. Make note of each of the load balancer's DNS names. ![![load balancers](../media/load-balancers.png)](../media/load-balancers.png)
+
+**Generate Certificates for secure connections to your MDAI Engine Instance**
+Follow the instructions in the [AWS ACM Certificate Request Guide](https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-request-public.html#request-public-console) to verify your domain and acquire a certificate for secure connections to your MDAI Engine instance.
+
+**Certificate Strategy**
+_Option 1: Default setup_
+
+1. **gRPC certificate** - enables a secure connection to the gRPC endpoint of your engine instance
+2. **non-gRPC cert** - enables a secure connection to the non-gRPC endpoint of your engine instance
+3. **console-ui certificate** - enables a secure connection to the engine console within your engine instance
+
+_Option 2: Wildcard setup_
+
+This will allow you to use a single certificate for all your endpoints.
+
+> Note: AWS doesn't provide free wildcard certificates.
+
+**Update the DNS Mapping**
+
+1. Log into your Domain Administrator Tool
+2. Update DNS mapping ([GoDaddy Example](https://www.godaddy.com/help/add-a-cname-record-19236)) to add a CNAME Record for each relevant Load Balancers created by engine.
+
+**DNS CNAME mappings**
+There is a 1:1 ratio for each load balancer endpoint to CNAME Record.
+
+1. **gRPC CNAME** - maps the the gRPC endpoint(s) of your engine instance to your custom-domain. We recommend using `grpc-#` for your CNAME record.
+2. **non-gRPC CNAME** - maps the the non-gRPC endpoint(s) of your engine instance to your custom-domain. We recommend using `non-grpc-#` for your CNAME record.
+3. **console-ui CNAME** - maps the MDAI Engine Console within your engine instance to your custom-domain. You'll be able to access the endpoint <cname>.<your-domain>.<domain-suffix> _e.g., https://mydecisive-console.example.com/_

README.md

@@ -10,6 +10,10 @@ Welcome to the docs! This repo is used to build [the decisive engine docs site](
 
 - This site is built using [mdBook](https://github.com/rust-lang/mdBook). You can install the `mdbook` utility by following [these instructions](https://rust-lang.github.io/mdBook/guide/installation.html).
 
+```sh
+cargo install mdbook mdbook-toc
+```
+
 ### Local development
 
 Use `mdbook serve` to spin up a local development server, at [http://localhost:3000](http://localhost:3000) by default. Make edits and reload the page to see updated content.

book.toml

@@ -1,6 +1,14 @@
 [book]
-authors = ["weird-bytecode"]
+authors = ["DecisiveAI, Inc."]
 language = "en"
 multilingual = false
 src = "src"
 title = "MyDecisive Engine Documentation"
+
+[output.html.fold]
+enable = true
+level = 2
+
+[preprocessor.toc]
+command = "mdbook-toc"
+renderer = ["html"]

src/JILL_ALL_DOCS.md

@@ -0,0 +1,18 @@
+---
+title: Documentation
+linkTitle: Docs
+menu: { main: { weight: 10 } }
+---
+
+MyDecisive.ai, also known as MDAI, is an Open-source framework
+for instrumenting, generating, collecting, and exporting telemetry data such as
+[traces](/docs/concepts/signals/traces/),
+[metrics](/docs/concepts/signals/metrics/), and
+[logs](/docs/concepts/signals/logs/).
+
+As an industry-standard, OpenTelemetry is
+[supported by more than 40 observability vendors](/ecosystem/vendors/),
+integrated by many [libraries, services, and apps](/ecosystem/integrations), and
+adopted by [numerous end users](/ecosystem/adopters).
+
+![OpenTelemetry Reference Architecture](/img/otel-diagram.svg)

src/SUMMARY.md

@@ -1,11 +1,94 @@
-# Summary
+# MyDecisive (MDAI) Engine
 
-[Introduction](./intro.md)
+- [Introduction]()
 
+  - [Overview](./intro/overview.md)
+  - [Architecture](./intro/architecture.md)
+  - [The MDAI Engine](./intro/intro.md)
+  - [Expectations](./intro/expectations.md)
+
+- [Installation]()
+
+  - [Supported Methods for Installing an MDAI Engine](./install/installation.md)
+  - [Local Install]()
+    - [Quick Start](./install/local/quick-start.md)
+  - [AWS Install]()
+    - [Introduction](./install/aws/start.md)
+    - [Prerequisites](./install/aws/prerequisites.md)
+    - [Configure AWS SSO](./install/aws/aws-sso.md)
+    - [Configure Your Engine](./install/aws/configure.md)
+    - [Deploy Your Engine](./install/aws/deploy.md)
+    - [Configure Ingress](./install/aws/ingress.md)
+    - [Verify Installation](./install/aws/verify.md)
+  - [Life Cycle Management]()
+    - [Disable the Engine](./install/aws/disable.md)
+    - [Enable the Engine](./install/aws/enable.md)
+    - [Destroy the Engine](./install/aws/destroy.md)
+  - [Testing your Engine]()
+    - [Generate & Receive Telemetry](./install/aws/generate-telemetry.md)
+
+- [Usage]()
+  - [Using the MDAI Engine Console](./console/mdai-console.md)
+
+<!--
 # Usage Guide
 
 - [Installation](./install/install.md)
   - [To an existing k8s cluster](./install/k8s-helm.md)
   - [To a new AWS EKS cluster](./install/k8s-cdk.md)
 - [Configuration](./Operation/config.md)
 - [Troubleshooting](./troubleshooting.md)
+
+
+
+#### Collector requirements
+
+#### Sizing and Scaling
+
+
+#### Processor Architecture
+
+
+
+### Configure
+### Install
+
+- Installation instructions for deploying EKS locally or on-premises
+- Setup instructions for OpenTelemetry and Prometheus components
+- Basic configuration steps
+
+-----------------------------------------------------------------------
+
+## Contributing
+- Guidelines for contributing code, documentation, or bug fixes
+- Code repository location (e.g., GitHub)
+- Contribution guidelines and code review process
+
+## Community Engagement
+- Links to community forums, mailing lists, or chat channels
+- How to get support (e.g., FAQs, support tickets)
+- Opportunities for community involvement
+
+## Risk and Disclaimers
+- Potential risks associated with using pre-alpha software
+- Disclaimer about stability, data loss, and other issues
+- Recommended backup and recovery procedures
+
+## Legal and Licensing
+- License information for the pre-alpha release
+- Copyright notices and third-party dependencies
+- Terms of use for early adopters
+
+## Future Development
+- Planned features and improvements for upcoming releases
+- Roadmap for transitioning from pre-alpha to alpha/beta stages
+- Community feedback integration process
+
+## Glossary
+- Definitions of technical terms and acronyms used in the documentation
+
+## Appendices
+- Additional resources for testers and contributors
+- Release notes for the pre-alpha version
+- Frequently Asked Questions (FAQs) specific to the pre-alpha release
+-->

src/console/mdai-console.md

@@ -0,0 +1,83 @@
+# The MyDecisive Engine Console
+
+<!-- toc -->
+
+## Introduction
+
+The MyDecisive Engine console provides an at-a-glance display of all the telemetry data that has flowed through your MDAI Engine in the last hour. It is a web application accessible via HTTPS within the engine.
+
+## Data Flow Overview
+
+The console offers three primary areas to help you understand how your data flows:
+Sankey Diagram: 
+A. The main section displays a Sankey diagram showing the data flow.
+B. Filters: Filters at the top allow you to customize what you see.
+C.Sidebar: On the right, a sidebar summarizes your data totals and data reduction.
+
+### Sankey Diagram
+<div style="text-align: center">
+<img src="../media/console-data-flow.png" alt="The MDAI Engine Console showing data flows" style="width: 600px" />
+</div>
+
+<br>
+
+The initial view (A) of the diagram shows how your different data types (Metrics, Logs, and Traces) flow from receivers to exporters, allowing you to see the overall data volumes flowing through the system.
+
+Hovering over any component highlights the specific data path(s), helping you zero in on a specific issue.
+
+<!--Screenshot of the hover view - make it exactly the same size and ratio as the initial screen-->
+<div style="text-align: center">
+<img src="../media/console-data-flow.png" alt="The MDAI Engine Console showing data flows" style="width: 600px" />
+</div>
+
+<br>
+
+## Pipeline View
+
+<div style="text-align: center">
+<img src="../media/console-pipelines.png" alt="The MDAI Engine Console showing pipeline configuration" style="width: 600px" />
+</div>
+
+<br>
+
+Clicking on any component in the Sankey diagram will filter and switch the view to a pipeline view. This shows all data types interacting with that component and highlights its configuration in the textual configuration view, allowing you to see the composition of your pipeline.
+
+<br>
+
+<div style="text-align: center">
+<img src="../media/console-viz-mode.png" alt="The MDAI Engine Console with pipelines mode switch highlighted" style="width: 600px" />
+</div>
+
+<br>
+
+As you view the console to understand your data flows, you can switch between Pipeline and Data flow view at any time to see the appropriate elevation.
+
+## Filters
+
+<div style="text-align: center">
+<img src="../media/console-filters.png" alt="The MDAI Engine Console showing content filter options" style="width: 600px" />
+</div>
+
+<br>
+
+The filters (B) at the top-right of the console allow you to constrain the data flow and pipeline views to specific components or data types. These filters can be combined, and options are automatically refined based on your selections.
+
+#### Top Talkers and Top Listeners
+
+The Top Talkers filter shows the top three receivers by data volume, and the Top Listeners filter shows the top three exporters by data volume.
+
+#### More Filters
+
+The More Filters menu allows you to filter your view by receiver, processor, data type, or exporter.
+
+
+## Summary of Data Totals and Data Reduction
+
+The console sidebar (C) displays:
+
+- A count of receivers, pipelines, and exporters running on your collectors
+- The total volume of data flowing through them in the last hour
+
+The sidebar updates with other changes on the page, allowing you to optimize your pipelines for maximum cost efficiency.
+
+> Note: Traffic flowing to debug exporters is not counted in data flow totals.