Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DTT2 - Workflow engine - Implement observability plugin execution ID #5091

Open
4 tasks done
Tracked by #4837
QU3B1M opened this issue Mar 11, 2024 · 4 comments · May be fixed by #5115
Open
4 tasks done
Tracked by #4837

DTT2 - Workflow engine - Implement observability plugin execution ID #5091

QU3B1M opened this issue Mar 11, 2024 · 4 comments · May be fixed by #5115
Assignees

Comments

@QU3B1M
Copy link
Member

QU3B1M commented Mar 11, 2024

Epic: #4837


Description

The objective of this issue is to investigate and analyze the implementation of an execution_id as identifier of the workflow execution in order to improve the tracking for the Observability module

Tasks

  • Investigate how to implement an unique ID for the execution.
  • Implement the ID generation in the engine.
  • Implement auto-injection of the ID in the workflow as global variable (to call this id inside the yaml without having it defined explicitly).
  • Update the influxdb plugin to use this execution ID as primary key.
@QU3B1M
Copy link
Member Author

QU3B1M commented Mar 11, 2024

  • Defined usage UUID as unique identifier for the worklow execution
  • Applied required changes to configure this execution_id as global variable in the worklfow

@QU3B1M
Copy link
Member Author

QU3B1M commented Mar 18, 2024

  • Implement 'native_vars' dict in the workflow file to store all the engine's global/magic variables.
    deployability/modules/workflow_engine/workflow_processor.py
    class WorkflowFile:
        ...
        def __init__(self, workflow_file_path: Path | str, schema_path: Path | str = None, native_vars: dict = {}) -> None:
            ...
            self.__native_vars = native_vars
            ...
        def __process_workflow(self):
            ...
            workflow_vars = self.workflow_raw_data.get('variables', {})
            variables = {**workflow_vars, **self.__native_vars}
            for task in self.workflow_raw_data.get('tasks', []):
                task_collection.extend(self.__expand_task(task, variables))
            ...
  • Update WorkflowProcessor to generate the execution_id and inject the native vars into the other classes (for now only to WorkflowFile)
    deployability/modules/workflow_engine/workflow_processor.py
    class WorkflowProcessor:
        ...
        def __init__(self, workflow_file: str, dry_run: bool, threads: int, log_level: str = 'INFO', schema_file: Path | str = None):
            ...
            # Initialize the intenal variables.
            self.__execution_id = self.__generate_execution_id()
            # Initialize the public variables.
            self.task_collection = WorkflowFile(workflow_file, schema_file, self.get_native_vars()).task_collection
            ...
        def get_native_vars(self) -> dict:
            return {'execution_id': self.__execution_id}
    
        def __generate_execution_id(self) -> str:
            return str(uuid.uuid4())
  • Update InfluxDBReporter to use the execution_id if defined or generate a new one.
    deployability/plugins/influxdb_reporter/pytest_influxdb/reporter.py
    class InfluxDBReporter:
        ...
        def __init__(self, config: Config, config_file: str = None) -> None:
            ...
            self.execution_id: str = config.getoption('--execution-id') or str(uuid.uuid4())
            ...
    
        def __get_report_body(self, test_report: TestReport, datetime: str) -> dict:
            ...
            tags = {
                'execution_id': self.execution_id,
                'test': test_report.fspath,
                'markers': self.__get_pytest_marks(test_report.keywords),
                'when': test_report.when,
            }
            ...

@fcaffieri fcaffieri added level/task Task issue and removed level/subtask Subtask issue labels Mar 18, 2024
@QU3B1M
Copy link
Member Author

QU3B1M commented Mar 20, 2024

  • The native/magic vars are also configured as environment vars to each execution, this way any module/script can access this values by checking the Env vars.
  • A new task tag env was implemented in the workflow structure, this tag lets the user set environment variables for the execution of the task.
    Example:
      - task: "test-magic-vars"
        description: "Use a simple script to the new magic & env variables."
        do:
          this: process
          with:
            path: python3
            args:
              - /home/quebim/Wazuh/repos/wazuh-qa/deployability/test.py
            env:
              TEST: "Im testing"
    

@QU3B1M QU3B1M linked a pull request Mar 20, 2024 that will close this issue
@rauldpm rauldpm changed the title DTT1 - Iteration 3 - Observability module - Implement execution ID DTT2 - Iteration 3 - Observability module - Implement execution ID Mar 20, 2024
@pro-akim
Copy link
Member

Review Notes

LGTM

@rauldpm rauldpm changed the title DTT2 - Iteration 3 - Observability module - Implement execution ID DTT2 - Observability module - Implement execution ID Mar 21, 2024
@rauldpm rauldpm changed the title DTT2 - Observability module - Implement execution ID DTT2 - Workflow engine - Implement observability plugin execution ID Mar 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants