Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add execution contexts #69

Draft
wants to merge 21 commits into
base: 4.0.x
Choose a base branch
from
Draft

add execution contexts #69

wants to merge 21 commits into from

Conversation

leo-schick
Copy link
Member

In WIP feature discussion in #68

@leo-schick leo-schick self-assigned this Apr 25, 2022
@leo-schick
Copy link
Member Author

leo-schick commented Apr 26, 2022

Current status

the BashExecutionContext and SshBashExecutionContext works, but logging should be optimized

Sample app

local_config.py

from mara_pipelines.contexts.bash import BashExecutionContext, SshBashExecutionContext
patch(mara_pipelines.config.execution_contexts)(lambda: {
    'bash': BashExecutionContext(),
    'srv-mara-2': SshBashExecutionContext(
        host='srv-mara-2'
    ),
})

test.py

from mara_pipelines.pipelines import Pipeline, Task
from mara_pipelines.commands.bash import RunBash

test_pipeline = Pipeline(
    id='test',
    description="",
    context='srv-mara-2')

test_pipeline.add(
    Task(id='ls',
         description="",
         commands=[
             RunBash('ls -l')
         ]))

test_pipeline.add(
    Task(id='ls2',
         description="",
         commands=[
             RunBash('ls -l .venv')
         ],
         context='bash'),
    upstreams=['ls'])

Live execution output

image

Output after execution:

image

Open tasks

  • 🐛 the log message exit execution context '{context_alias}' is not saved in the database log
  • 🚀 use better logging, don't use print, see mara_pipelines.execution.py
  • 🚀 add logging events for enter/exit of a execution context
  • 🚀 start/stop docker container when not started when calling method __enter__
  • 🚀 show somehwere in the nodes page which execution context is used

leo-schick added a commit to leo-schick/mara-singer that referenced this pull request May 5, 2022
@leo-schick leo-schick force-pushed the feat-execution-context branch 2 times, most recently from 5954023 to e8e3c76 Compare May 5, 2022 14:22
@leo-schick
Copy link
Member Author

Currently known issue: when a execution context is used which is not defined in the configuration the pipeline execution does not throw any message and just hangs

@leo-schick leo-schick mentioned this pull request May 30, 2022
leo-schick added a commit to leo-schick/mara-singer that referenced this pull request Sep 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants