Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Expose "meta" model config dictionary on TaskInstance #698

Closed
wseaton opened this issue Nov 20, 2023 · 3 comments · May be fixed by #700
Closed

Expose "meta" model config dictionary on TaskInstance #698

wseaton opened this issue Nov 20, 2023 · 3 comments · May be fixed by #700
Labels
area:config Related to configuration, like YAML files, environment variables, or executer configuration area:logging Related to logging, like log levels, log formats, error logging, etc

Comments

@wseaton
Copy link

wseaton commented Nov 20, 2023

I'm working on integrating a PagerDuty setup with a Cosmos repo:

def on_failure_callback(context: Context):
    summary = f"Airflow: DBT DAG {context.get('task_instance').dag_id} task failure"
    severity = "error"
    source = context.get("task_instance").task_id

    dedup_key = f"{context.get('task_instance').dag_id}-{context.get('task_instance').task_id}-{context.get('execution_date')}"

    pd_hook = PagerdutyEventsHook(pagerduty_events_conn_id="default_pagerduty")
    pd_hook.send_event(
        summary=summary,
        severity=severity,
        source=source,
        group=context.get("task_instance").dag_id,
        dedup_key=dedup_key,
        links=[{"href": context.get("task_instance").log_url, "text": "Airflow Log"}],
    )

This works well, but I'd love to be able to have metadata from a model like the .meta.owner field or even the tags attached to the Context object to be able to more intelligently route alerts within my callbacks. Is this possible to implement?
@dosubot DosuBot
Copy link

dosubot bot commented Nov 20, 2023

🤖

Hello @wseaton,

I'm Dosu, a friendly bot assisting with the Cosmos repository. I'm here to help you navigate through issues, answer your questions, and guide you on your contribution journey while we wait for a human maintainer.

I'm currently reviewing your issue regarding the exposure of the "meta" model config dictionary on TaskInstance. I'll provide a detailed response soon.

@dosubot dosubot bot added area:config Related to configuration, like YAML files, environment variables, or executer configuration area:logging Related to logging, like log levels, log formats, error logging, etc labels Nov 20, 2023
@dosubot DosuBot
Copy link

dosubot bot commented Nov 20, 2023

🤖

Hello @wseaton,

I'm Dosu, a bot that is here to assist you with Cosmos repository. I'm designed to help with bug resolution, answer queries, and guide you on how to contribute to this project. While we wait for a human maintainer, feel free to ask me anything related to the repository.

Based on the current state of the Cosmos repository, there isn't a built-in mechanism to attach metadata to the TaskInstance object. The metadata is created for tasks and used to create Airflow tasks or task groups, but it's not being attached to the TaskInstance object. You can see this in the cosmos/airflow/graph.py file.

As for accessing the .meta.owner field or tags from the Context object, I wasn't able to find a direct way to do this within the Cosmos codebase.

However, it's possible to extend the functionality of Cosmos to include this feature. You could potentially modify the TaskInstance object to include the metadata you need, or create a new object that inherits from TaskInstance and includes additional metadata fields. This would require a good understanding of both the Cosmos and Airflow codebases, and careful testing to ensure that the changes don't introduce any unexpected behavior.

Please note that this is just a suggestion and might not be the best solution for your specific use case. It's also possible that there's a better solution that I wasn't able to find. If you need further assistance, please provide more details about your setup and requirements.

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

@wseaton wseaton mentioned this issue Nov 20, 2023
Draft
2 tasks
@dosubot DosuBot
Copy link

dosubot bot commented Feb 19, 2024

Hi, @wseaton,

I'm helping the Cosmos team manage their backlog and am marking this issue as stale. From what I understand, the issue requests the exposure of the "meta" model config dictionary on TaskInstance to enable the use of metadata such as the .meta.owner field or tags within callbacks for more intelligent alert routing. Dosu has provided information on the current state of the Cosmos repository and suggested potential ways to extend the functionality of Cosmos to include the desired feature. They emphasized the need for a good understanding of both the Cosmos and Airflow codebases and careful testing to ensure that the changes do not introduce unexpected behavior.

Could you please confirm if this issue is still relevant to the latest version of the Cosmos repository? If it is, please let the Cosmos team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you!

@dosubot dosubot bot added the stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed label Feb 19, 2024
@dosubot dosubot bot closed this as not planned Won't fix, can't repro, duplicate, stale Feb 26, 2024
@dosubot dosubot bot removed the stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed label Feb 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:config Related to configuration, like YAML files, environment variables, or executer configuration area:logging Related to logging, like log levels, log formats, error logging, etc
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Draft: Add config to TaskInstance context wseaton/astronomer-cosmos
1 participant
@wseaton