Update Airflow AWS MWAA deployment docs #3860

DimedS · 2024-05-08T15:58:42Z

Description

This PR introduces a section on deploying to Airflow AWS MWAA.

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

Read the contributing guidelines
Signed off each commit with a Developer Certificate of Origin (DCO)
Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added a description of this change in the RELEASE.md file
Added tests to cover my changes
Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

ankatiyar · 2024-05-10T10:18:40Z

Also reminder to add quotes to the dataset factory names in the docs ""

docs/source/deployment/airflow.md

ankatiyar · 2024-05-10T10:49:57Z

docs/source/deployment/airflow.md

+  client_kwargs:
+    aws_access_key_id: *********************
+    aws_secret_access_key: ******************************************
+```


I haven't tried this yet, but do the credentials in the local/credentials.yml work on the cloud platform? This file ideally shouldn't be uploaded to the cloud at all. Maybe setting up env variables?

I tried that approach with the local environment, and it works. We can change the environment from local to prod, but am I correct in understanding that there isn't a big difference?

For setting environment variables, I am using an env_var.sh file that is located in the same S3 bucket as the conf folder, which is currently packed into plugins.zip. Therefore, I don't think it makes much sense to use environment variables for security reasons because, ultimately, they will be stored in the same S3 bucket. However, this S3 bucket is secure and does not have public access, so I think it's not a problem.

ankatiyar · 2024-05-10T10:51:43Z

docs/source/deployment/airflow.md

+s3fs
+```
+
+6. Archive your `conf` folder into a `conf.zip` file and upload it to `s3://your_S3_bucket` for later use in the Airflow container. This file will be unzipped into the `plugins` folder within the MWAA Airflow container.


Worth noting that the kedro package command will create an archived conf-<package-name>.tar.gz file in the dist/ folder without the local conf which should be used here ^

Do you have any suggestions on how this file should be used? Should we simply unpack it to extract the conf folder, or should we run kedro run directly with the conf archive?

It doesn't need to be unpacked at all, OmegaConfigLoader can handle .tar.gz and .zip confs

I changed that but I don't know how to be with export KEDRO_LOGGING_CONFIG="plugins/logging.yml", I currently copy logging.yml separetely, but maybe you know better solution.

docs/source/deployment/airflow.md

Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: Dmitry Sorokin <40151847+DimedS@users.noreply.github.com>

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

merelcht

Generally looks good 👍

Maybe it's helpful to add links to the different "how to" sections in the introduction of the page, so users won't have to scroll to see what all the different options are.

docs/source/deployment/airflow.md

merelcht · 2024-05-16T15:38:11Z

docs/source/deployment/airflow.md

+```
+
+### Deployment on AWAA
+1. Archive your three files: `new_kedro_project-0.1-py3-none-any.wh`l and `conf-new_kedro_project.tar.gz` located in `new-kedro-project/dist`, and `logging.yml` located in `new-kedro-project/conf/` into a file called `plugins.zip` and upload it to `s3://your_S3_bucket`. This archive will be later unpacked to the `/plugins` folder in the working directory of the Airflow container.


I think we should provide the command/instructions on how to archive, because not all users might know how to do that.

Good point, I've added it!

merelcht · 2024-05-16T15:39:01Z

docs/source/deployment/airflow.md

+
+### Deployment on AWAA
+1. Archive your three files: `new_kedro_project-0.1-py3-none-any.wh`l and `conf-new_kedro_project.tar.gz` located in `new-kedro-project/dist`, and `logging.yml` located in `new-kedro-project/conf/` into a file called `plugins.zip` and upload it to `s3://your_S3_bucket`. This archive will be later unpacked to the `/plugins` folder in the working directory of the Airflow container.
+2. Create a new `requirements.txt` file, add the command to install your Kedro project archived in the previous step, and upload it to `s3://your_S3_bucket`:


I'm not quite sure I understand this. What do you mean with "add the command to install your Kedro project archived in the previous step"?

Agree, I rephrased that.

Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Dmitry Sorokin <40151847+DimedS@users.noreply.github.com>

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

DimedS · 2024-05-17T10:44:05Z

Generally looks good 👍

Maybe it's helpful to add links to the different "how to" sections in the introduction of the page, so users won't have to scroll to see what all the different options are.

Thank you for the review, @merelcht . I agree and have added an Introduction section.

merelcht

The added introduction is great and makes the guide easier to follow! 👍

docs/source/deployment/airflow.md

Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Dmitry Sorokin <40151847+DimedS@users.noreply.github.com>

ankatiyar

Great work @DimedS!

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

astrojuanlu

Thanks for this detailed guide @DimedS ⭐

The prose is necessarily long because there are several steps involved. One thing I'd consider for the future is that, rather than directing users up and down in the document ("Complete steps 1-4 from..."), it would be good to produce self-contained documents, even if they are redundant. Some pre-requisites for that:

Understand if there's something we can improve in Kedro to actually simplify the process Deployment support/Kedro deployer #2058 (comment)
Improve the usability of the documentation Improve the usability of the Framework documentation home page kedro-viz#1474 (it's already difficult to navigate these long pages)
If needed, use Sphinx includes to reuse content and avoid duplication

Other than that, I haven't tested these steps but I'm thrilled to see we have more detailed docs now 💯

Update airflow MWAA deployment docs

272c51e

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

DimedS linked an issue May 8, 2024 that may be closed by this pull request

kedro-airflow: Testing and Docs for kedro-airflow deployment on Major Cloud providers kedro-org/kedro-plugins#651

Closed

DimedS requested a review from ankatiyar May 8, 2024 15:58

DimedS added 2 commits May 8, 2024 17:01

Linting

9fa15c4

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

DAG Modification

aeb0110

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

ankatiyar reviewed May 10, 2024

View reviewed changes

DimedS and others added 4 commits May 13, 2024 10:51

Apply suggestions from code review

3b3b198

Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: Dmitry Sorokin <40151847+DimedS@users.noreply.github.com>

Merge branch 'main' into kedro-airflow-docs-deployment-cloud-providers

898939d

Merge branch 'main' into kedro-airflow-docs-deployment-cloud-providers

8ebe270

PR comments

62c917f

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

DimedS marked this pull request as ready for review May 15, 2024 15:59

DimedS requested review from yetudada and astrojuanlu as code owners May 15, 2024 15:59

Fix credentials

36a64a6

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

merelcht reviewed May 16, 2024

View reviewed changes

DimedS and others added 3 commits May 17, 2024 10:54

Apply suggestions from code review

65c2771

Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Dmitry Sorokin <40151847+DimedS@users.noreply.github.com>

Fix comments

4593bdd

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

Add introduction

a3e4f5c

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

DimedS requested review from ankatiyar and merelcht May 17, 2024 10:44

merelcht approved these changes May 17, 2024

View reviewed changes

docs/source/deployment/airflow.md Outdated Show resolved Hide resolved

DimedS and others added 2 commits May 17, 2024 14:06

Update docs/source/deployment/airflow.md

bc2d40e

Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Dmitry Sorokin <40151847+DimedS@users.noreply.github.com>

Merge branch 'main' into kedro-airflow-docs-deployment-cloud-providers

634c5a1

ankatiyar approved these changes May 17, 2024

View reviewed changes

ankatiyar mentioned this pull request May 20, 2024

kedro-airflow: Dataset Factories doesn't work on deployed project kedro-org/kedro-plugins#660

Closed

DimedS and others added 2 commits May 20, 2024 18:41

Merge branch 'main' into kedro-airflow-docs-deployment-cloud-providers

b4ce977

Update README.md

e3d38f1

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

DimedS merged commit 56961af into main May 20, 2024
10 checks passed

DimedS deleted the kedro-airflow-docs-deployment-cloud-providers branch May 20, 2024 16:40

astrojuanlu mentioned this pull request May 21, 2024

Deployment support/Kedro deployer #2058

Open

astrojuanlu reviewed May 21, 2024

View reviewed changes

DimedS mentioned this pull request May 24, 2024

fix(airflow): fix the link to correct deployment manual kedro-org/kedro-plugins#697

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Airflow AWS MWAA deployment docs #3860

Update Airflow AWS MWAA deployment docs #3860

DimedS commented May 8, 2024 •

edited

ankatiyar commented May 10, 2024

ankatiyar May 10, 2024

DimedS May 13, 2024

ankatiyar May 10, 2024

DimedS May 13, 2024

ankatiyar May 14, 2024

DimedS May 15, 2024

merelcht left a comment

merelcht May 16, 2024

DimedS May 17, 2024

merelcht May 16, 2024

DimedS May 17, 2024

DimedS commented May 17, 2024

merelcht left a comment

ankatiyar left a comment

astrojuanlu left a comment

Update Airflow AWS MWAA deployment docs #3860

Update Airflow AWS MWAA deployment docs #3860

Conversation

DimedS commented May 8, 2024 • edited

Description

Developer Certificate of Origin

Checklist

ankatiyar commented May 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

merelcht left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DimedS commented May 17, 2024

merelcht left a comment

Choose a reason for hiding this comment

ankatiyar left a comment

Choose a reason for hiding this comment

astrojuanlu left a comment

Choose a reason for hiding this comment

DimedS commented May 8, 2024 •

edited