[Documentation SIG] Notes from 11/3/2020 Call

[ryw]

In attendance

• @ryw @kaxil @pcandoalmeida @jward-bw

Rough notes

• There are too many links at the top level of Airflow docs. We should establish concise, clear top level groupings for documentation. This will require a change to the website to allow for deeper nesting of the main docs nav.
• Right nav shouldn't repeat structure of the left nav
• How-To guides are improved in latest docs, will be nice to get 2.0 out so they show up on the main website
• Improvements we make to doc structure will not be back-ported to 1.10.x docs - a bit of motivation to upgrade to Airflow 2
• We feature some "core assumptions" content that is intuitive coming into Airflow - for example scheduler interval, xcom, etc.
• We should expand and improve content around using Airflow on Kubernetes. It's common for newcomers to feel that it's not a core capability.
• FAQ content should be moved to where it belongs.
• Improve content on DAG Visibility - it's thinly documented how to set up Airflow to give certain users visibility of certain dags, creating roles in the RBAC, etc.
• There is a big index https://airflow.apache.org/docs/stable/#content which is duplicated by the left nav. If this content is on the docs home page, perhaps we should not show full detail.
• Rather than referring to docs built from master branch as latest - we should call these docs dev or devel or edge or master -- latest implies "latest release"
• Create a clear top-level section on contributing to Airflow covering breeze, setting up virtual environment, etc.
• Top-level section on "Setting up Airflow": Local w/ Docker, Local w/ Python, Local w/ Kubernetes, Production K8s, Production Non-Kubernetes with a matrix of solutions across the major clouds (GCP, AWS, Azure). We could summarize/endorse best practices, linking to external projects/services.
• A while back we decided not to position Airflow as a Kubernetes/Cloud-native project, focusing more on it's generic pythonic nature. However a lot of users are using or want to use K8s with Airflow, so we should probably "move up" the visibility of how well Airflow pairs with K8s.
• Running locally with docker-compose is left up to the user to figure out. Maybe Airflow CLI should support this, possible that Astronomer could work on this as we have this functionality in our CLI. Confluent Connect has nice sample docker files, we should look to that for inspiration.
• We should make it clear on why you should choose each executor. In particular, with the addition of CeleryKubernetesExecutor how do we recommend what Executor you should use? Let's build some sort of comparison table.
• We should have a comparison guide vs. other modern orchestrator options: Dagster, Luigi, Prefect
• Guide: How to use Airflow w/ Ray Project (could mention Modin)
• Expand ecosystem page
• Deep dive guides into Airflow internals - scheduler, dag serializer
• Continue to build out guides on how to use each operator
• Reorganize operator content to put the parts together (right now content is spread into multiple sections)
• Searching in docs is broken
• Configurations content is very important, but is buried way at bottom of the list
• Sort configurations alpha, fix right column (too narrow)
• Move providers to top level - reconcile w/ the website content of providers as well (we prob need a whole meeting to figure out an optimal solution to this) - also related to item above "Reorganize operator content to put the parts together"
• Should we move some Astronomer Guides content to the docs

---

We have github issues for some of this, other things we don't. We didn't discuss next steps beyond publishing these notes. Next meeting we could spend some time sorting/prioritizing these issues, but don't let that stop you from taking some action in the mean time 👍