An Intro to Airflow at January 2022 IndyPy

We kicked off the 2022 season of IndyPy — Indiana’s largest Python meetup founded in 2007 by Six Feet Up CTO and AWS Community Hero, Calvin Hendryx-Parker — with an “Introduction to Airflow.” In this presentation and live demo, Rocco Pascale, a Field Engineer at Astronomer, demonstrates how anyone with Python knowledge can deploy a workflow using Apache Airflow — an open-source platform created to programmatically author, schedule and monitor workflows.

As Rocco explains, by writing a single DAG (Directed Acyclic Graph) file in Python using existing provider packages — Astronomer currently has 82 providers and is regularly expanding its library — users can begin to define complex relationships between data and actions.

Read our live Twitter thread, and watch the video below for a deeper dive into Airflow, DAGS, tasks, operators and the core components that make Airflow run including the webserver, scheduler, metastore, executor, worker and triggerer.

Watch the presentation:

Links and Resources

• The Astronomer Registry includes a full list of providers, sample DAGs and more: https://registry.astronomer.io
• Rocco Pascale’s GitHub: https://github.com/ReadytoRocc
• Airflow Guides: https://www.astronomer.io/guides
  • Deferrable Operators: https://www.astronomer.io/guides/deferrable-operators