Migrating Complex Workflows to Airflow

The company's custom web framework uses Python and SQLAlchemy to manage workflows for a variety of processes, including order fulfillment and lab work. Each of these workflows includes a variety of tasks and timelines. The framework was developed and built on over time, and, as a result, it became very complex and featured a significant amount of cruft.

Before Airflow could be implemented, Six Feet Up’s experts had to gain a deeper understanding of the custom framework to ensure a smooth transition.

Understanding a Custom Framework

First, Six Feet Up’s developers used Docker to create a local Airflow development environment. From there, they worked with the company's developers to answer initial questions before diving into the code base to figure out the inner workings of the software. This process included:

• finding the entry point into the code,
• tracing the code from that entry point to understand the workflow engine; and
• determining how individual workflows interacted with the workflow engine and how code was structured for each task.

Implementing Airflow

Six Feet Up's developers started by modifying the Airflow quickstart (generic steps used to implement Airflow) so that Airflow could access the company's package repository. This also allowed Airflow to be integrated into the company's data pipeline with very little setup work.

Then, Six Feet Up’s experts established a local development environment that linked Airflow to the company's Django-based backend services. Developers modified the integration tests to trigger DAGs in Airflow, and when the workflow migrations to Airflow were completed, modifications were made to those tests to confirm the data was accurate on completion of the DAG runs.

Migrating Workflows

With the groundwork done, Six Feet Up’s developers could begin the work of migrating workflows to Airflow. The company's workflows are organized into conductors, or umbrella workflows, that manage sub-workflows.

While the goal was for the final structure of the conductors and workflows to roughly match the existing structure, Six Feet Up’s experts consulted with the company's developers about condensing some of the workflows to make them run more quickly. This process required collaboration because while parallelizing tasks could boost efficiency, it could also make the workflow structure more complex, which would make onboarding developers more difficult.

Six Feet Up’s experts translated several workflows to Airflow, then used the company's user interface to run through the workflows. The company's Django-based backend services would then process the data coming through the workflow engine in Airflow. As a result, the Six Feet Up team could match the data with what would happen when that process was run through the company's custom web framework.