-
Improving Big Data: A Guide to Enhanced Pipelines
Seeking to improve your big data pipeline? This blog walks you through enhancements made to a client's system using Airflow Datasets, DAG Dependencies, Azure Durable Functions, and edge cases. Learn how we added functionality and flexibility by streamlining data integration, minimizing cost increases, and creating a scalable pipeline development process.
Located in
Blog
-
Real Python: Managing Big Data with Airflow
On Episode 142 of The Real Python Podcast, Calvin Hendryx-Parker, Six Feet Up’s CTO and AWS Community Hero, discusses a recent project that utilized Apache Airflow to make a statewide health system’s big data architecture faster and more manageable.
Located in
Company
/
News
-
Too Big for DAG Factories?
As your infrastructure scales up, how you go about managing all DAGs in Airflow becomes very important. One method would be to create a “DAG factory,” which can churn out thousands of DAGs dynamically from a single configuration file.
Located in
Blog
-
Building a Big Data Pipeline With Cloud Native Tools
Six Feet Up helped a statewide health system with 19 hospitals under its umbrella — and petabytes of data to aggregate daily — rebuild its infrastructure and implement a new world of cloud native and open source tooling, including Airflow, Spark, Delta Lakes (delta.io) and Terraform.
Located in
Projects
-
Data Warehousing in Microsoft Azure
Data Warehousing is the process of storing and processing large amounts of data from many disparate sources.
Located in
Blog