Apache Airflow—the open source workflow management platform for data engineering pipelines, used by a variety of leading companies such as Uber, Ford, and LinkedIn—is an invaluable tool to have in an organization’s orchestration toolbox. However, effectively scaling multiple Airflow deployments requires a deft hand, defined by optimized resource management, streamlined workflow development, and robust observability.
Naveen Sukumar, head of product marketing and DevRel, Astronomer, and Jacob Roach, field data engineer, Astronomer, joined DBTA’s webinar, Best Practices for Running Airflow at Scale, offering their expertise in how to properly manage large instances of Airflow while avoiding both performance bottlenecks and developer inefficiencies.
Apache Airflow is the most used open source tool for orchestrating data pipelines, according to Sukumar, growing in popularity from its origins in 2014. With 40 million monthly downloads, 3,300 monthly contributors, 40,700 GitHub stars, and a 60,000 Slack community, Apache Airflow’s popularity attests to its utility.
There are several key benefits to running Airflow at scale, noted Sukumar, including:
- Improved pipeline reliability
- Reduced downtime and incidents
- Lower operational costs
- Increased engineering productivity
Leading companies that have unlocked these benefits, such as Uber and Robinhood, employ the following patterns to enable Airflow at scale:
- Automation of environment provisioning
- Multi-tenancy and team isolation
- Rigorous CI/CD practices
- Upgrades and version control
- Robust monitoring and observability
- Strong security and governance frameworks
Sukumar and Roach then delved into each of these patterns, highlighting their value for scaling Airflow pipelines.
Regarding environment provisioning, Roach explained that one of the primary challenges of activating Airflow at scale is actually creating those very environments. Standing up a new Airflow instance is an incredibly time-consuming process, involving provisioning servers or Kubernetes clusters, configuring Airflow, setting up a metadata database, and more. When attempting to achieve scale, this becomes a major bottleneck, forcing teams to wait weeks for new environments if created manually.
Automation and managed platforms are the solution to this challenge, achieved through:
- Infrastructure-as-Code (Terraform, Helm, etc.) to script the creation of Airflow deployments
- A fully managed Airflow service such as Astronomer’s Astro, which provides a control plane to spin up new Airflow deployments on-demand in minutes, affording each team an isolated Airflow without the operational burden
- Standardizing an Airflow deployment template—e.g., containerize Airflow, use a pre-built image—so new instances are consistent
- Implementing auto-scaling for workers (Celery/K8s Executors) to handle workload spikes; if using one cluster, use pools/queues to allocate resources per team
Multi-tenancy is not one of Airflow’s strong suits out-of-the-box, noted Sukumar. For instance, when a single Airflow instance is shared between teams, resource utilization becomes imbalanced if one team’s workload is heavier than the other. Not only does this impact overall performance, it can also lead to one team inadvertently accessing or impacting another team’s data.
Sukumar offered the following as best practices for isolating workspaces:
- Isolate teams by providing separate Airflow deployments per team to segregate resources. This avoids noisy neighbor issues by design. Each team gets their own Airflow (or their own “workspace”) with dedicated resources.
- Integrate Airflow with your corporate identity provider (OAuth/OIDC, SAML) for SSO. Use SCIM (System for Cross-domain Identity Management) to automatically sync users and team assignments to Airflow roles. Managed Airflow services such as Astro support SSO/SCIM out-of-the-box.
- Establish clear RBAC roles for different user types (e.g., viewer, developer, admin) in each workspace and apply least privilege.
- If on one cluster, use resource quotas/pools per team to mitigate resource contention. Astronomer’s Airflow 3.0 introduces task execution isolation (no direct metadata DB access from tasks) which greatly improves multi-tenant safety.
This is only a snippet of the full Best Practices for Running Airflow at Scale webinar. For the full webinar, featuring the entire discussion of Airflow implementation patterns, more detailed explanations, a Q&A, and more, you can view an archived version of the webinar here.