Silent killers of downstream efficiency—pipelines failing, data arriving late, and unnoticed schema changes—are difficult to catch with traditional observability tools. Often focusing more on the warehouse and not at the orchestration layer where these issues begin, enterprises need to invest in the next-generation of observability defined by detecting and resolving issues faster with orchestration-native approaches.
Ashley Kuhlwilm, senior product marketing manager, Astronomer, and Chris George, principal sales engineer, Astronomer, joined DBTA’s webinar, Why Data Observability Needs an Orchestration-First Approach, to illuminate how enterprises must shift to pipeline-aware observability to deliver trusted data for analytics and AI use cases.
While data teams are under growing pressure to deliver reliable, trusted data—faster and at lower cost, challenges such as rising complexity, tool sprawl, and fragile pipelines prevent its fruition, according to Kuhlwilm. As a result, data observability has grown from a “nice-to-have” idea to a mission-critical driver of trusted data, analytics, and AI; according to Gartner, “By 2026, 50% of enterprises implementing distributed data architectures will adopt data observability tools, up from just 20% in 2024.”
With real-time monitoring and a focus on proactive issue prevention, enterprises look to data observability to improve data quality, build trust, and ensure reliable analytics and AI outcomes. However, observability alone isn’t enough to guarantee reliable data, noted George, where standalone tools:
- Detect issues only after they hit the warehouse or dashboards
- Let data quality issues slip through pipelines unnoticed
- Increase the presence of fragmented tools that create blind spots and slow resolution
The solution is orchestration context, shedding light on the pipeline layer to drive truly proactive observability—not just end results of data after it’s already flowed through the system. Without orchestration context, pipeline failures may go unnoticed until they fully break downstream systems. As a result, teams spend excessive time investigating the failure, where unresolved issues can further lead to missed SLAs or late data delivery.
Orchestration context enables organizations to:
- Catch pipeline and task failures immediately before they impact downstream systems
- Get full visibility into data flow and execution context to quickly identify exactly where and why failures occur
- Resolve issues faster to maintain data freshness, meet SLAs, and deliver trusted, on-time data to downstream systems
George recommended that viewers look for the following in modern observability tools to cultivate proactive, orchestration-focused visibility:
- Pipeline-aware lineage and orchestration context: Surface real-time lineage, data quality issues, and task-level execution context to detect issues early and ensure trust in downstream data products.
- SLA tracking at the data product level: Enable data product owners to define, track, and enforce SLAs aligned to business outcomes to maintain accountability for on-time, reliable data delivery.
- Detailed, end-to-end visibility: Provide a high-level overview of system health and granular task-level insights to identify patterns, pinpoint failures quickly, and improve operational reliability.
- Cost visibility and performance optimization: Link resource consumption and spend directly to pipelines and data products to drive optimization, control costs, and improve operational efficiency.
- Low overhead and seamless platform integration: Integrate natively with your existing data stack to deliver observability, lineage, quality, and cost insights in one streamlined solution—without complex setup or added operational burden.
Kuhlwilm then introduced Astronomer’s two solutions, Astro and Astro Observe, which emphasize DataOps, or “a consistent framework for delivering high-quality, trusted data products quickly and at scale. At its core, it’s not just about tools or automation, it’s also about changing how teams work together and how they think about data as a strategic asset.”
Astro is a unified, fully managed platform for DataOps, designed to enable organizations to build, run, and observe data pipelines leveraging Apache Airflow. Its mission is simple, according to Kuhlwilm, helping “teams build and run trusted data pipelines so they can deliver reliable, on-time data without operational headaches.”
With Astro Observe, Astronomer delivers orchestration-native observability across the entire data stack. It enables pipeline-aware observability, built for Airflow; unified visibility across health, quality, and cost; and simplified operations with lower overhead.
This is only a snippet of the full Why Data Observability Needs an Orchestration-First Approach webinar. For the full webinar, featuring more detailed explanations, a Q&A, a demo, and more, you can view an archived version of the webinar here.