Skip to main content
Back

Databricks' Lakeflow Wants AI Agents to Replace Your Entire Data Stack

Databricks overhauls Lakeflow at Data + AI Summit with AI agents that build, run, and fix your data pipelines — targeting Fivetran, Airflow, and Kafka in one move

  • Lakeflow overhaul at Data + AI Summit: Databricks announced a major evolution of Lakeflow, its unified ingestion, transformation, and orchestration platform.
  • 100+ connectors + free tier: Lakeflow Connect now supports 100+ native connectors with a free tier of 100 DBUs/day covering up to 100M records daily.
  • 5ms streaming, no Flink needed: Real-Time Mode for Spark Declarative Pipelines hits 5ms end-to-end latency, eliminating the need for a separate Apache Flink engine.
  • Genie ZeroOps — self-healing pipelines: A new background AI agent detects failures, performs root-cause analysis, proposes fixes, and validates them in a sandbox before human approval.
  • Kafka-compatible without Kafka: Zerobus Ingest now accepts Kafka producer traffic via compatible APIs — no code changes required for existing producers.
  • Direct threat to Fivetran, Airflow

Data engineering has always been the unglamorous backbone of AI. Before any model runs, someone has to wire up dozens of connectors, babysit pipelines, and debug failures at 2am. Databricks just announced a sweeping upgrade to Lakeflow, its unified data engineering platform, that aims to hand most of that toil to AI agents , and in doing so, take direct aim at a sprawling ecosystem of specialized tools.

One platform to rule the modern data stack

Lakeflow is designed as a unified platform for all of data engineering , from ingestion to transformation and orchestration , with every capability fully integrated and centrally governed by Unity Catalog, giving AI agents a single, trusted source of real-time context. The announcement, made at the Data + AI Summit in San Francisco, covers five major product areas that collectively replace what most teams currently stitch together from Fivetran, Apache Airflow, Apache Kafka, and Apache Flink.

The default 2026 enterprise data stack centers on Snowflake or Databricks for storage, dbt for transformation, Apache Airflow for orchestration, and Airbyte or Fivetran for ingestion. Databricks is now betting it can collapse that entire stack into one platform , and that the agentic era makes the timing right.

What actually shipped

  • Lakeflow Connect , 100+ connectors: Lakeflow Connect is expanding to support more than 100 native, managed connectors across enterprise applications, databases, file sources, and cloud storage , covering sources like Jira, GitHub, Confluence, SharePoint, Google Drive, and Outlook. A new free tier gives customers 100 free DBUs per day, supporting up to 100 million records daily across popular managed SaaS and database connectors.
  • Zerobus Ingest , Kafka without Kafka: Zerobus Ingest handles high-volume event data with no message bus required, delivering near real-time writes in under 5 seconds and throughput up to 100MB/s (over 10GB/s per table). Critically, Kafka-Compatible APIs (Beta) let existing Kafka producers push data straight to Databricks with no code changes required. Meta is already using it in production: "We cut our end-to-end pipeline latency to under a minute with Zerobus Ingest and Spark Declarative Pipelines."
  • Spark Declarative Pipelines , 5ms latency: Real-Time Mode (RTM) for Spark Declarative Pipelines is now in Public Preview, achieving end-to-end latencies as low as 5 milliseconds without the complexity and cost of managing separate engines like Apache Flink. This is a significant leap , previously, getting sub-second streaming required running a second specialized engine alongside Spark.
  • Genie Code + Genie ZeroOps , the agentic layer: Genie Code is now deeply integrated into every aspect of the Lakeflow user experience, letting you create ingestion connectors, build pipelines in Python and SQL, and develop jobs with tasks, triggers, and dependencies. The newly announced Genie ZeroOps is a purpose-built background AI agent that monitors and manages data and AI assets, detecting failures and performing root-cause analysis using data quality metrics, error logs, and lineage data from Unity Catalog. It generates proposed fixes and validates them in a safe, isolated sandbox environment , but applying a fix is done with human-in-the-loop, so you stay in control.
  • Lakeflow Designer , GA: Now generally available, Lakeflow Designer is a visual, AI-powered, no-code interface that empowers teams to build pipelines using a drag-and-drop canvas and natural-language prompts, letting business analysts build production-ready ETL pipelines without writing code. Every visual flow built in Designer natively runs on a production-ready Spark Declarative Pipeline, ensuring zero translation loss without complex handoffs.
  • Lakeflow Jobs , 50+ integrations: Lakeflow Jobs is Databricks' native orchestration engine for running complex production DAGs, scheduling, and firing AI agents, with data teams increasingly consolidating legacy orchestrators like Apache Airflow onto this single platform. New external orchestration capabilities let teams trigger Snowflake jobs, fire custom REST APIs, or manage Slack and PagerDuty alerts without rebuilding integrations from scratch.

The strategic bet underneath it all

This isn't just a feature release , it's a consolidation play. Databricks' most aggressive architectural strategy is to create a unified Data Engineering Stack, reducing fragmentation in enterprise data engineering services. The introduction of Lakebase and Lakeflow as an end-to-end native ETL orchestration engine is textbook business consolidation , and represents a direct attack on both Snowflake's Unistore architecture and the Microsoft Fabric integrated workspace model.

The timing is deliberate. AI agents need fresh, governed, real-time data to function reliably. To build a complete "enterprise memory" and ground AI agents like Databricks Genie, you need seamless access to the latest governed context spanning every area of your business. A fragmented stack of five different tools , each with its own auth, lineage, and monitoring , makes that nearly impossible. A unified platform makes it tractable.

Who wins, who feels pressure

For Databricks customers, the free tier on Lakeflow Connect and the consolidation of orchestration into Lakeflow Jobs represent real cost reduction opportunities. Teams building pipelines on Azure Databricks can build pipelines up to 25x faster while reducing ETL costs by up to 83%. Panasonic, one early adopter, reported cutting Power BI refresh times by 50% after migrating from a legacy ETL stack to Lakeflow Connect.

The pressure falls squarely on the point-solution vendors. Apache Airflow was built for the last era of data platforms; Lakeflow Jobs is positioned to power the next one with unified governance, event-driven pipelines, and AI-native workflows. Fivetran, which has built its business on managed SaaS connectors, now faces a free-tier competitor embedded directly in the platform where its customers already run their workloads. Confluent, which monetizes managed Kafka, faces Zerobus Ingest offering Kafka-compatible APIs with no migration required.

What's now possible that wasn't before

The most underappreciated part of this announcement is Genie ZeroOps. Self-healing pipelines have been a long-standing aspiration in data engineering , the idea that a system could detect a failure, identify the root cause, propose a fix, test it in isolation, and hand it to a human for approval. Genie Code supports the full data engineering lifecycle , from development and orchestration to monitoring and debugging , within a single agent experience, with tasks that used to take weeks now done in hours.

Lakeflow Connect managed ingestion pipelines now automatically record lineage from source tables to their destination tables in Unity Catalog, giving agents a fuller picture of where data comes from and how it moves. That lineage context is what makes ZeroOps possible at all , without knowing the full data graph, an agent can't safely reason about what broke or what a fix might affect downstream.

On real-world data science tasks, Databricks found Genie Code more than doubled the success rate of leading coding agents, going from 32.1% to 77.1%. That benchmark matters because it's the kind of number that determines whether a team actually trusts an agent to touch production pipelines , or keeps it in demo mode.

What to watch next

Several of the biggest features , Kafka-Compatible APIs, OpenTelemetry support, and scheduled tasks for Genie Code , are still in Beta or Public Preview. The real test will be whether Genie ZeroOps can handle the messy, underdocumented pipelines that exist in most production environments, not just clean demo scenarios. More than 20,000 organizations worldwide , including adidas, AT&T, Mastercard, and 70% of the Fortune 500 , rely on Databricks , which means the rollout of these agentic capabilities will happen at significant scale. If ZeroOps works reliably in the wild, it could meaningfully change the on-call burden for data engineering teams.