The Agent Cloud: Databricks’ Bet on the Future of AI — Matei Zaharia and Reynold Xin
TL;DR
Matei Zaharia and Reynold Xin detail Databricks' open-source 'Agent Cloud' platform (Omnigen), arguing that standardized protocols and persistent infrastructure—not just better models—will determine which enterprises successfully deploy collaborative, secure AI agents at scale.
☁️ The Agent Cloud Architecture 3 insights
Open-source protocols beat proprietary fragmentation
Databricks open-sourced Omnigen to establish a standard API for agent interoperability, betting that network effects from community integrations will win over closed systems, similar to Spark's ecosystem strategy.
Universal abstraction across AI harnesses
The platform normalizes disparate coding agents (Claude Code, Codex, OpenAI SDK) into a single interface, insulating developers from constant API churn while enabling portable, multi-model workflows.
Infrastructure over 'vibe coding'
While AI enables rapid prototyping, production multi-party coordination requires designed protocols and APIs rather than ad-hoc generated code that cannot maintain state or security across organizational boundaries.
🔧 Solving Real Engineering Friction 3 insights
From internal frustration to infrastructure
The product originated from Databricks engineers building custom agent UIs and literally tethering laptops to keep cloud coding sessions alive, revealing critical gaps in persistent compute and cross-team collaboration.
Cloud sandboxes replace local development
Omnigen provides persistent, shareable cloud environments that eliminate the need to keep personal machines running to maintain agent sessions, effectively decoupling development from individual hardware.
Security as a core design constraint
Unlike personal coding tools, enterprise agents require server-side hosting with SSO authentication and audit trails to gain security team approval for accessing production data.
📊 Data Infrastructure Meets AI 3 insights
The data-first AI paradigm
While generic AI reasoning capabilities improve rapidly, unlocking enterprise value requires getting data into unified infrastructure first—'magic' emerges only when AGI sits atop accessible, well-governed data.
Consolidation beyond the modern data stack
Just as the data stack evolved from fragmented best-of-breed tools into unified platforms, agent infrastructure is converging toward integrated systems rather than chained frameworks requiring excessive vendor coordination.
Agent coordination as the new OS layer
The architecture functions like an operating system for agents, providing the coordination layer between AI models, tools, and users that makes multi-agent workflows portable, observable, and scalable across organizations.
Bottom Line
Enterprises should prioritize open, standardized agent infrastructure that unifies data access, persistent cloud compute, and security controls to capitalize on rapidly improving AI models, rather than building fragmented, non-portable agent workflows that cannot scale beyond individual developers.
More from Latent Space
View all
AI Security After Codex and Claude Code — Zico Kolter & Matt Fredrikson, Gray Swan
Gray Swan co-founders Zico Kolter and Matt Fredrikson explain why AI systems require a fundamentally different security approach than traditional software, highlighting how their automated red teaming system 'Shade' has begun to outperform human experts at finding model vulnerabilities. They emphasize the urgent need to treat AI agents as inherently untrusted entities capable of correlated failures across the software ecosystem.
⚡️Every product of the future will be a living system — Ronak Malde, Trajectory.ai
Ronak Malde explains leaving DeepMind (and $2 billion in acquisition earnings) to found Trajectory.ai, arguing that AI products must evolve from static tools into "living systems" that continually learn from real-world user corrections across enterprise verticals like legal and finance.
The AI Frontier: from FLOPs to Megawatts — Anjney Midha, AMP
Anjney Midha argues that AI infrastructure is facing a crisis of inefficiency and cultural misalignment, proposing that compute be treated as a utility through an Independent System Operator model that pools multi-cloud resources while embedding community incentives directly into unit economics.
🔬 The Limits of AI in Science - Why We Need Self-Driving Labs — Joseph Krause, Radical AI
Joseph Krause explains why AI alone cannot discover new industrial materials—unlike biology, alloys cannot be represented as simple strings and require physical ground truth across synthesis, microstructure, and processing. Radical AI is building self-driving labs to close the loop between AI hypothesis generation and automated experimentation, aiming to compress the 15-30 year materials development timeline.