Waymo’s AI Secrets: How Continuous Evaluation and Human Oversight Drive Autonomous Vehicle Innovation

Text
Waymo, the self-driving car company under Alphabet, has established a rigorous framework for deploying AI that prioritizes continuous evaluation, human oversight, and alignment with real-world outcomes. This approach, described by Manasi Joshi, Waymo’s director of engineering for systems intelligence and machine learning, as “eval-forced development” or “eval-centric development,” ensures that AI projects are not deemed ready based on model performance alone but only after thorough, ongoing testing. According to Waymo, its autonomous vehicles have driven more than 220 million fully autonomous miles as of 2026, achieving 17 times fewer serious crash injuries per mile compared to human drivers. Joshi emphasized that the company’s evaluation processes are integral to engineering, with project readiness determined by the maturity of their testing frameworks. “The stage at which our projects are maturing can be easily kind of transpired based on the eval maturity that they showcase,” she said during a presentation at VB Transform 2026. Eval-centric development requires enterprises to treat evaluation as a continuous process, not a one-time task. Waymo’s methodology includes testing during model training, after training, and within simulations, combining datasets, performance metrics, and scalable infrastructure. This approach challenges companies to move beyond pre-launch testing and instead monitor systems as underlying models, user behavior, and data evolve. “Eval is not a one-time task to launch a model,” Joshi said. “It’s a continuous process spanning driving, simulation, and validation.”

Safety remains central to Waymo’s evaluation hierarchy. The company leverages first-party driving logs, third-party data, and simulations to expose its systems to billions of synthetic miles. Task owners select specialized data and metrics for high-risk scenarios, such as interactions with vulnerable road users or complex environments like construction zones. Joshi noted that similar principles apply to enterprise AI applications, where testing must address rare but high-impact failures—such as financial losses, legal liabilities, or reputational harm—rather than focusing solely on routine operations. Human oversight is another cornerstone of Waymo’s strategy. While automated systems handle much of the technical evaluation, final decisions about deployments and service-area expansions require input from internal safety leaders. “This is not AI-driven and completely automated and zero human oversight,” Joshi said. “Human lives are at stake.” This balance between automation and human judgment is critical for maintaining trust, particularly in safety-critical systems. Efficiency also plays a role in Waymo’s AI strategy. The company optimizes resource use across data storage, model training, and simulation, prioritizing “data efficiency” by selecting high-value training examples rather than relying on sheer volume. Waymo’s transition to transformers, large language models, and vision-language-action models reflects its focus on scalable, multimodal AI. However, the company’s infrastructure must balance real-time inference in vehicles with broader system requirements, a challenge shared by enterprises deploying AI in diverse environments. Joshi highlighted that even internal AI tools, such as agents used by engineers to analyze data or triage issues, require rigorous evaluation. These agents are not deployed without verification, as unreliable outputs could lead to wasted effort or flawed decisions. For enterprises, this underscores a broader lesson: agentic AI demands more than powerful models. Organizations must define clear objectives, use representative evaluation data, maintain continuous testing, and ensure accountability through named human decision-makers.

“Earning trust is supremely important,” Joshi said. Text
Subheading
The Role of Continuous Evaluation in AI Deployment

Waymo’s evaluation process extends beyond initial testing, incorporating ongoing assessments throughout a system’s lifecycle. This includes tests conducted during training, post-training validation, and simulations that mimic real-world conditions. Joshi explained that the company’s infrastructure is designed to handle these evaluations at scale, ensuring that performance metrics align with business outcomes rather than generic benchmarks. For example, Waymo’s simulations expose its systems to scenarios that are rare but critical, such as unexpected pedestrian behavior or adverse weather conditions. These tests are tailored to the specific risks of autonomous driving, but the methodology applies broadly. Enterprises developing AI agents for customer service, for instance, must similarly test edge cases—such as handling sensitive inquiries or resolving complex technical issues—rather than focusing solely on common interactions. Text
Subheading
Balancing Automation and Human Oversight

Despite its reliance on AI, Waymo maintains strict human oversight in critical decision-making. Safety leaders review software releases and expansions of service areas, ensuring that automated systems do not operate without accountability. This hybrid approach addresses a key challenge for enterprises: how to leverage AI’s efficiency without compromising safety or transparency. Joshi noted that fully automated deployment is not viable in high-stakes environments. “Human lives are at stake,” she said, highlighting the ethical and practical reasons for retaining human judgment. Text
Subheading
Efficiency Without Compromising Reliability

Waymo’s focus on efficiency does not come at the expense of reliability. The company optimizes resource use across data extraction, model training, and simulation, prioritizing performance over raw computational power. This strategy is particularly relevant as enterprises face growing demands for AI capabilities without proportional increases in infrastructure. By emphasizing data efficiency and model distillation, Waymo demonstrates how AI can be both scalable and sustainable. Its use of multimodal models, including vision-language-action systems, reflects a broader industry shift toward more versatile AI architectures. However, these advancements require robust evaluation to ensure they meet real-world requirements. Text
Subheading
Implications for Enterprise AI Adoption

Waymo’s approach provides a framework for enterprises seeking to deploy AI responsibly. Key takeaways include the need for continuous evaluation, human oversight, and alignment with business outcomes. As Joshi stated, “Earning trust is supremely important,” a sentiment that resonates across industries. For companies developing AI agents, the lesson is clear: success depends on more than technical capability. Organizations must invest in evaluation infrastructure, define clear objectives, and maintain accountability. Waymo’s experience underscores that AI deployment is as much about process as it is about technology.