Enterprise cloud platforms scaling to millions of concurrent users frequently encounter tightly coupled microservices that trigger cascading failures beyond the reach of static runbooks or threshold alerts. During high-velocity events such as Black Friday flash sales, Site Reliability Engineers must rapidly diagnose noisy distributed telemetry to prevent outages costing millions of dollars per minute. While generative AI copilots have improved code authoring and log summarization, they often struggle to reliably remediate live production issues. Rather than unverified pull requests or generic troubleshooting tips, on-call engineers require governed, self-evolving systems capable of addressing active infrastructure bottlenecks.
To tackle this challenge, Google Cloud developed a research reference architecture codenamed Project Chrysalis, combining three foundational research breakthroughs into a unified, production-grade system running on the Gemini platform and the Agent Development Kit. During simulated Black Friday cache-stampede outages, this governed architecture dramatically cut the time-to-first-mitigation from 15 to 45 minutes down to 1.8 seconds. Furthermore, it reduced HTTP 5xx error rates down to 0.0%, dropped P99 latency significantly, and evaluated 400 counterfactual traffic policies in BigQuery in approximately 0.5 milliseconds.
Operating without unconstrained black-box risks, the architecture utilizes Google Cloud's defense-in-depth security stack to keep autonomous actions cryptographically bound. Model Armor sanitizes incoming telemetry against prompt injections and data leaks, while Google Cloud Key Management Service and Binary Authorization middleware sign synthesized patches and block unverified code. Additionally, the Open Policy Agent Policy Controller enforces strict SLA guardrails, and Dual-Custody Human-in-the-Loop governance mandates explicit SRE sign-off before permanent changes alter the agent hierarchy.
Conventional AI agents remain frozen at deployment with fixed system prompts and pre-registered tools, but self-evolving architectures extend these baseline capabilities at runtime. When novel infrastructure bottlenecks strike, the system synthesizes situation-specific mitigation tools, scores the telemetry impact of every intervention, and codifies verified guardrails for future incidents. This approach addresses the SRE Trilemma by balancing exploration safety, structural adaptability, and multi-step credit assignment through three core research pillars deployed on Google Cloud.
The first pillar involves serverless counterfactual simulation known as Dream-RSI, where production request traces are exported to Google BigQuery to replay hundreds of candidate token-bucket configurations in sub-millisecond vectorized memory sweeps. The second pillar employs tri-level co-evolution, named APEX, which separates agent evolution into governed timescales to prevent fragile band-aids or system instability. The third pillar utilizes step-level Shapley attribution via AgentEvolver, computing marginal Shapley credit across each execution span using pre- and post-intervention telemetry deltas to isolate beneficial patches from erroneous ones.
