InfoQ Homepage News Datadog Brings Production Experimentation to AI Agent Observability.
Nov 17, 2026 3 min read.
Sources indicate that datadog is extending its Agent Observability platform with Online Experiments, allowing engineering teams to compare AI agent versions against real production traffic rather than relying exclusively on predefined evaluation datasets. The capability combines agent evaluation scores with operational measurements such as latency, token consumption, execution cost, and errors, helping teams establish whether changes to models, prompts, retrieval strategies, or tools actually improve an agent.
The announcement addresses a growing challenge in agent development: a change that performs better in a controlled test may behave very differently when exposed to unpredictable user requests. Datadog's approach brings controlled experimentation, production telemetry, and execution traces together, giving teams evidence of both the outcome and the behaviour behind it.
Offline evaluations remain an crucial part of agent development. Teams can run different versions against predefined datasets, assess responses and identify known regressions before deployment.
It has been revealed that however, these evaluations are only as representative as the datasets and scoring criteria behind them. Upcoming user workflows, unexpected requests, and changing tool responses can expose weaknesses that existing test cases never anticipated.
Online experiments address this limitation by evaluating agent behaviour against real user interactions. Teams can compare a control version with a modified variant by randomly distributing production traffic between them, reducing the influence of differences in user requests.
This provides a more realistic view of how an agent performs under actual workloads, while also helping teams identify newly introduced scenarios that should eventually become part of their offline regression datasets.
Datadog integrates its Agent Observability evaluators with its Experiments capability, allowing evaluation scores to be compared alongside conventional operational metrics.
Teams can measure response quality, accuracy, and completeness while tracking execution duration, token consumption, estimated cost, and workflow errors.Feature Flags manage traffic allocation, letting engineers expose a change to a small percentage of customers before rolling it out more widely.
According to recent reports, this is important because improvements rarely come without trade-offs. A cheaper model might require more retries, while a revised retrieval strategy could improve evidence quality but increase execution time.
Sources indicate that datadog also connects experiment results to individual agent traces. Engineers can inspect tool calls, supplied arguments, and returned results to understand why a particular variant behaved differently.
According to the article, the experiment compared GPT-5.6 Luna against GPT-6 Luna, splitting traffic evenly between the two. An LLM-based evaluator scored investigation quality from one to five, considering accuracy, completeness, and whether conclusions were supported by available evidence.
They claim the results showed a 44.3% reduction in estimated cost for the newer model, with quality scores remaining close at 3.84 and 3.86, respectively.
Recent findings suggest that however, the operational results revealed potential concerns. The newer model highlighted estimated increases in execution duration and workflow errors, although the confidence intervals were too broad to establish definitive regressions.
Inspecting the traces revealed that the newer model also supplied invalid tool parameters more frequently, usually recovering after retries. This behaviour was not sufficiently visible in the overall quality score.
Datadog used that finding to improve tool handling before proceeding with the model change. The failed calls also provided candidates for additional offline regression tests.
Datadog is not alone in trying to solve this challenge. Tools such as LangSmith and Langfuse provide agent tracing and evaluation capabilities, while experimentation platforms such as LaunchDarkly support controlled feature rollouts. Datadog's approach connects these concerns within its existing observability ecosystem, linking evaluation results directly to production experiments and execution evidence.
According to recent reports, datadog's broader proposition is that online experiments can create a feedback loop between production behaviour and development testing. Unexpected failures become new regression cases, while subsequent experiments establish whether changes actually resolve them.
Sources indicate that the important shift is from asking whether an agent passes a predefined evaluation to establishing whether it performs better under real conditions. For agent engineering, production experimentation may become as important as traditional software testing.
Microsoft Ignite 2026; Explore What's Next in AI and Software Engineering - Learn how to architect agentic systems, modernize cloud infrastructure, and tackle emerging engineering challenges. Get practical insights from Microsoft experts and industry leaders. Microsoft Ignite is less than two weeks away! Register now to join us in person or attend online for free, November 17β20.
It has been revealed that A round-up of last weekβs content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example.
