Setup & Installation
npx skills add https://github.com/datadog-labs/agent-skills --skill dd-llmo-eval-trace-rca
https://github.com/datadog-labs/agent-skills/tree/main/dd-llmo/eval-trace-rca
View on GitHubWhat This Skill Does
Performs root cause analysis on production LLM application traces by using eval judge verdicts as diagnostic signal. It walks the trace tree to find where failures actually originate, not just where they were detected, then produces a structured report with concrete fix recommendations.
Instead of manually clicking through dozens of failing traces in Datadog to guess at patterns, this runs a structured open-coding-to-axial-coding analysis and pinpoints root causes in specific spans with before/after fix suggestions.
When to use it
- Diagnosing why an LLM app's eval pass rate dropped overnight
- Tracing a RAG retrieval failure back to the specific span that provided bad context
- Finding system prompt deficiencies that cause repeated agent tool misuse
- Separating real app failures from false positives caused by miscalibrated eval criteria
- Building a ranked failure taxonomy from scattered eval results across an ml_app