Setup & Installation
npx skills add https://github.com/sanity-io/agent-toolkit --skill content-experimentation-best-practices
https://github.com/sanity-io/agent-toolkit/tree/main/skills/content-experimentation-best-practices
View on GitHubWhat This Skill Does
Covers experiment design, statistical analysis, and CMS-managed variant setup for content A/B and multivariate testing. Includes hypothesis frameworks, sample size calculation, p-values, confidence intervals, and a catalog of 17 common pitfalls across design, execution, and interpretation. Addresses both the statistics layer and the CMS integration layer.
Instead of discovering flawed experiment design after the fact, this gives you the statistical and CMS-specific guidance upfront so you don't have to reverse-engineer why your results are inconclusive.
When to use it
- Calculating required sample size before launching a headline A/B test
- Setting up field-level content variants in a CMS for a pricing page experiment
- Identifying why a previously significant test result failed to replicate
- Choosing between frequentist and Bayesian analysis for a low-traffic content experiment
- Structuring a hypothesis with a measurable success metric before writing any variant copy