/ portfolio /
Measurement and experimentation for AI products.
AI product leadership needs a measurement spine. Without incrementality, experiments, and guardrail metrics, teams can mistake model enthusiasm for business value.
Focus areas
Incrementality
Separating value created by the system from value that would have happened anyway.
A/B testing
Designing experiments for AI features where ranking, timing, targeting, and learning loops can all move.
Statistical significance
Making uncertainty understandable for product and executive decisions without flattening the nuance.
Contextual bandits
Using continuous optimisation where static campaign testing is too slow or too wasteful.
Planned assets
- Incrementality measurement primer for AI leaders.
- A/B testing checklist for AI features.
- Guardrail metrics for personalisation and recommendation systems.
- Contextual multi-armed bandits for campaign optimisation.
Recommended next
Keep reading from this thread
From 500 indexed pages and articles.
- Wiki concept A/B Test Acceleration The disciplined use of concurrency, faster metrics, variance reduction, adaptive allocation, and valid early-stopping methods to shorten experiment timelines Related by ab testing
- Wiki concept A/B Testing Mindset An A/B testing mindset is the operating posture that treats failed tests as expected learning, keeps hypotheses moving, and uses data rather than Related by measurement
- Wiki concept Statistical Significance Testing The practice of deciding whether an observed experiment result is likely to reflect a real effect rather than random Related by ab testing