Recommendation Systems in Production
The Short Answer
Recommendation systems create value when they become decisioning systems: they rank options, learn from behaviour, respect business constraints, and improve measurable outcomes. The best systems are not only predictive. They are governed product loops.
From Recommenders To Decisioning
Many teams treat recommendations as a model problem: choose an algorithm, train on historical behaviour, and display ranked items. In production, the harder question is what decision the system is making and how that decision changes the business.
A recommender may decide which product to show, which message to send, which offer to prioritise, which content to surface, or which next action a customer should receive. That makes it a product, data, experimentation, and governance problem at the same time.
Production Architecture
| Layer | Role |
|---|---|
| Signals | Behaviour, context, inventory, customer state, product attributes |
| Candidate generation | Build the eligible set of possible recommendations |
| Ranking | Score options by relevance, value, and constraints |
| Business rules | Apply eligibility, safety, margin, stock, and policy controls |
| Delivery | Serve recommendations inside the product or channel |
| Feedback | Capture clicks, conversions, dismissals, overrides, and downstream value |
| Experimentation | Prove incremental impact against control groups |
What Good Looks Like
Best-in-class recommendation systems have four qualities:
- They optimise for a real product outcome, not just offline accuracy.
- They incorporate constraints such as availability, margin, frequency, and customer trust.
- They are measured with experiments, not only dashboards.
- They are explainable enough for product and business teams to improve them.
Metrics That Matter
Useful metrics include:
- Incremental conversion or revenue.
- Average order value or customer lifetime value.
- Engagement quality, not only click-through rate.
- Diversity, novelty, and repetition controls.
- Customer opt-outs, complaints, or negative signals.
- Long-term retention or satisfaction impact.
FAQ
What is the difference between personalization and recommendation?
Personalization adapts an experience to a user. Recommendation is one mechanism for personalization: ranking options based on signals, context, and objectives.
Why do recommenders need experimentation?
Offline model metrics cannot prove business value. A recommendation can look accurate historically while failing to change behaviour or creating unwanted side effects. Controlled experiments test incremental impact.
What makes a recommendation system production-ready?
It needs stable data, monitoring, guardrails, business constraints, serving infrastructure, feedback capture, and an experimentation framework.
Related Reading
- Recommendation Systems Portfolio
- AI Decisioning
- Measurement and Experimentation
- AI Product Experimentation
Turn the idea into an operating system
Explore the portfolio proof and related AI wiki concepts, then connect the page back to measurable product, governance, and adoption work.
Keep reading from this thread
From 494 indexed pages and articles.
- Portfolio proof Recommendation and Decisioning Systems Portfolio notes on recommendation systems, next-best-action, churn prevention, search relevance, and decisioning systems Related by recommendation system
- Insight AI Measurement and Experimentation How to measure AI product impact with evals, adoption metrics, online experiments, guardrails, and cost tracking Related by insights
- Insight Insights Essays and executive notes on enterprise AI in production, recommendation systems, ecommerce AI, measurement, and AI operating models Related by recommendation system