← Back to the exam

Free study guide

AI Product Manager — Free Curriculum

This free curriculum covers every domain assessed in the Oxford Certified AI Product Manager examination: deciding when to use AI, framing ML problems, data, evaluation, generative AI, user experience, responsible AI, experimentation, MLOps, build-versus-buy and stakeholder management. Read each chapter, note the key points, and then sit the 75-question examination.

Ready? Take the free exam 95 min · 75 questions
Chapter 1

When to use AI

AI should solve a real user problem better than simpler alternatives. Machine learning fits problems where patterns are complex, change over time or are hard to express as rules — fraud detection, recommendations, image recognition, forecasting. Problems with simple, stable rules (such as calculating tax) are better solved with conventional software.

Before investing, confirm the user problem, the value AI adds, data availability and the cost of errors. Define clear acceptance criteria: the performance level and constraints (accuracy, latency, cost, fairness) needed for the feature to be useful and launchable.

  • Start with the user problem, not the technology
  • Use ML for complex, changing patterns; use rules for simple, stable logic
  • Check data availability and the cost of wrong predictions
  • Agree acceptance criteria before development
Key takeaway: AI is a means to solve a problem — never the goal itself.
Chapter 2

Framing problems as machine-learning tasks

Supervised learning learns from labelled examples. Classification predicts categories (will this customer churn — yes or no?); regression predicts numbers (next week's order volume). Unsupervised learning finds structure without labels, such as clustering customers into segments. Reinforcement learning learns through rewards from actions.

A well-framed ML problem answers: what decision or action will the prediction support, how will success be measured for users and the business, what data exists and is it labelled, and what happens when the model is wrong? A "minimum viable model" — simple but good enough — gets real user feedback early.

  • Classification = categories; regression = numbers; clustering = groups without labels
  • Supervised learning needs labelled outcomes
  • Link every prediction to a decision or action
  • Start with a minimum viable model and iterate
Key takeaway: A precise problem definition prevents months of building the wrong model.
Chapter 3

Data strategy and quality

Model quality depends on data quality: "garbage in, garbage out". Training data must represent the real users and situations the product will serve; unrepresentative data leads to poor or unfair performance for some groups. Labelling quality improves with clear guidelines, multiple annotators and agreement checks.

Data is split into training, validation and test sets. The test set is never used during training, so it estimates performance on unseen data. Overfitting — memorising training data, including noise — produces excellent training scores but poor real-world results. Synthetic data can help when real data is scarce or sensitive, but its realism must be validated, and it should not replace evaluation on real data.

  • Representative data prevents poor and unfair results
  • Invest in labelling guidelines and quality checks
  • Keep a held-out test set
  • Watch for overfitting
  • Validate synthetic data carefully
Key takeaway: Your data strategy is your model strategy.
Chapter 4

Evaluating models

Accuracy is misleading on imbalanced data: a model that predicts "not fraud" for every transaction can be 99.5% accurate and useless. A confusion matrix counts true positives, false positives, true negatives and false negatives. Precision asks: of the items flagged, how many were correct? Recall asks: of all actual positives, how many were found?

The right balance depends on business costs. Medical screening with specialist review prioritises recall to avoid missed cases; a spam filter that hides important emails has a false-positive problem. Adjusting the decision threshold trades precision against recall. Offline metrics do not always predict production impact, so online experiments are still needed. Communicate performance to executives in business terms: losses prevented, time saved, risks and limitations.

  • Never rely on accuracy alone for imbalanced problems
  • Precision = correctness of positives; recall = coverage of positives
  • Choose thresholds based on the cost of each error type
  • Validate offline results with online experiments
  • Translate metrics into business impact
Key takeaway: The best metric is the one that reflects the real cost of being wrong.
Chapter 5

Generative AI and large language models

Large language models generate text by predicting the next token based on patterns learned in training. They can produce fluent but false statements — hallucinations. The context window limits how much text the model can consider at once.

Retrieval-augmented generation (RAG) retrieves relevant documents and provides them as context, grounding answers in company knowledge and enabling citations. Prompt engineering shapes behaviour; fine-tuning helps when a consistent style, format or specialised behaviour cannot be achieved with prompts. Evaluation requires a realistic test set with clear criteria, combining automated checks and human review, plus red-teaming to find harmful or incorrect outputs.

Security matters: prompt injection hides instructions inside content (emails, web pages) to hijack an assistant. Treat external content as untrusted, limit the assistant's permissions and require confirmation for sensitive actions.

  • LLMs predict tokens; they can hallucinate
  • RAG grounds answers in your documents and enables citations
  • Prompting first; fine-tune for consistent specialised behaviour
  • Evaluate with realistic test sets and red-teaming
  • Defend against prompt injection with least privilege and confirmations
Key takeaway: Generative AI is powerful but probabilistic — design for verification.
Chapter 6

Designing the AI user experience

AI outputs are probabilistic, so the experience must set clear expectations, show uncertainty where useful and keep users in control: let them edit, undo or reject outputs, and present generated content as a draft. Collect feedback — both implicit (clicks, purchases, skips) and explicit (ratings) — to improve models.

Human-in-the-loop designs route uncertain or high-impact cases to people while automating clear ones. When the assistant cannot answer reliably, it should decline gracefully or hand off to a human. Watch for automation bias — users accepting suggestions without checking — and design reviews and friction where errors are costly. Users should know when they are interacting with AI.

  • Set expectations and disclose AI involvement
  • Keep users in control: edit, undo, reject
  • Route low-confidence or high-impact cases to humans
  • Collect structured feedback
  • Counter automation bias in high-stakes decisions
Key takeaway: Good AI UX builds the right amount of trust — not blind trust.
Chapter 7

Responsible AI, privacy and regulation

Bias can come from unrepresentative data, biased labels or proxies for protected characteristics. Responsible teams test performance across user groups, document intended use and limitations in model cards, provide explanations for important decisions and offer a way to contest them.

Privacy by design minimises personal data, removes or pseudonymises unnecessary identifiers before training and ensures a lawful basis. The EU AI Act takes a risk-based approach: some practices, such as social scoring by public authorities leading to unjustified detrimental treatment, are prohibited; high-risk uses such as recruitment or credit scoring face strict requirements on data, documentation, transparency and human oversight; chatbots carry transparency obligations. Marketing must be accurate — never claim an AI "never makes mistakes".

  • Test across groups and document limitations
  • Explainability supports trust and accountability
  • Minimise and protect personal data
  • Know the EU AI Act risk categories
  • Keep marketing claims honest
Key takeaway: Responsible AI is a product requirement, not an afterthought.
Chapter 8

Experimentation and product metrics

A/B tests measure the causal impact of an AI feature against a control group. Define a primary metric that reflects user value (for example task completion time or suggestion acceptance) and guardrail metrics that must not worsen (complaints, latency, error rates). Run experiments for a planned duration with sufficient sample size rather than stopping at the first positive day.

Business metrics should reflect real value: for fraud detection, losses prevented balanced against false alarms and customer friction; for assistants, time saved and successful task completion.

  • Primary metric = user value; guardrails = things that must not get worse
  • Plan duration and sample size in advance
  • Avoid vanity metrics such as tokens generated
  • Measure business value and costs together
Key takeaway: Experiments turn opinions about AI value into evidence.
Chapter 9

MLOps: running models in production

Models degrade. Data drift changes the input distribution; concept drift changes the relationship between inputs and outcomes. MLOps practices version data, code and models; automate training and deployment pipelines; and monitor performance and data quality with alerts.

Shadow deployments run a new model alongside the current one without affecting users, allowing safe comparison. When performance drops — for example after a major change in customer behaviour — investigate drift, retrain or recalibrate with recent data and strengthen monitoring. Product managers remain responsible for AI performance after launch.

  • Version everything: data, code, models
  • Monitor quality and drift continuously
  • Automate retraining and deployment
  • Use shadow deployments to compare safely
  • Ownership continues after launch
Key takeaway: Launching a model is the beginning of its life, not the end of the project.
Chapter 10

Build, buy, cost and latency

Third-party foundation model APIs offer fast time to market and low upfront cost; building proprietary models makes sense when unique data and the capability are core differentiators. Reduce vendor lock-in by abstracting model calls behind an internal interface and keeping evaluation sets to compare providers.

Inference cost and latency shape the product. Route simple requests to smaller models, cache repeated answers, shorten prompts and stream responses to reduce waiting. Contracts must state how the provider uses, stores and retains your data, including whether it is used for training.

  • Buy for speed; build for differentiation
  • Abstract providers and keep evaluation sets
  • Manage cost with model routing and caching
  • Reduce latency with streaming and smaller models
  • Check data-usage terms in every AI contract
Key takeaway: Cost, latency and data terms are product decisions, not just engineering details.
Chapter 11

Leading AI products and stakeholders

AI product managers coordinate data scientists, ML engineers, designers, user researchers, legal, privacy and compliance teams and business leaders. Early roadmaps emphasise learning milestones that validate value, feasibility and data quality before scaling.

Communicate in business language, be explicit about limitations and safeguards, and align on acceptance criteria and success metrics before development. Honest communication — internally and externally — protects users and the company's reputation.

  • Coordinate technical, design, legal and business stakeholders
  • Roadmap learning milestones before scaling
  • Speak in business outcomes and be clear about limits
  • Align on criteria and metrics early
Key takeaway: Trust in AI products is built through clear, honest communication.

Ready? Take the free exam

Take the exam and earn your digital badge free of charge. After passing, you may order the official certificate.

Start free exam