What Is This?

RecSys Auto Research KB is a knowledge base for recommender-system auto research, rewritten and expanded from the Datawhale open-source project fun-rec.

📖 Read online: visit https://haozhe-xing.github.io/fun-rec-mdbook/ for the latest version.

The book follows two main threads:

• Discriminative recommendation: retrieval, ranking, re-ranking, multi-task learning, multi-scenario modeling, debiasing, cold start, and other foundational capabilities of industrial recommender systems.
• Generative recommendation: semantic IDs, generative ranking, end-to-end recommendation, recommendation reasoning, diffusion models, and hands-on practice building a generative recommender.

If you want to move from "algorithm principles" to "system practice" and understand the full arc from the classic architecture to the generative paradigm, this book is for you.

Content Map

The book has 11 Parts, best read in order; if you already know the basics of recommender systems, feel free to jump straight to the topics you care about.

See SUMMARY.md for the full table of contents.

1. Install dependencies

cargo install mdbook
cargo install mdbook-katex

2. Local preview

./serve.sh

./serve.sh builds both language editions and starts a local preview server. Open the URL printed in the terminal to read the book — the language is auto-selected at the site root based on your browser settings.

3. Build static sites

mdbook build

Build artifacts are written to the book/ directory (book/zh/ and book/en/, one per edition; ./serve.sh wraps the same build for both configs).

Project Structure

This is a bilingual project: the Chinese and English editions share the same structure and are built separately.

.
├── README.md                 # Home page and project intro
├── SUMMARY.md                # mdBook table of contents
├── book.toml                 # mdBook config (Chinese edition)
├── book-en.toml              # mdBook config (English edition)
├── serve.sh                  # Build/serve script for both editions
├── GLOSSARY.md               # Glossary
├── src/
│   ├── zh/                   # Chinese source (chapter sources + appendix)
│   ├── en/                   # English source (chapter sources + appendix)
│   ├── zh/images/            # SVG figures (Chinese edition)
│   ├── en/images/            # SVG figures (English edition)
│   ├── zh/viz/               # Interactive visualizations (Chinese edition)
│   └── en/viz/               # Interactive visualizations (English edition)
└── book/
    ├── zh/                   # Build output (Chinese edition)
    └── en/                   # Build output (English edition)

Recommended Reading Paths

Beginner path

For readers new to recommender systems who want to build a complete knowledge framework:

Part 1 → Part 2 → Part 3 → Part 4 → Part 5

You will first master the classic cascading architecture of recommender systems, then understand how retrieval, ranking, and re-ranking work together.

Advanced engineering path

For readers who have already worked on recommendation algorithms or recommendation engineering and want to strengthen their system-design skills:

Part 2 → Part 3 → Part 4 → Part 11

You will focus on candidate generation, preference prediction, list-level optimization, and online serving architecture in the industrial recommendation pipeline.

Generative recommendation path

For readers interested in LLMs, generative ranking, semantic IDs, and next-generation recommender systems:

Part 5 → Part 6 → Part 7 → Part 8 → Part 9 → Part 10 → Part 11

You will start from the paradigm shift and work through the modeling, inference, and system deployment of generative recommendation step by step.

Who Is This Book For?

• Recommendation algorithm learners: build a systematic grounding in the core models and technical roadmaps of recommender systems.
• Machine learning engineers: understand the full engineering loop of a recommender system, from offline training to online serving.
• Recommender system practitioners: catch up on new directions such as generative recommendation, semantic IDs, and end-to-end recommendation.
• Interview candidates: build a clear mental map and vocabulary for recommender-system topics.

Writing Conventions

• Each chapter opens with badges for the chapter number, estimated reading time, and difficulty level.
• Math uses $inline$ and $$display$$ notation, rendered by mdbook-katex.
• Figures live in images/, and interactive visualizations live in viz/.
• Wherever possible, each chapter includes common mistakes, key takeaways, an FAQ, chapter connections, and tiered practice problems.

Contributing

Issues and pull requests are welcome:

• Fix typos, formulas, figures, or broken links.
• Add recommender-system papers, industrial case studies, or engineering lessons.
• Improve chapter structure, example code, exercises, or visualizations.
• Propose new recommender-system topics you would like to see covered.

When submitting, please keep:

• Consistent terminology: follow GLOSSARY.md first.
• Consistent structure: mirror the organization of existing chapters.
• Clear explanations: favor intuition, boundary conditions, and engineering trade-offs.

Acknowledgments

This book is rewritten from the Datawhale open-source project fun-rec. Thanks to the original authors and community contributors for building open learning materials on recommender systems.

Thanks also to the researchers and engineers in the recommender-system, information-retrieval, machine-learning, and LLM communities. Much of this book draws on public papers, industrial experience sharing, and open-source community discussions.

Recommender systems are among the most critical pieces of infrastructure in the modern internet, yet their internal logic is far more complex than "helping users find content." This part first helps you build a three-dimensional cognitive framework: from the two micro-level paradigms, to the two industrial technology routes, to the macro-level ecosystem balance; it then lands on the engineering foundation, so you understand how business fields become model-usable features and Embeddings.

What This Part Covers

What You'll Be Able to Do After This Part

• 🟢 Describe the two fundamental paradigms of the recommendation problem (discriminative scoring vs generative sequence generation) and their core formulas
• 🟢 Explain why industrial recommendation adopts the "retrieval—ranking—re-ranking" three-stage funnel, and what each stage is responsible for
• 🟡 Analyze how end-to-end generative architectures dissolve the cascading architecture's objective misalignment, information loss, and computational fragmentation
• 🟡 Locate every later chapter on the "capability evolution" curve using the book's technology map
• 🟡 Explain how business fields enter the model through slotId / featureSign / value, and distinguish Sparse, Dense, bucketed, and Embedding representations

Core Concepts

Prerequisites

• Basic machine learning concepts (supervised learning, probability, vector representation)
• Basic familiarity with Python and neural networks

No prior recommender systems knowledge is required — this part is precisely the starting point built for beginners.

Tips for This Part

1. Build the framework first, sweat the details later. The first two sections focus on the "cognitive map"; specific algorithms unfold gradually in later parts.
2. Read the two paradigms comparatively. Each time you encounter a new model, first decide whether it is discriminative or generative.
3. Memorize the three-stage funnel. It is the organizing thread of Parts 2–4.
4. Treat 1.3 as the representation foundation. When you later see Embeddings, feature crossing, or vector concatenation, return to slotId / featureSign / value to check where the information actually lives.

Let's dive in! 🚀

What Is a Recommender System

📝 Before You Continue: This chapter is the foundational starting point and requires no prior recommender systems knowledge. You only need to know one basic fact: "machine learning is letting a model learn patterns from data."

When you open your phone in the morning to catch up on news, or browse around an e-commerce app, you may not realize it: a sophisticated system is making thousands of judgments within milliseconds, deciding what you see and what you miss. This is the recommender system — one of the most fundamental pieces of infrastructure in the modern internet.

But "helping users find content they're interested in" is only a surface-level description. To truly understand it, we need to observe it from three levels: from the most microscopic single prediction, to the industrial-scale process at scale, to the macroscopic ecosystem balance. Only by moving through these levels in turn can you grasp the core logic of recommender systems.

After reading this chapter, you will be able to:

• Explain in one sentence the microscopic problem recommender systems solve, and distinguish its two fundamental paradigms
• Write down the core formulas of discriminative and generative recommendation, and explain how the questions they ask differ
• Explain why industrial recommendation adopts the three-stage funnel of "retrieval—ranking—re-ranking," and what each stage is responsible for
• Describe how end-to-end generative architectures dissolve the three major pain points of cascading architectures
• Understand the long-term value of recommender systems from the ecosystem triangle (users/creators, content, platform)
• Complete 4 leveled practice problems to consolidate the three perspectives

1.1.0 Three Perspectives: From Micro to Macro

The biggest mistake in understanding recommender systems is "seeing only the algorithms." The same system looks completely different when viewed at different scales:

Let's unpack each in turn.

1.1.1 The Micro Perspective: Two Fundamental Paradigms of the Recommendation Problem

Let's start from the most basic unit. The core problem facing a recommender system seems simple — how do we find the most valuable content for a user? But there are two fundamentally different approaches to answering this question.

Whichever approach is taken, the system must first deeply understand three key elements:

• Understanding the User — who you are and what your interests are. Historical behavior is the most important signal; explicit feedback (like a "not interested" button) and profile information (age, region) provide clues; real-time intent (what you just searched) is equally critical.
• Understanding the Item — its content attributes (category, duration, quality) and statistical attributes (view count, ratings, engagement trends).
• Understanding the Context — a weekday morning or a weekend night? Commuting on the subway or relaxing at home? Subtle differences significantly affect preferences.

💡 Key Insight: The two paradigms share the same inputs (both must understand user, item, and context), but they ask fundamentally different questions. This point determines all subsequent differences in architecture and optimization objectives.

The First Approach: Discriminative Recommendation

It defines recommendation as: given a specific "user—item—context" triple, predict the probability that the user will take a positive action on that item. The core is a scoring function:

$$Score=f(User,Item,Context)$$

The system evaluates each candidate item one by one, computes a score, and then recommends the best. This is like a judge who scores every contestant one by one and finally selects the top scorers.

Discriminative recommendation integrates user features $U$, item features $I$, and context features $C$ to score each candidate item one by one, predicting the likelihood of a "valuable connection."

🧠 Mental Model: The Talent-Show Judge

Think of recommendation as a "talent-show judge." The stage is full of contestants (candidate items), and the judge (the model) scores each contestant individually, then hands out passes by score. The judge never "announces the list directly" — scoring is the judge's only job.

The Second Approach: Generative Recommendation

It fundamentally redefines the problem: instead of evaluating candidates one by one, the model directly "creates" the recommendation result based on its understanding of the user and context. The core is a generation function:

$$[I_1,I_2,\dots ,I_k]=g(User,Context)$$

The model takes the user's historical interaction sequence and current context as input, and directly generates a sequence of recommended items through autoregressive decoding. This is like a friend who knows your taste — no need to comb through all the options; they just say "here's what you should watch next."

Generative recommendation takes the user's historical interactions and context as input, and directly outputs a sequence of recommended items through a generative model — no need to evaluate candidates one by one.

🤔 Why do both paradigms coexist? Discriminative recommendation is extremely mature and stable at "selecting the best from a finite candidate set"; generative recommendation has huge potential for "creating in an open space." The former asks "will the user like this item?", while the latter asks "what does the user want to see next?" — the former selects; the latter creates.

Four Stages of Capability Evolution

Looking at a longer timeline, recommendation algorithms show a clear trajectory of capability evolution, one that runs through both paradigms:

1. Memorization on pure IDs — collaborative filtering treats items as opaque symbols, memorizing co-occurrence patterns like "people who watched A also watched B."
2. Generalization through deep learning — deep networks generalize knowledge to unseen user—item combinations through feature crossing and sequence modeling, but items remain atomic IDs.
3. Understanding through semantic IDs — when items are encoded as structured tokens carrying semantics, the system begins to truly "understand" content meaning; new items can be recommended without accumulating behavioral data.
4. Reasoning with large models — instead of implicitly computing scores, the model explicitly analyzes intent, evaluates matches, and gives reasons, evolving from a "pattern matcher" into a "reasoner that can explain its decisions."

💡 Key Insight: These four stages do not linearly replace one another; they stack and coexist. In today's industrial systems, ID-based collaborative filtering remains a major retrieval channel, deep generalization supports every stage from retrieval to ranking, while semantic IDs and large-model reasoning are emerging at the frontier.

1.1.2 The Industrial Perspective: Two Technology Routes at Scale

Having understood the two basic paradigms, we immediately face a shared practical challenge: scale. A typical video platform has hundreds of millions of users and over a hundred million items, and recommendation must complete the entire process — from a massive item pool to a personalized list — within millisecond-level latency. If the page takes more than a few seconds to load, most users will leave.

⚠️ Warning: The core tension in recommender-system engineering is — how do we find the best result from a massive candidate pool within extremely limited time? The two paradigms offer fundamentally different answers.

The Discriminative Answer: A Multi-Stage Pipeline

The core difficulty of the discriminative paradigm is that computing a match score between every user and every item would instantly overwhelm even the strongest servers. Industry's answer is a staged funnel architecture, using the three-stage "retrieval—ranking—re-ranking" pipeline to progressively narrow the candidates, balancing efficiency and effectiveness.

• ① Retrieval — quickly filter a few thousand possibly relevant candidates from the full item pool. Its motto is "better to over-include than to miss," prioritizing coverage over precision; models are simple and features are limited (e.g., using collaborative filtering to find similar users, or retrieving based on content similarity).
• ② Ranking — where the prediction function $f$ truly shines. It deploys the most complex deep models, fusing the full set of user/item/context features to compute a precise score for each candidate, maximizing prediction accuracy.
• ③ Re-ranking — the final optimization over the ranked list. It solves the problem that "the highest-scoring list ≠ the best-experience list": introducing diversity and novelty to avoid aesthetic fatigue when the top ten are all similar content, while handling business rules such as ads and operations.

💡 Key Insight: The essence of the three-stage pipeline is — use different strategies at different stages, progressively filtering from "possibly relevant" to "best match." Retrieval pursues coverage, ranking pursues precision, re-ranking pursues experience; all three are indispensable.

The Generative Answer: End-to-End Generation

Generative recommendation proposes a radically different approach: if the model can directly "generate" results, why do we still need multi-stage filtering? Generative recommendation treats the user's historical interaction sequence as "context," and directly decodes a sequence of item tokens through autoregressive models like Transformers — the entire process is completed end-to-end within a single unified model, with no retrieval/ranking/re-ranking cascade.

💡 Key Insight: The end-to-end architecture eliminates three core pain points of cascading architectures:

• Objective misalignment — retrieval optimizes relevance, ranking optimizes click-through rate, re-ranking optimizes diversity, each fighting its own war;
• Information loss — high-quality items filtered out at retrieval are forever invisible to later stages;
• Computational fragmentation — different stages use different models, making it hard to fully exploit modern GPU capacity.

The interactive demo below lets you experience the funnel process of "hundreds of millions → one list" firsthand:

Click "Next Step" or "Autoplay" to watch the candidate pool shrink step by step from hundreds of millions down to the final list of 10, and observe the different responsibilities each stage carries.

📊 Data Point: The two architectures currently develop in parallel in industry: the discriminative pipeline serves mature scenarios stably through "divide and conquer"; the generative architecture shows huge potential in frontier exploration through "end-to-end optimization."

1.1.3 The Macro Perspective: Building a Win-Win Ecosystem

Placing recommender systems in a broader view reveals a deeper question: is a technically perfect recommender system necessarily a truly excellent one? The answer is often no.

Consider the classic "accuracy trap": a user has just added a phone to their cart, and the system recommends that same phone to them. Click-through and conversion rates may approach 100%. By the metrics this is extremely "accurate," but what value does it create? Almost none — the user was going to buy it anyway; the recommendation merely repeats information the user already knows, with no incremental value.

💡 Key Insight: The ultimate goal of a recommender system is not to blindly maximize technical metrics, but to build a healthy ecosystem where all participants benefit long-term. The ecosystem rests on three pillars: users and creators, content, and the platform — the three are interdependent.

• Users and creators — sitting at the two ends of content consumption and supply. Users are the ultimate service target; the system should help them discover content they are "genuinely interested in but haven't yet encountered," rather than trapping them in an "ever-narrowing" filter bubble. Creators are the core of content supply; distribution capability directly determines their survival space and motivation. The supply side is shifting from primarily professional teams (PGC) toward ordinary users creating spontaneously (UGC) as the main body, with AI-assisted generation (AIGC) emerging as well, blurring the boundary between consumers and producers.
• Content — the medium connecting users and creators, and the true "atomic unit" of distribution. A healthy system must not only distribute popular content, but also continuously surface promising content, avoiding "concentration at the head, drowning of the long tail."
• The platform — the ecosystem's coordinator. It must optimize effectiveness — satisfaction and time spent — while also attending to long-term health (diversity, suppressing low quality, protecting creator motivation), sometimes sacrificing short-term metrics in exchange for long-term trust.

⚡ Pro Tip: A truly excellent recommender system is a delicate balancer — finding dynamic equilibrium among users, creators, content quality, and platform development. This requires designers to be not only technical experts, but also to possess an ecosystem mindset.

⚠️ Common Mistakes in 1.1

Chapter Summary

📌 Key Takeaways

❓ FAQ

Q1: Which is better, discriminative or generative?

A: Neither is absolutely superior. Discriminative approaches are stable and efficient in mature scenarios; generative approaches have great potential in frontier exploration. The two currently develop in parallel in industry — choose based on your business stage.

Q2: Why not just show users the entire item pool and let them pick?

A: Among hundreds of millions of items, users will only look at a tiny fraction. The value of recommendation is precisely doing subtraction on the user's behalf within a space of hundreds of millions, and doing so within milliseconds — this is an engineering reality, not a design preference.

Q3: Why can the retrieval stage use "simple models"?

A: Retrieval's motto is "better to over-include than to miss"; its goal is coverage, not precision. The candidate set will be finely filtered by ranking afterward, so the retrieval side can trade lightweight models for speed.

🔗 Connections to Later Chapters

• 1.2 (book overview) maps this section's three perspectives onto the book's technology map, locating each chapter on the capability-evolution curve.
• 2.1–2.5 (retrieval) expands the concrete algorithm families of "retrieval" within the three stages.
• 3.1–3.5 (ranking) dives into the engineering implementation of the discriminative scoring function $f$.
• 5.3 (evolution of the generative paradigm) echoes this section, systematically tracing the leap from discriminative to generative.

Practice Problems

Work through all problems in order — they get progressively harder. Each has a complete solution you can reveal after trying it yourself.

Problem 1.1.1 — Distinguishing Paradigms 🟢 Easy

Given the following two system descriptions, determine whether each is closer to the discriminative or generative paradigm, and explain why.

• (a) The system computes a "probability the user clicks" for each candidate ad, ranks by probability, and shows the top 5.
• (b) The system reads the user's last 20 plays and directly outputs "the 3 video IDs you might want to watch next."

Problem 1.1.2 — Completing the Pipeline 🟢 Easy

A recommender system faces 100 million items and finally shows 10. Fill in the correct stage names in the brackets so it matches the industrial funnel:

Full item pool (100M) → [ ① ] → Candidates (~2000) → [ ② ] → Candidates (~200) → [ ③ ] → Final list (10 items)

Full item pool (100M) → [ Retrieval ] → Candidates (~2000)
                      → [ Ranking ] → Candidates (~200)
                      → [ Re-ranking ] → Final list (10 items)

Problem 1.1.3 — Analyzing the Accuracy Trap 🟡 Medium

A news app finds that immediately recommending another "World Cup final" story to a user who just finished reading one achieves a 95% click-through rate. The product manager concludes the recommendation is "highly precise." Point out what's wrong with this conclusion, and propose a more reasonable evaluation perspective.

🏆 Challenge: Making the Design Trade-off

Suppose you need to build a recommender system from scratch for a new app with 10 million daily active users. Write an argument of no more than 150 words explaining: given sparse early-stage data and limited compute, why should you prioritize the discriminative three-stage pipeline over an end-to-end generative architecture? Also indicate which kinds of scenarios could pilot generative approaches once the business matures.

Book Overview and Technology Map

📝 Before You Continue: Please read the three perspectives in 1.1 first. This chapter unfolds that "three-dimensional understanding" into a navigable technology map, helping you locate every chapter that follows.

Once you understand the core logic of recommender systems, the real challenge is turning that understanding into actionable technical solutions. Researchers and engineers have proposed hundreds of algorithms, and beginners often feel lost: how do these models relate to each other? Which one for which scenario? How do you assemble a complete system?

This chapter gives you a map. The book is organized along two main storylines — discriminative and generative — while also threading together the capability evolution from "memorization and generalization" to "understanding and reasoning."

After reading this chapter, you will be able to:

• State the logic behind the book's two halves and where each lands
• Map every chapter of the fundamentals half (Ch0–Ch4) onto the three-stage pipeline or frontier trends
• Explain why "retrieval → ranking → re-ranking" is the organizing thread of the discriminative storyline
• Anticipate how the generative storyline will unfold in later versions (Ch5–Ch10)
• Complete 4 leveled practice problems to verify your grasp of the technology map

1.2.0 One Map, Two Storylines

The book's organization can be summarized in one diagram: the horizontal axis is time/capability evolution (from pure-ID memorization to LLM reasoning), and the vertical axis is the two paradigm storylines (discriminative vs generative).

• First half: industrial practice of discriminative recommendation — progressively deepening along the "retrieval → ranking → re-ranking" pipeline; the main battlefield of memorization and generalization capabilities.
• Second half: the technology landscape of generative recommendation — starting from the foundational paradigm, through Scaling Laws, end-to-end modeling, and reasoning capability to diffusion models; the exploration ground of understanding and reasoning capabilities.

💡 Key Insight: The two storylines are not isolated. Generative architectures often "end-to-end-ify" mature discriminative modules; meanwhile, discriminative techniques such as semantic IDs and feature crossing provide the representational foundation for generative approaches.

1.2.1 First Half: Industrial Practice of Discriminative Recommendation (Covered in This Edition)

This edition (the fundamentals half) fully covers the first half plus the bridging trends chapter — 5 parts in total:

Retrieval: From 100 Million to Thousands (Part 2)

Retrieval is the pipeline's starting point: it must filter from hundreds of millions of items down to thousands of candidates within milliseconds. Its technical evolution unfolds across five sections:

• Collaborative filtering — the classic starting point: from ItemCF's item similarity, through Swing's industrial optimization and UserCF's user perspective, to matrix factorization mapping users/items into latent vectors, pioneering vectorization.
• I2I vector retrieval — transplanting Word2Vec's sequence-modeling ideas into recommendation: from Item2Vec's direct transfer, to EGES fusing side attributes, to Airbnb embedding business objectives into sequence construction.
• Two-tower models (U2I) — encoding users and items separately as vectors, represented by FM, DSSM, and YoutubeDNN, enabling efficient vector retrieval.
• Sequence retrieval — attending to the temporal information earlier methods ignored: MIND uses multiple vectors to represent diverse interests; SDM separates long- and short-term preferences and fuses them dynamically with gating.
• Streaming index retrieval — stepping outside model-internal compression: Trinity uses clustering statistics to preserve full historical interests; Streaming VQ lets the index structure adapt to data distribution in real time.

Ranking: From Thousands to Hundreds (Part 3)

Ranking scores thousands of candidates precisely — the main battlefield of deep generalization:

• Wide & Deep — jointly training a linear model and a deep network, establishing the foundational "memorization + generalization" framework.
• Feature crossing — from FM's second-order crossing, through DeepFM and xDeepFM, toward automatic high-order crossing.
• Sequence modeling — DIN uses attention to dynamically activate history based on the candidate; DIEN explicitly models the temporal evolution of interests.
• Multi-objective / multi-scenario — MMoE and ESMM balance multiple objectives; multi-tower and dynamic weights adapt to cross-scenario differences.

Re-ranking: From Hundreds to One Screen (Part 4)

Ranking output is often highly homogeneous; re-ranking optimizes the whole-list experience while preserving relevance:

• Greedy re-ranking — MMR linearly combines relevance and diversity; DPP uses a determinant framework to control diversity more precisely.
• Personalized re-ranking — PRM uses a Transformer to model mutual influence among items, achieving end-to-end personalized list generation.

1.2.2 Preview of the Second Half: Generative Recommendation (Covered in Later Versions, Ch5–Ch10)

To help you build a complete picture, here is a brief preview of the second half, so you know where the road leads after this edition:

📝 Note: This edition focuses on the fundamentals half (Ch0–Ch4). The second half will be continued in later editions under the same book-writer conventions.

⚠️ Common Mistakes in 1.2

Chapter Summary

📌 Key Takeaways

❓ FAQ

Q1: Why does the fundamentals half stop at Ch4 instead of including Ch5's generative content?

A: Ch5 is the cornerstone of the generative storyline, and its conceptual density rises sharply (semantic IDs, Transformers, diffusion models). First solidifying the industrial practice of discriminative recommendation, then moving into generative, makes for steadier learning.

Q2: Must retrieval, ranking, and re-ranking all use deep learning?

A: No. Retrieval often uses lightweight methods (collaborative filtering, two-tower); only ranking deploys the most complex deep models; re-ranking can use rules or lightweight models. Complexity increases as candidates shrink.

Q3: What order should I read in?

A: Strictly in Part 1→5 order. Later chapters often presuppose earlier ones (e.g., sequence modeling builds on feature crossing).

🔗 Connections to Later Chapters

• Part 2 Retrieval (Ch2.1–2.5) expands this chapter's "Retrieval: from 100 million to thousands" into five algorithm families.
• Part 3 Ranking (Ch3.1–3.5) dives into the industrial implementation of the discriminative scoring function $f$.
• Part 4 Re-ranking (Ch4.1–4.2) wraps up the three-stage pipeline and connects to the trends chapter.
• Part 5 Trends (Ch5.1–5.3) bridges with "debiasing / cold start / generative," pointing toward the second half in later editions.

Practice Problems

Problem 1.2.1 — Locating Chapters 🟢 Easy

Place each of the following algorithms into its correct position in the pipeline (write the corresponding Part/chapter): (a) DPP re-ranking　(b) YoutubeDNN two-tower　(c) DeepFM feature crossing　(d) Swing collaborative filtering

Problem 1.2.2 — Classifying Capability Stages 🟢 Easy

Classify the following techniques into one of the four capability stages — "memorization / generalization / understanding / reasoning": (a) Collaborative-filtering co-occurrence　(b) Deep feature crossing　(c) Semantic IDs (RQ-VAE)　(d) OneRec-Think explicit reasoning

Problem 1.2.3 — Probing Motivations 🟡 Medium

Why did "feature crossing" evolve from FM's second order to xDeepFM's automatic high order? Use a concrete example to illustrate: what kind of pattern would second-order crossing alone miss?

🏆 Challenge: Designing Your Reading Route

Suppose a colleague "only knows SQL and basic logistic regression, has never touched deep learning," but urgently needs to get up to speed on your company's recommendation ranking module within two weeks. Based on this chapter's technology map, design a two-week learning route for them (listing which chapters to read per day/phase, what to skip, and why), and explain your reasoning (within 150 words).

Feature and Embedding Basics

📝 Before You Continue: Please first read the user—item—context triple in 1.1 and the technology map in 1.2. You need to be able to read basic C++ and know that a hash function can map a string to an integer deterministically; no machine learning background is required.

What a recommender system ultimately processes is not an abstract "user interest," but a set of concrete fields: gender=male, city=Beijing, ad_id=10001, estimated_ctr=0.073. Business code understands these fields, but a neural network only accepts numeric tensors. A bridge is missing between the two.

That bridge is feature processing. It turns raw business fields into slotId + featureSign + value, and the model service then looks up the meaningless IDs to obtain learnable Embedding vectors. This pipeline looks like nothing more than a data-format conversion, yet it determines what the model can see, how it generalizes, and whether training and online serving stay consistent.

1.3.0 From Business Fields to Model Input

A ranking request typically carries four kinds of information at once:

All of this model-usable information is collectively called a feature. But raw values cannot be fed into the model as-is: strings cannot participate in matrix multiplication; the numeric magnitude of a business ID carries no semantics; and the scales of different continuous values can differ by a factor of billions.

The figure above shows the full division of responsibilities: the business service generates features, while the model service looks up Embeddings, combines features, and performs the network computation. The two are connected by a stable feature protocol.

💡 Key Insight: Feature processing is not just "turning things into numbers." It must simultaneously preserve categorical identity, numeric magnitude, and engineering stability, while ensuring that offline training and online prediction see the same representation.

1.3.1 The Triple: slotId / featureSign / value

Industrial systems typically organize a feature as the following record:

struct FeatureInfo {
  int slotId;              // which field or feature group this feature belongs to
  int64_t featureSign;     // lookup key for the specific value
  float value;             // numeric value or weight
};

Each of the three fields manages one thing:

For example, when the user's gender is male, it can be expressed as:

FeatureInfo sex_feature{
  .slotId = SEX,
  .featureSign = gen_feasign_string(SEX, "male"),
  .value = 1.0F
};

This reads as: "Go to the SEX shelf, find the entry for the value 'male', with weight 1.0 for this occurrence."

Continuous features divide the labor differently. If the estimated click-through rate is 0.073:

FeatureInfo ctr_feature{
  .slotId = AD_ETR_DENSE,
  .featureSign = 1,
  .value = 0.073F
};

Here featureSign=1 is just a fixed placeholder; the real information lives in value. Keep this contrast in mind:

⚠️ Warning: featureSign should be treated as an opaque 64-bit key. If it is generated as uint64_t and then stored in an int64_t, values with the highest bit set will appear negative under common two's-complement implementations. As long as downstream lookups use the same bit pattern, this is usually harmless — but never perform magnitude comparisons, abs(), or signed modulo on it.

1.3.2 How featureSign Is Generated

One common design places slotId in the high 32 bits and the ID of the specific value in the low 32 bits:

This way, even if the low-bit IDs of two fields happen to coincide, as long as their slotIds differ, the final keys still differ.

String Values: Hash First, Then Combine

static uint64_t gen_feasign_string(
    uint64_t slot_id,
    const std::string& value) {
  const uint32_t value_hash = gen_hash_new(value.data(), value.size());
  const uint64_t slot_bits = (slot_id << 32) & 0xffffffff00000000ULL;
  return slot_bits | value_hash;
}

This code works in three steps:

1. gen_hash_new deterministically maps the string to a 32-bit integer.
2. slot_id << 32 moves the feature type into the high 32 bits.
3. A bitwise OR | combines the high-bit type and the low-bit value into one key.

For example, even if the low-bit hashes of SEX=male and AGE_BUCKET=20 both equal 111, the final keys remain [SEX][111] and [AGE_BUCKET][111] respectively. Slot segmentation isolates different fields from each other.

💡 Key Insight: featureSign works like "class number + student number." The student number only distinguishes students within a class; the class number in the high bits keeps identical student numbers from different classes from colliding.

Integer Values: Encode Directly When Possible, Skip the Hash

If the value is already a small integer — for example network type 4, weekday 2, or bucket ID 7 — it can go directly into the low 32 bits:

static uint64_t gen_feasign_int32(
    uint64_t slot_id,
    uint32_t value) {
  const uint64_t slot_bits = (slot_id << 32) & 0xffffffff00000000ULL;
  return slot_bits | value;
}

As long as the value can be fully represented by a 32-bit unsigned integer, this encoding introduces no extra hash collisions. Therefore, small enums and bucket IDs should prefer the integer version.

Hash Collisions Are a Trade-off, Not an Error

Once strings are compressed into a 32-bit space, collisions are unavoidable. The low 32 bits offer only $2^{32}$ positions; by the birthday paradox, once a slot has about 77,000 distinct values, the probability of at least one collision exceeds 50%.

After two distinct values collide, they share one Embedding, and the model can no longer tell them apart. So why do industrial systems still commonly use this Feature Hashing? Because it buys stateless, scalable feature generation: no giant string vocabulary needs to be maintained, and new values get a key immediately.

Analysis:

• Benefit: No central vocabulary needed; new values are handled naturally; online services scale horizontally.
• Cost: Some semantic confusion; the original value cannot be recovered from the key; high-cardinality features are harder to debug.
• Mitigation: Enlarge the hash space, use double hashing, filter low-frequency values, or maintain an independent vocabulary for critical high-cardinality slots.

1.3.3 Why Embeddings Are Needed

This is the most easily confused — and most critical — point of the chapter:

💡 Key Insight: featureSign is the key for looking up an Embedding; it is not the Embedding itself. It is responsible for stably indicating "which category this is," but not for expressing how this category relates to other categories.

The difference between the two:

From One-Hot to Embedding

Suppose CITY has only three values: Beijing, Shanghai, and Shenzhen. The most direct encoding is One-Hot:

Beijing    -> [1, 0, 0]
Shanghai   -> [0, 1, 0]
Shenzhen   -> [0, 0, 1]

One-Hot has two properties: first, it never falsely creates magnitude relations; second, all categories are equidistant from each other. But when an ad ID has 10 million values, a single sample must logically occupy a 10-million-dimensional space, with only one position in the vector set to 1. Feeding such ultra-high-dimensional sparse vectors directly into the network makes both parameter count and computation prohibitive.

The Embedding layer can be viewed as a large matrix $E\in {\mathbb{R}}^{N\times d}$. Multiplying a One-Hot vector by the matrix is essentially selecting one row from $E$:

$$\underset{N\text{-dim One-Hot}}{\underbrace{[0,\dots ,1,\dots ,0]}}E=E_i\in {\mathbb{R}}^d$$

So in practice you never actually construct the One-Hot. Just pass in the featureSign of the category and directly look up $E_i$. This saves computation and lets the model, through training, pull categories with similar behavior into nearby regions of the vector space.

Why You Can't Use featureSign Directly as an Embedding

Suppose there are three ads:

Sneaker ad    -> featureSign = 105
Basketball ad -> featureSign = 980001
Baby formula  -> featureSign = 106

If the sign were used directly as a scalar input, the model would be handed absurd geometric relationships: the distance between sneakers 105 and baby formula 106 is only 1, while the semantically closer basketball ad 980001 is nearly a million away. This distance is determined entirely by chance of numbering or hashing, and represents nothing about behavioral similarity.

Using the sign directly has four further problems:

1. Spurious ordering. 500222 > 500111 does not mean one category is "bigger" or "better."
2. Spurious distances. Two hash values being close does not mean the two categories are similar; being far apart does not mean dissimilar.
3. No way to learn category semantics. The sign is a fixed integer; it will never move toward "more like basketball" or "more like sneakers" during backpropagation.
4. Numeric precision risk. If a 64-bit sign is converted to float32, beyond $2^{24}$ many adjacent integers can no longer be distinguished exactly, and different keys may be rounded to the same float.

An Embedding instead provides a set of trainable parameters for each key. If users who watch basketball content often click sneaker ads, training will gradually pull the two categories' vectors together; the baby-formula vector may end up in a different region. Distances between categories are learned from data, not dictated by hash values.

⚠️ Warning: "Converting the sign into 8 binary digits, splitting it by decimal digit, or normalizing it" is still not an Embedding. These operations merely expose an arbitrary ID in another way and cannot produce learnable category semantics. The correct approach is to treat the sign as an index and look up an independent, trainable vector.

A Complete Lookup

Suppose SEX=male generates featureSign=500111, and the Embedding table currently stores:

E[500111] = [ 0.12, -0.03, 0.45]
E[500222] = [-0.21,  0.34, 0.08]

During the forward pass, the model uses 500111 to look up the first row [0.12, -0.03, 0.45]. If this prediction's error back-propagates to this feature, only E[500111] and the related network parameters are updated — the integer 500111 itself is never modified.

This shows the strict separation of responsibilities:

featureSign: stably locates parameters; unchanged before and after training
Embedding:   carries learnable semantics; continuously updated during training

Why the Same Value Shares One Embedding

Two users who both have SEX=male generate the same featureSign and therefore look up the same Embedding. This does not make the two users indistinguishable, because the model sees a combination of many slots:

Sharing actually brings generalization: the samples of all male users jointly update the "male" parameter, while other features — age, city, identity — continue to preserve individual differences.

What Determines the Embedding Dimension

The dimension is not decided by the C++ feature-generation code, but by the model configuration, usually set per slot or per feature group:

The larger the dimension, the stronger the representational capacity, but the higher the memory, communication, and compute costs; with insufficient data it is also more prone to overfitting. It is a hyperparameter balancing model capacity against system cost — not "cardinality grows, so keep adding dimensions."

1.3.4 How Sparse and Dense Features Are Encoded

Both sparse and dense features fit into slotId + featureSign + value, but the field that "actually carries the information" is completely different on the two paths:

A sparse feature uses featureSign to select "which row of parameters"; a dense feature usually fixes the sign and uses value to determine "how much to scale the same set of parameters."

Sparse Features: Encoding "Which Category"

Sparse features take values from a finite or enumerable set. Their numeric form is mere identity and should not be interpreted as continuous magnitude. Network type 4 does not mean it is twice network type 2; ad ID 10002 is not "better" than 10001.

A sparse feature typically goes through five steps:

1. Normalize the raw value. Unify encoding, casing, whitespace, and missing values — for example, map an empty city to __UNKNOWN__.
2. Choose the slot. CITY, SEX, and AD_ID each have their own slotId.
3. Generate the sign. Strings are hashed first; small integers can be encoded directly into the low 32 bits.
4. Set the weight. Single-valued categories usually use value=1.0.
5. Look up the Embedding. The model uses the sign to select one row in the parameter table.

Example 1: String Category SEX=male

Suppose SEX has slotId=12, and assume hash("male")=0x3A91F20B. Then:

High 32 bits: slotId = 12       -> 0x0000000C00000000
Low 32 bits:  hash("male")      -> 0x000000003A91F20B
Final sign                      -> 0x0000000C3A91F20B

The business service generates:

const std::string normalized_sex = user.sex.empty()
    ? "__UNKNOWN__"
    : user.sex;

features.emplace_back(
    SEX,
    gen_feasign_string(SEX, normalized_sex),
    1.0F);

The model-side handling can be written as:

embedding = E_SEX[0x0000000C3A91F20B]
          = [0.12, -0.03, 0.45]
output    = 1.0 × embedding
          = [0.12, -0.03, 0.45]

Here value=1.0 merely states "this category occurs in this instance." What actually distinguishes male, female, and unknown are the different signs, and the Embeddings each of them maps to.

Example 2: Integer Enum APN_TYPE=4

The network type is already a small integer; there is no need to convert it to a string and hash it first:

features.emplace_back(
    APN_TYPE,
    gen_feasign_int32(APN_TYPE, 4),
    1.0F);

This generates [APN_TYPE][4]. Compared with the string version, it is more intuitive and introduces no additional 32-bit hash collisions.

How Multi-Value Sparse Features Are Handled

"User interest tags" may simultaneously include basketball, running, photography. In this case one slot yields multiple signs:

[INTEREST][hash(basketball)]  value=1.0
[INTEREST][hash(running)]     value=1.0
[INTEREST][hash(photography)] value=1.0

After looking up the three Embeddings, the model usually applies sum, mean, weighted pooling, or attention-based aggregation. If the number of tags varies a lot, mean avoids the bias of "more tags, larger vector norm"; if different tags carry different importance, you can put business weights into value.

Analysis:

• Strengths: No spurious ordering; parameters learned independently per category; behavioral patterns shared through the vector space.
• Costs: High-cardinality slots produce large tables; low-frequency categories get under-trained vectors.
• Key check: The same business value must produce exactly the same sign offline and online.

Dense Features: Encoding "Exactly How Much"

Dense features are numbers with continuous magnitude semantics. CTR=0.08 is genuinely larger than CTR=0.02; watching 100 seconds is usually longer than watching 10. Encoding should preserve this numeric relationship rather than creating a separate Embedding for every decimal value.

Dense features have two common implementations across frameworks:

1. Direct scalar input. Concatenate the normalized $x$ with other vectors and feed it into the MLP.
2. Unified slot protocol. Use a fixed featureSign=1 to look up a parameter vector $W$, and output $xW$.

This chapter's engineering uses the second. It is mathematically equivalent to projecting a one-dimensional scalar into $d$ dimensions through a bias-free linear layer:

$$\text{dense\_vector}=x\times W,W=E_{slot}[1]$$

A dense feature typically goes through four steps:

1. Validate and fall back. Handle missing values, NaN, inf, and illegal negatives.
2. Transform and normalize. Depending on the distribution, use it directly, log1p, Min-Max, Z-Score, or quantile transformation.
3. Fix the sign. The slot uniformly uses featureSign=1, meaning only one set of projection parameters is needed.
4. Put the number into value. value=x; the model computes x × E[1].

Example 1: Already-Normalized CTR=0.073

CTR itself lies in $[0,1]$, so it can be used directly at first:

double ctr = ad.estimated_ctr;
if (!std::isfinite(ctr)) ctr = 0.0;
ctr = std::clamp(ctr, 0.0, 1.0);

features.emplace_back(
    AD_ETR_DENSE,
    1,
    static_cast<float>(ctr));

Suppose this slot's fixed parameter vector is:

W = E_AD_ETR_DENSE[1] = [0.40, -0.20, 0.10]

Then the vector this sample passes to the upper network is:

0.073 × W = [0.0292, -0.0146, 0.0073]

Another sample with CTR=0.20 still looks up the same $W$, but the output becomes [0.08, -0.04, 0.02]. The dense path thus preserves the continuous relationship "0.20 is larger than 0.073."

Example 2: Long-Tailed Watch Duration watch_seconds=3600

Duration usually follows a long-tailed distribution. Putting 3600 directly into value would make its magnitude dwarf features like CTR. You can apply log1p first:

double seconds = watch_seconds;
if (!std::isfinite(seconds) || seconds < 0.0) seconds = 0.0;
seconds = std::min(seconds, 86400.0);  // cap the top at 24 hours

const float encoded = static_cast<float>(std::log1p(seconds));
features.emplace_back(WATCH_TIME_DENSE, 1, encoded);

3600 seconds is compressed to about 8.19, preserving the monotonic "longer" relation while reducing the dominance of extreme values over gradients and other features.

📝 Note: The E[1] in a dense slot is sometimes colloquially called an "Embedding" too, but its role differs from a sparse categorical Embedding. A sparse Embedding is "one row per category"; a dense slot has only a single fixed row, essentially a set of learnable projection weights.

⚠️ Warning: Dense features cannot be stuffed into value unchecked. Large-magnitude values like timestamps, amounts, and counts will drown out other features and amplify gradients. Scale compression and outlier handling must come first.

Common processing methods:

Whichever method you choose, you must specify fallback rules for NaN, inf, missing values, and abnormal negatives. The min/max/\mu/\sigma values, quantile points, and truncation thresholds must also be shared between offline training and online serving.

Bucketing: Which Interval Does It Fall In

Bucketed features first cut a continuous value into intervals, then treat the bucket ID as a category:

static uint32_t cut_bucket(double value, uint32_t width) {
  return static_cast<uint32_t>(value / width);
}

If eCPM=57 and the bucket width is 20, the bucket ID is 2. The final expression is [AD_ECPM][2] with value=1.0.

This is not a dense feature. It expresses "which interval it falls in"; each bucket has its own Embedding, so non-monotonic interval effects can be learned.

The raw cut_bucket still has three engineering traps:

1. Negative values: values in $(-width,0)$ may silently truncate into bucket 0; more negative values may exceed the representable range when converted to an unsigned integer — do not rely on that result.
2. NaN / inf: converting to an integer has no usable semantics and may trigger undefined behavior.
3. Zero bucket width: division by zero yields an invalid result.

A safer implementation should validate parameters first, clamp abnormal values, and set a top bucket:

#include <algorithm>
#include <cmath>
#include <cstdint>
#include <stdexcept>

uint32_t safe_cut_bucket(
    double value,
    double width,
    uint32_t max_bucket) {
  if (!std::isfinite(width) || width <= 0.0) {
    throw std::invalid_argument("bucket width must be positive");
  }
  if (!std::isfinite(value) || value < 0.0) {
    value = 0.0;  // ← KEY LINE: unified fallback for abnormal inputs
  }

  const double raw_bucket = std::floor(value / width);
  const double capped = std::min(raw_bucket,
                                 static_cast<double>(max_bucket));
  return static_cast<uint32_t>(capped);
}

Analysis:

• Equal-width bucketing: simple to implement, but long-tailed distributions often cram most samples into the first few buckets.
• Equal-frequency bucketing: more balanced samples per bucket, but depends on stable quantile statistics.
• Logarithmic bucketing: suits positive values spanning several orders of magnitude, balancing head and tail.
• Top-bucket capping: prevents extreme values from creating hordes of sparse buckets that appear only once or twice.

Why Bucketing and Dense Are Often Used Together

Using both is not pointless duplication. Bucketing tells the model "which interval it's in"; dense tells the model "exactly how much." They express the same business quantity from different angles.

1.3.5 The Embedding Table and Engineering Boundaries

Table Size Is Determined by the Number of Unique Values

The main memory of an Embedding table can be roughly estimated as:

$$\text{Memory}\approx \sum _{slot}\left(|V_{slot}|\times d_{slot}\times 4\text{ bytes}\right)$$

where $|V_{slot}|$ is the number of unique featureSigns in that slot, and $d_{slot}$ is the Embedding dimension. Real systems must also account for hash-table metadata, keys, pointers, and optimizer states; optimizers like Adam may additionally store one or two same-sized copies of state.

Table size does not directly depend on request volume. If a billion users have only three genders, the SEX slot still needs only a handful of entries; but GUID is nearly "one value per user," so its scale grows with the user base.

Admission, Eviction, and OOV

High-cardinality slots keep producing new keys, so production systems must constrain table growth:

"What happens when a key is not found" must be confirmed before adding any new feature. If downstream defaults to "randomly initialize and write immediately," a high-cardinality feature can bloat the parameter table in a short time; if a zero vector is returned, that feature contributes nothing during cold start, and the model must rely on other generalizable features.

How Embeddings Are Trained

Embeddings are part of the recommendation model, trained jointly with the upper network:

1. The forward pass looks up vectors by featureSign.
2. Multiple vectors are concatenated or pooled with dense features.
3. The DNN outputs click-through rate, conversion rate, or a ranking score.
4. Prediction error back-propagates, updating both the Embeddings and the network parameters.

Only keys covered by training samples receive effective updates. Long-tail features that appear only once or twice have vectors close to their random initialization — occupying memory while injecting noise. This is exactly why frequency-based admission exists.

Offline/Online Consistency

This is the most hidden and most common source of incidents in feature engineering. Training samples and online requests must be strictly aligned:

• Hash algorithm, seeds, and string encoding;
• slotId enum values and candidate-position offsets;
• Bucket boundaries, bucket widths, and top buckets;
• Normalization statistics;
• Missing-value, outlier, and casing rules;
• String trim, concatenation order, and separators.

These bugs often do not crash. The service still returns scores, monitoring may be all green — the model just looks up "wrong but legal" parameters, and online performance quietly degrades.

⚡ Pro Tip: Treat changing the hash, changing slots, or changing bucket boundaries as "swapping the model," not a routine code hotfix. Prioritize sharing one feature library between offline and online; before launch, sample real requests and compare the triples generated on both sides field by field.

1.3.6 Complete Case Study: Adding an eCPM Feature to Ads

Suppose the business already has ad.ecpm, and we want to preserve both its exact magnitude and its non-linear interval effects. The sound approach is to create two different slots:

#include <algorithm>
#include <cmath>
#include <cstdint>
#include <vector>

struct FeatureInfo {
  uint32_t slot_id;
  uint64_t feature_sign;
  float value;
};

enum Slot : uint32_t {
  AD_ECPM_BUCKET = 101,
  AD_ECPM_DENSE = 102
};

uint64_t make_int_sign(uint32_t slot_id, uint32_t value_id) {
  return (static_cast<uint64_t>(slot_id) << 32) | value_id;
}

uint32_t safe_bucket(double value,
                     double width,
                     uint32_t max_bucket) {
  if (!std::isfinite(value) || value < 0.0) value = 0.0;
  if (!std::isfinite(width) || width <= 0.0) return 0;
  const double bucket = std::floor(value / width);
  return static_cast<uint32_t>(
      std::min(bucket, static_cast<double>(max_bucket)));
}

void append_ecpm_features(double raw_ecpm,
                          std::vector<FeatureInfo>& output) {
  const double clean_ecpm =
      (!std::isfinite(raw_ecpm) || raw_ecpm < 0.0) ? 0.0 : raw_ecpm;

  const uint32_t bucket = safe_bucket(clean_ecpm, 20.0, 100);
  output.push_back({
      AD_ECPM_BUCKET,
      make_int_sign(AD_ECPM_BUCKET, bucket),
      1.0F
  });

  const float dense_value =
      static_cast<float>(std::log1p(clean_ecpm));
  output.push_back({
      AD_ECPM_DENSE,
      1,
      dense_value
  });
}

With input raw_ecpm=57, the two features express:

Analysis:

• Expressiveness: bucketing captures non-linearity; dense preserves continuous variation — the two complement each other.
• Parameter cost: the bucket slot has at most 101 parameter rows; the dense slot has a single vector.
• Consistency requirement: the offline side must use the same bucket width, top bucket, log1p, and abnormal-value fallback.
• Launch check: snapshot-test the triple for normal values, negatives, NaN, and extreme values respectively.

When adding a real feature, confirm each item on this checklist:

• Is it a category, a continuous value, or does it need the dual "bucketed + dense" representation?
• Are the string encoding, casing, trim, and concatenation rules fixed?
• Can small integers be encoded directly, avoiding unnecessary hashing?
• Are the dense value's magnitude, normalization, and abnormal-value fallback clearly specified?
• Should bucketing use equal-width, equal-frequency, or logarithmic boundaries? Is a top bucket set?
• Is the scale of unique values and the Embedding memory acceptable?
• Are high-cardinality slots governed by admission and eviction rules?
• What does the model service return for OOV?
• Do offline training and online serving reuse the same logic and configuration?
• Has a field-by-field consistency comparison been completed before launch?

⚠️ Common Mistakes in 1.3

Chapter Summary

📌 Key Takeaways

❓ FAQ

Q1: Why can't we just use featureSign as the Embedding?

A: Because the sign is an arbitrary ID — its magnitude and distances carry no business semantics, and converting a 64-bit sign to float32 may also lose precision. An Embedding is a set of independent trainable parameters looked up by sign; similarity between categories is learned from data such as clicks and conversions, not determined by hash values.

Q2: Why do dense features also need a featureSign?

A: Under the unified slot protocol, it still has to locate the slot's set of weights. The fixed key means all samples share that set of parameters, scaled by different values.

Q3: Should a continuous value be bucketed or dense?

A: Use dense when you need exact magnitude and continuous variation; use bucketing when the relation is clearly non-linear or you need interval effects. Important features often use both representations at once.

Q4: Why doesn't the Embedding table grow linearly with request volume?

A: Repeated occurrences of the same sign reuse the same parameter. The table grows with "the number of unique values"; only high-cardinality slots like GUID and ad ID can approach user- or item-level scale.

Q5: Why does changing the bucket width usually require retraining the model?

A: After the width changes, the same business value maps to a different key, and the old Embeddings' semantics no longer line up. Rule changes must ship together with a new model.

🔗 Connections to Later Chapters

• Section 2.3 (two-tower models) aggregates multiple groups of user and item Embeddings into the two side vectors, then uses them for large-scale retrieval.
• Section 3.1 (Wide & Deep) feeds sparse Embeddings and continuous features into the Deep part, establishing the ranking model's generalization capability.
• Section 3.2 (feature crossing) further studies how second- and higher-order interactions occur among different features' Embeddings.
• Section 6.4 (codebook quantization and semantic IDs) presents another discrete representation: making item IDs themselves carry hierarchical semantics.
• Part 11 (generative recommender systems in practice) chains offline feature generation, online feature serving, and model deployment into a complete engineering loop.

Practice Problems

Work through them in order. Later problems progressively add scale, outliers, and consistency constraints.

Problem 1.3.1 — Dividing Labor in the Triple 🟢 Easy

Given CITY=Beijing and estimated_ctr=0.08, explain for each whether the key information should go in featureSign or value.

Problem 1.3.2 — Telling Keys from Vectors 🟢 Easy

Suppose hash("male") = hash("20") = 111. Would SEX=male and AGE_BUCKET=20 get the same featureSign? And can the final sign be used directly as a model vector?

Problem 1.3.3 — Choosing a Representation 🟡 Medium

You need to add "purchase amount over the last 30 days," with an extremely long-tailed distribution. You want to preserve the magnitude, but also suspect non-monotonic effects across spending intervals. Design the feature representation.

Problem 1.3.4 — Tracking Down a Silent Failure 🔴 Hard

A new model's offline AUC is normal, and the online service reports no errors, yet performance drops markedly. Investigation finds that offline Python hashes after value.strip().lower(), while the online C++ hashes the raw string directly. Explain the cause and design a fix plus a recurrence-prevention plan.

🏆 Challenge: Estimating the Cost of a High-Cardinality Feature

An AD_ID slot has 50 million active keys, an Embedding dimension of 16, using FP32. How much memory do the vectors alone require? If training additionally keeps two same-sized optimizer states, what is the minimum total? Then explain why actual deployment is even larger.

Retrieval is the starting point of the "retrieval — ranking — re-ranking" three-stage funnel. It must quickly narrow a universe of hundreds of millions of items down to a few thousand candidates within millisecond-level latency — following the principle of "rather over-include than miss", its goal is coverage, not precision. Even a mediocre retriever can be tolerated, but if it misses the truly relevant items, the downstream ranking and re-ranking stages can do nothing to recover them.

This part unfolds along the technical evolution in five chapters: starting from the classic collaborative filtering, moving to vector retrieval (I2I), which ports sequence modeling ideas into recommendation, then the two-tower model (U2I), which uses deep networks for efficient retrieval, followed by the sequential/temporal information ignored by the previous methods (sequential retrieval), and finally stepping outside the "compress inside the model" paradigm to preserve full historical interests with a streaming index. Together they form the methodological map of the industrial retrieval layer.

What This Part Covers

What You'll Be Able to Do After This Part

• 🟢 Distinguish neighborhood-based collaborative filtering (ItemCF / UserCF) from model-based matrix factorization, and articulate their respective strengths against sparsity
• 🟢 Explain how Swing exploits bipartite-graph structure to filter noise, and how EGES uses item-specific attention to solve cold start
• 🟡 Derive the $O(kn)$ simplification of FM's second-order interaction term, and show how it can be reorganized into a two-tower inner product
• 🟡 Contrast the fundamental difference between two-tower models and sequential retrieval (MIND / SDM) in terms of "user representation": single vector vs. multiple vectors / long-short fusion
• 🔴 Analyze how Trinity and Streaming VQ use cluster statistics and streaming indexes to solve "interest amnesia" and "index staleness"
• 🔴 Complete 25+ graded practice problems across the 5 chapters, consolidating the full path from co-occurrence to vector search

Core Concepts

Prerequisites

• You have finished Part 1 (especially the three-stage funnel in 1.1 and the technology map in 1.2)
• Basic linear algebra (vector inner products, matrices), probability (softmax, cosine similarity), and general neural network knowledge
• Familiarity with Python and the basic concept of embeddings

Retrieval-layer methods are relatively lightweight and coverage-oriented, with mostly controllable complexity; but the vectorization approaches (matrix factorization, two-tower, sequential) require you to be comfortable with embeddings and gradient descent.

Tips for This Part

1. Understand the motivation before the formulas. Each method exists to fix a limitation of its predecessor — ItemCF is dominated by popular items, hence Swing; co-occurrence is sparse, hence matrix factorization.
2. Follow the main thread of "how users/items are represented". From CF's ID co-occurrence, to MF's latent vectors, to the two-tower's independent encoders, to sequential retrieval's multiple vectors, the representations grow ever more refined.
3. Mind retrieval efficiency. Methods that can precompute item vectors offline (two-tower, I2I) are usually the easiest to scale.
4. Practice with the visualizations. Every chapter's interactive HTML and SVG pieces are worth clicking through yourself, grounding abstract formulas in the intuition of "how the candidate pool shrinks".

Let's dive in! 🚀

Collaborative Filtering

📝 Before You Continue: Please read first the discussion of "retrieval as the starting point of the three-stage funnel" in Part 1, Section 1.1, and the "retrieval: from billions to thousands" thread in 1.2. This chapter covers the most classic family of methods in the retrieval layer.

When you open a shopping app, how does the system decide "what else you might like"? The most naive yet most powerful intuition comes from Collaborative Filtering (CF): inferring individual preferences from the collective behavior of "people" and "items" — people who like the same things tend to have similar tastes; items liked by the same group of people tend to be similar in nature.

The idea of collaborative filtering is nearly as old as recommender systems themselves, but it goes far beyond "finding similar things". From neighborhood-based ItemCF / UserCF, to Swing built for industrial robustness, to matrix factorization mapping users and items into latent vectors, this chapter walks you through the evolution from statistical co-occurrence to vector representation.

After reading this chapter, you will be able to:

• Compute item and user similarity with cosine similarity / Pearson correlation, and articulate the difference between the two
• Explain how Swing filters random noise through bipartite-graph substructures, and how Surprise mines complementary items
• Describe the trade-offs between UserCF and ItemCF regarding user cold start and explainability
• Describe how matrix factorization (FunkSVD / BiasSVD) alleviates data sparsity with low-rank latent vectors
• Complete 5 graded practice problems, consolidating the full path from co-occurrence to vectorization

2.1.0 Two Perspectives on Collaborative Filtering

Collaborative filtering can be split along two directions: item-based (ItemCF) asks "what else is similar to the items you liked"; user-based (UserCF) asks "what else do people similar to you like". The former fits industrial scenarios better (item sets are stable and can be precomputed offline), while the latter feels more natural in scenarios with strong social flavor.

Whichever perspective you take, the core revolves around one concept — co-occurrence: two items interacted with by the same set of users, or two users interacting with the same set of items. The more frequent the co-occurrence, the higher the similarity. As we will see in the next three sections, all CF methods are just different answers to "how to define and exploit co-occurrence".

2.1.1 ItemCF: Item-Similarity-Based Collaborative Filtering

The core idea of ItemCF is that user interests are coherent: people who like an item tend to be interested in similar items. When recommending to a user, the system first identifies the items the user interacted with recently (seed items), then finds the most similar candidates for each seed, and finally aggregates the scores.

Computing Item Similarity

Most real-world scenarios have only implicit feedback (clicks, purchases), no ratings. ItemCF quantifies item similarity with cosine similarity:

$$w_{ij}=\frac{\mathbf{C}[i][j]}{\sqrt{|\mathcal{N}(i)|\cdot |\mathcal{N}(j)|}}$$

where $|\mathcal{N}(i)|$ is the total number of users who interacted with item $i$, and $\mathbf{C}[i][j]$ is the co-occurrence count of the two items (the number of users who interacted with both). The denominator normalizes the co-occurrence count, preventing popular items from dominating by sheer interaction volume — exactly the biggest trap of naive co-occurrence.

Recommending Candidate Items

Given the similarity matrix, the online flow has three steps: ① take a few hundred items the user recently interacted with as seeds; ② find the Top-10 similar items for each seed, quickly generating a large pool of candidates; ③ compute the user's interest score for candidate item $i$:

$$p(u,i)=\sum _{j\in \mathcal{N}(u)}{w}_{ij}\cdot r_{uj}$$

$\mathcal{N}(u)$ is the set of items the user interacted with, and $r_{uj}$ is the user's interest strength on item $j$ (set to 1, or weighted by interaction time/type). Finally, rank all candidates by score and take the Top-N.

🧠 Mental Model: A Borrower's Reading List

Think of items as "books" and users as "borrowers". ItemCF's logic: if Alice borrowed The Three-Body Problem and Ball Lightning, and Bob also borrowed The Three-Body Problem, then the system guesses Bob will probably like Ball Lightning too — because these two books are always borrowed by the same crowd. The point is not the content of the books, but "who reads them".

Computational Efficiency Optimizations

Brute-force computation of all item-pair similarities is $O(|\mathcal{I}{|}^2\cdot |\mathcal{U}|)$, but the vast majority of item pairs share no common users, so their similarity is necessarily 0. A user–item inverted index speeds this up dramatically: maintain an interaction list per user, and when traversing, pair up items within each list, accumulate the co-occurrence matrix $\mathbf{C}[i][j]$, then divide by the normalization terms. The optimized complexity is roughly $O(R\cdot \bar{m})$ ($R$ is the total number of interactions, $\bar{m}$ the average number of items per user), far below brute force in sparse settings.

Similarity for Rating Data (Pearson Correlation)

When the system has explicit ratings (e.g., 5 stars), the Pearson correlation coefficient is more robust than cosine, because centering removes differences in rating distributions across items:

$$w_{ij}=\frac{{\sum }_{u\in {\mathcal{U}}_{ij}}(r_{ui}-{\bar{r}}_i)(r_{uj}-{\bar{r}}_j)}{\sqrt{{\sum }_{u\in {\mathcal{U}}_{ij}}(r_{ui}-{\bar{r}}_i{)}^2}\sqrt{{\sum }_{u\in {\mathcal{U}}_{ij}}(r_{uj}-{\bar{r}}_j{)}^2}}$$

Based on this, one can predict a user's rating for an unseen item:

$${\hat{r}}_{u,j}={\bar{r}}_j+\frac{{\sum }_{k\in {\mathcal{S}}_j}{w}_{jk}\left(r_{u,k}-{\bar{r}}_k\right)}{{\sum }_{k\in {\mathcal{S}}_j}{w}_{jk}}$$

In large-scale systems, for computational and sparsity reasons, most still use cosine similarity supplemented with weighted normalization.

Analysis: ItemCF can precompute the full item similarity matrix offline; online it only needs to fetch the Top-N similar items of the seed items, with very low latency and strong explainability. But it is powerless against item cold start (new items have no co-occurrence), and its similarity is fixed, hard to fuse with contextual features. It suits scenarios with stable item sets and dense interactions.

2.1.2 Swing: Similarity Optimization for Industrial Scenarios

ItemCF is naive and effective, but industrial deployment exposes clear problems: popular items dominate results due to high co-occurrence; noise such as random misclicks is treated equally. Swing offers an elegant answer — analyze substructures of the user–item bipartite graph to filter noise.

Its core insight: if multiple users bought the same pair of items despite sharing few other co-purchases, then the association between that pair is more trustworthy. In other words, the more "specific" the co-purchase behavior, the greater its contribution to similarity.

Computing Item Similarity

Let $U_i$ and $U_j$ be the sets of users who interacted with items $i$ and $j$. For each pair of common users $(u,v)$, if they share few other co-purchases (smaller $|I_u\cap I_v|$), then their joint choice of this item pair is more specific and should contribute a higher score:

$$s(i,j)=\sum _{u\in U_i\cap U_j}\sum _{v\in U_i\cap U_j}\frac{1}{\alpha +|I_u\cap I_v|}$$

$\alpha$ is a smoothing coefficient preventing tiny denominators from causing numerical instability. To reduce the excessive influence of active users, a user weight $w_u=1/\sqrt{|I_u|}$ is introduced:

$$s(i,j)=\sum _{u\in U_i\cap U_j}\sum _{v\in U_i\cap U_j}{w}_u\cdot w_v\cdot \frac{1}{\alpha +|I_u\cap I_v|}$$

As shown, users A and B have 4 swing subgraphs $[A,h,B]$, $[A,t,B]$, $[A,r,B]$, $[A,p,B]$. If $\alpha =1$ and A and B have 4 other shared behaviors, the user pair $[A,B]$ contributes $1/5$; h and p additionally share t and r, contributing two extra $1/3$ terms, so finally $s(h,p)=13/15$, higher than $s(h,t)=1/5$. Combinations with few but "exclusive" co-occurrences score higher — this is exactly how Swing filters popular-item noise.

Surprise: Complementary Item Recommendation

Swing scores already capture associations, but handling complementary items (buying a phone case after a phone) remains hard — complementary relations are directional and time-ordered. The Surprise algorithm measures complementary relevance at three levels: category, item, and cluster:

• Category level: compute conditional probabilities between categories with a user-category matrix, ${\theta }_{i,j}=N(c_{i,j})/N(c_j)$, and adaptively truncate the long tail using the maximum relative drop.
• Item level: consider purchase order and time interval — the closer in time, the stronger the complementarity:

$$s_1(i,j)=\frac{{\sum }_{u\in U_i\cap U_j}\frac{1}{1+|t_{ui}-t_{uj}|}}{\Vert U_i\Vert \times \Vert U_j\Vert }$$

• Cluster level: run label propagation on a graph of billions of items (edge weights are Swing scores) to cluster and alleviate sparsity, then combine linearly:

$$s(i,j)=\omega \cdot s_1(i,j)+(1-\omega )\cdot s_2(i,j)$$

Analysis: Swing significantly improves robustness while preserving ItemCF's efficiency, making it an evergreen of industrial I2I retrieval; the cost is building and traversing the bipartite graph, with higher computation than naive ItemCF. Surprise goes further for complementary scenarios, but introduces multi-level hyperparameters and clustering steps, raising engineering complexity.

2.1.3 UserCF: User-Similarity-Based Collaborative Filtering

Mirror to ItemCF, UserCF assumes: users with similar historical behavior will have similar future preferences. It first finds the "neighbors" most similar to the target user, then predicts the target user's interests from the neighbors' behavior.

Computing User Similarity

Given users $u$ and $v$ with item sets $N(u)$ and $N(v)$, three common measures:

• Jaccard coefficient (implicit feedback only):

$$w_{uv}=\frac{|N(u)\cap N(v)|}{|N(u)\cup N(v)|}$$

• Cosine similarity (accounting for activity differences):

$$w_{uv}=\frac{|N(u)\cap N(v)|}{\sqrt{|N(u)|\cdot |N(v)|}}$$

• Pearson correlation (with ratings; centering removes rating-habit differences):

$$w_{uv}=\frac{{\sum }_{i\in I}(r_{ui}-{\bar{r}}_u)(r_{vi}-{\bar{r}}_v)}{\sqrt{{\sum }_{i\in I}(r_{ui}-{\bar{r}}_u{)}^2}\sqrt{{\sum }_{i\in I}(r_{vi}-{\bar{r}}_v{)}^2}}$$

Recommending Candidate Items

Select the $K$ users with the highest similarity as the neighbor set ${\mathcal{S}}_u$. A simple weighted average predicts ratings:

$${\hat{r}}_{u,p}=\frac{{\sum }_{v\in S_u}{w}_{uv}{r}_{v,p}}{{\sum }_{v\in S_u}{w}_{uv}}$$

The bias-aware version further removes personal habits:

$${\hat{r}}_{u,p}={\bar{r}}_u+\frac{{\sum }_{v\in S_u}{w}_{uv}(r_{v,p}-{\bar{r}}_v)}{{\sum }_{v\in S_u}{w}_{uv}}$$

At serving time, find the $K$ most similar users for the target user, collect their interacted items as candidates, compute interest scores $p(u,i)={\sum }_{v\in S_u\cap N(i)}{w}_{uv}\cdot r_{vi}$, rank, and take the Top-N. The optimized complexity is roughly $O(R\cdot \bar{n})$, far below $O(|U{|}^2)$.

Analysis: UserCF excels in scenarios where user interests converge, such as "trending news" or "breaking events", and naturally supports social recommendation through "discovering similar people". But when users vastly outnumber items, computation and storage costs are high, and user cold start is hard (new users have no behavior). In industry, ItemCF is more common, because its item set is stable and can be fully precomputed offline.

2.1.4 Matrix Factorization: From Similarity to Vector Representation

Both UserCF and ItemCF face a fundamental challenge: data sparsity. Real interaction matrices are extremely sparse, leaving too few common ratings to compute reliable similarities. Matrix factorization takes a different route — instead of explicitly computing similarities, it learns latent vector representations for users and items, letting distances in the vector space naturally reflect preferences. This marks CF's shift from statistical methods to machine learning methods.

The Dawn of the Latent Vector Era

Matrix factorization rests on two assumptions: the low-rank assumption — the seemingly complex rating matrix is actually governed by a few latent factors (such as "male-oriented vs. female-oriented", "serious vs. light"); and the latent vector assumption — every user/item can be represented by a vector encoding these factors.

FunkSVD: The Basic Model

FunkSVD factorizes the rating matrix into a user feature matrix and an item feature matrix. User $u$ is represented by a $K$-dimensional vector $p_u$, item $i$ by $q_i$, and the predicted rating is their inner product:

$${\hat{r}}_{ui}=p_u^T{q}_i=\sum _{k=1}^K{p}_{u,k}\cdot q_{i,k}$$

The optimization objective makes predictions approximate true ratings (over known ratings only):

$$\underset{P,Q}{\min }\frac{1}{2}\sum _{(u,i)\in \mathcal{K}}{\left(r_{ui}-p_u^T{q}_i\right)}^2$$

Update with gradient descent, where the error is $e_{ui}=r_{ui}-p_u^T{q}_i$:

$$p_{u,k}\leftarrow p_{u,k}+\eta \cdot e_{ui}\cdot q_{i,k},q_{i,k}\leftarrow q_{i,k}+\eta \cdot e_{ui}\cdot p_{u,k}$$

In practice, add L2 regularization against overfitting: $\min \frac{1}{2}\sum (\dots {)}^2+\lambda (\Vert p_u{\Vert }^2+\Vert q_i{\Vert }^2)$.

🧠 Mental Model: Taste Axes

Plot every user and every movie on a 2D chart: the horizontal axis is "male-oriented ↔ female-oriented", the vertical axis "serious ↔ light". Users who like The Princess Diaries and the movie itself both land in the "female-oriented, light" corner, so the inner product is naturally large. Even if two users have never watched the same movie, as long as they are close on the latent factors, they can recommend for each other — this is the key to how vector representation overcomes sparsity.

BiasSVD: The Improved Model

The basic model ignores a fact: some people are naturally generous raters ("pushovers"), others strict; some movies universally rate high due to star casts. BiasSVD introduces bias terms:

$${\hat{r}}_{ui}=\mu +b_u+b_i+p_u^T{q}_i$$

$\mu$ is the global average rating, $b_u$ the user bias, $b_i$ the item bias. The optimization objective updates the biases as well:

$$\underset{P,Q,b_u,b_i}{\min }\frac{1}{2}\sum _{(u,i)\in \mathcal{K}}{\left(r_{ui}-\mu -b_u-b_i-p_u^T{q}_i\right)}^2+\lambda (\Vert p_u{\Vert }^2+\Vert q_i{\Vert }^2+b_u^2+b_i^2)$$

$$b_u\leftarrow b_u+\eta (e_{ui}-\lambda b_u),b_i\leftarrow b_i+\eta (e_{ui}-\lambda b_i)$$

Analysis: Matrix factorization handles sparse data naturally (two users can be linked through latent factors without any shared rating), and inner-product retrieval is efficient. But it remains a linear model, hard to fuse with side information or complex feature crossings. This leads directly to the two-tower and deep models of later chapters.

⚠️ Common Mistakes in 2.1

Chapter Summary

📌 Key Takeaways

❓ FAQ

Q1: When should I use ItemCF vs. UserCF?

A: Use ItemCF when the item set is stable and you need to explain "why this similar item is recommended" (the industrial mainstream); use UserCF when user interests strongly converge (e.g., breaking news) or for social "similar people" recommendation. In user cold-start scenarios, ItemCF is more robust.

Q2: What exactly is the difference between cosine similarity and Pearson correlation?

A: Cosine only looks at the angle between interaction vectors and is affected by absolute item popularity; Pearson first centers (subtracts respective means), removing rating-habit differences between "pushovers vs. strict graders" and focusing on relative trends. Prefer Pearson with rating data; use cosine for implicit feedback.

Q3: Why does matrix factorization handle sparse data better than ItemCF?

A: ItemCF needs two items to share interacting users before similarity can be computed; matrix factorization, through a shared latent-factor space, lets two users with no common ratings recommend for each other via close latent vectors, generalizing to unseen combinations.

🔗 Connections to Later Chapters

• 2.2 (Vector Retrieval I2I) ports sequence modeling (Word2Vec) into similarity learning, and uses item attention to solve MF's difficulty in fusing side info.
• 2.3 (Two-Tower Model) upgrades MF's inner-product idea to deep-network encoding, enabling efficient U2I retrieval.
• 2.4 (Sequential Retrieval) further captures temporal interest dynamics ignored by ItemCF/MF.
• 3.x (Ranking) later applies complex deep models to finely rank the thousands of candidates retrieved in this chapter.

Practice Problems

Work through all problems in order — they get progressively harder. Each has a complete solution you can reveal after trying it yourself.

Problem 2.1.1 — Similarity Normalization 🟢 Easy

User A has interacted with 100 items, user B with 10 items, and they share 5 common items. Compute the Jaccard coefficient, and explain what would go wrong if you instead used "raw co-occurrence count = 5" as the similarity.

Problem 2.1.2 — ItemCF Scoring 🟢 Easy

User $u$ has interacted with items $\{j_1,j_2\}$, with interest strengths $r_{u{j}_1}=1$ and $r_{u{j}_2}=0.5$. Item $i$ has similarity $0.8$ with $j_1$ and $0.3$ with $j_2$. Compute the user's interest score $p(u,i)$ for candidate item $i$.

Problem 2.1.3 — The Specificity Intuition of Swing 🟡 Medium

Consider two item pairs $(h,p)$ and $(h,t)$. Users A and B both interacted with both pairs, and A and B's other shared behavior counts are 4 (for h,p) and 2 (for h,t). Let $\alpha =1$. Suppose $(h,p)$ has only this single pair of common users, while $(h,t)$ additionally receives a contribution of the same structure from another common user pair C and D. Compare $s(h,p)$ and $s(h,t)$, and explain what Swing is trying to filter.

Problem 2.1.4 — FunkSVD Gradient Update 🔴 Hard

Given a known rating $r_{ui}=4$, current $p_u=[0.5,0.2{]}^T$, $q_i=[1.0,0.5{]}^T$, $\eta =0.1$, no regularization. Compute by hand $p_u$ and $q_i$ after one gradient descent step (keep 3 decimal places).

🏆 Challenge: Design a Retrieval Combination

A short-video platform adds tens of millions of items daily, with an enormous long tail. Write about 150 words explaining how you would combine this chapter's ItemCF, Swing, and matrix factorization as multi-channel retrieval (what each channel is responsible for, how they complement each other), and identify which channel should backstop long-tail new items and why.

Vector Retrieval (I2I)

📝 Before You Continue: It is recommended that you first read ItemCF and matrix factorization in 2.1. This chapter is a natural extension of the "treat items as vectors" idea — the difference is that similarity is no longer derived from co-occurrence statistics, but determined by dense vectors learned through sequence modeling.

Collaborative filtering in 2.1 defines similarity by "who interacted with what together". But if an item has barely been interacted with (cold-start new product), co-occurrence statistics fail. More fundamentally: co-occurrence only tells you "related", without encoding items as semantic vectors, making it hard to further fuse attributes or perform nearest-neighbor search.

The protagonist of this chapter is the sequence modeling idea of Word2Vec. It rests on a simple yet profound assumption: words appearing in similar contexts have similar meanings. When we replace "sentence" with "a user's behavior sequence" and "word" with "item", the same machinery learns item representations where "semantically close means vectorially close", usable for I2I retrieval. From the most direct Item2Vec transfer, to attribute-fusing EGES, to Airbnb writing business objectives into the sequence — you will see how this line of work inches ever closer to industrial reality.

After reading this chapter, you will be able to:

• Explain the core formula of Word2Vec Skip-Gram, and why negative sampling is indispensable
• Describe how Item2Vec applies the "user behavior sequence = sentence" mapping to I2I vector retrieval
• Explain how EGES's item-specific attention solves cold start and sparsity
• Analyze how Airbnb's global context and same-market negative sampling bake "booking conversion" into training
• Complete 5 graded practice problems, consolidating sequence-modeling retrieval

2.2.0 From Words to Items: A Structural Analogy

Word2Vec's success rests on "co-occurrence reflects semantics". In natural language, a sentence consists of words whose co-occurrence reflects semantics; in recommendation, a user's interaction history can be viewed as a "sentence", with items as the "words". This is Item2Vec's entire starting point — structural isomorphism, ready for transfer.

In the next four sections, you will see how this seemingly naive mapping supports an entire family of I2I vector retrieval methods.

2.2.1 Word2Vec: The Theoretical Foundation of Sequence Modeling

📎 This section covers only the intuition of Skip-Gram and its transfer to recommendation. For the CBOW architecture, structural details of the center-word/context-word two-way weight matrices $\mathbf{W}$/${\mathbf{W}}^c$, the exact form of negative sampling, and word-vector analogy properties, see Appendix: Word2Vec Deep Dive.

Word2Vec includes two architectures: Skip-Gram (predicting context from the center word) and CBOW (predicting the center word from context). Skip-Gram performs better in recommendation and is more widely adopted.

The Skip-Gram Model

Given the center word $w_t$ at position $t$ of a sequence, the model maximizes the occurrence probability of all context words within its window (size $m$):

$$P(w_{t+j}|w_t)=\frac{e^{v_{w_{t+j}}^T{v}_{w_t}}}{{\sum }_{k=1}^{|V|}{e}^{v_{w_k}^T{v}_{w_t}}}$$

$v_{w_i}$ is the vector of word $w_i$, and $|V|$ is the vocabulary size. Softmax ensures probabilities sum to 1, and the inner product in the numerator measures the similarity between the center word and the context word.

🧠 Mental Model: The Guess-the-Neighbor Game

Imagine playing "I say a word, you guess what's likely next to it". Hearing "king", you'd probably guess "queen" or "castle". Skip-Gram makes the model play this game: it doesn't ask "do these two words co-occur", but "given the center word, what words are most likely around it" — through repeated guessing, the model is forced to place semantically similar words at nearby positions in the vector space.

Negative Sampling Optimization

Computing the Softmax denominator directly requires traversing the entire vocabulary, which is prohibitively expensive. Negative sampling turns the multi-class problem into multiple binary-class problems:

$$\log \sigma (v_{w_{t+j}}^T{v}_{w_t})+\sum _{i=1}^{n_{\mathrm{neg}}}{\mathbb{E}}_{w_i\sim P_n(w)}\log \sigma (-v_{w_i}^T{v}_{w_t})$$

where $\sigma (x)=1/(1+e^{-x})$ and $n_{\mathrm{neg}}$ is the number of negative samples. The intuition: raise similarity for real word pairs, lower similarity for randomly sampled negative word pairs. This paradigm is exactly the technical cornerstone of subsequent recommender model training.

Analysis: Skip-Gram + negative sampling is efficient and scalable, the theoretical prototype for direct transfer to recommendation; but it works on "words", so user behavior sequences must be correctly mapped into training corpora before it applies to recommendation.

2.2.2 Item2Vec: The Most Direct Transfer

Item2Vec's core insight is exactly the "structural isomorphism" from the previous section: treat user interaction history as a "sentence" and items as "words".

Model Implementation

Item2Vec directly adopts Word2Vec's Skip-Gram, but simplifies sequence construction — each user's interaction history is treated as a set rather than a sequence, ignoring temporal weighting (the window still depends on positions $i+j$ after sorting by time; it just no longer assigns different weights to different positions). The objective function stays the same:

$$\mathcal{L}=\sum _{s\in \mathcal{S}}\sum _{l_i\in s}\sum _{-m\le j\le m,j\ne 0}\log P(l_{i+j}|l_i)$$

where $l_i$ is an item, $m$ is the window size, and $P(l_{i+j}|l_i)$ takes the same Softmax form as Word2Vec. After training, each item gets a dense vector, enabling nearest-neighbor search for I2I retrieval.

Analysis: Item2Vec is extremely easy to implement (a few gensim calls), validating the feasibility of sequence modeling in recommendation; but it treats history as an unordered set, losing temporal order, and is powerless against new-item cold start — no interactions, no vector. These two points are exactly EGES's motivation.

2.2.3 EGES: Enhancing Sequences with Attribute Information

Item2Vec treats interaction history as an unordered set and cannot handle cold start. EGES (Enhanced Graph Embedding with Side information) addresses both with two innovations: session-level graphs better reflect behavior patterns, and fusing side information solves sparsity and cold start.

Building the Item Relation Graph

EGES splits sessions by a "one-hour time window", building directed edges only between consecutive behaviors within the window, with edge weights as transition frequencies. Compared to treating the whole history as one sequence, this more accurately captures continuous interest transitions within specific periods. Weighted random walks on the graph generate training sequences, with transition probabilities determined by edge weights:

$$P(v_j|v_i)=\{\begin{array}{ll}\frac{M_{ij}}{{\sum }_{j=1}^{|N_{+}(v_i)|}{M}_{ij}}& \text{if }{v}_j\in N_{+}(v_i)\\ 0& \text{if }{e}_{ij}\notin E\end{array}$$

Fusing Side Information

Pure behavior sequences learn poorly for sparse items. GES first aggregates the item ID vector and attribute vectors with a simple average:

$$H_v=\frac{1}{n+1}\sum _{s=0}^n{W}_v^s$$

$W_v^s$ is the vector of the $s$-th attribute, and $W_v^0$ is the item ID vector. But averaging assumes all attributes are equally important, which clearly doesn't hold (phones hinge on brand, daily necessities on price).

EGES's core innovation is item-specific attention — learning a set of weights per item to emphasize the more important attributes:

$$H_v=\frac{{\sum }_{j=0}^n{e}^{a_v^j}{W}_v^j}{{\sum }_{j=0}^n{e}^{a_v^j}}$$

$a_v^j$ is a learnable weight. For cold-start new items with no behavior sequence and no trained $a_v^j$, EGES degrades to mean pooling over attribute vectors, directly obtaining a meaningful representation so the item can be included in I2I retrieval.

Training uses Word2Vec-style negative sampling, with the loss:

$$L(v,u,y)=-[y\log (\sigma (H_v^T{Z}_u))+(1-y)\log (1-\sigma (H_v^T{Z}_u))]$$

Analysis: EGES significantly alleviates sparsity and cold start with side information, outperforming traditional methods on billion-scale data; the cost is maintaining an attention parameter matrix of $|V|\times (n+1)$, raising engineering and storage costs. It represents the "balancing behavior and content" school of industrial I2I retrieval.

2.2.4 Airbnb: Baking Business Objectives into the Sequence

As a short-term rental platform, Airbnb has non-standard listings, bookings sparser than clicks, critical geography, and a stronger need to drive final booking conversion rather than mere similarity. It redefined the "sequence".

Business-Oriented Sequence Construction

• Session segmentation: a new session starts when the gap between user clicks exceeds 30 minutes, more accurately capturing coherent intent within a specific search context.
• Differentiated behavior weights: the final booking carries a much stronger preference signal than a plain click, and should receive higher weight in training.

The Global Context Mechanism

Traditional Skip-Gram only looks at local context within the sliding window. Airbnb forms positive pairs between the user's finally booked listing $l_b$ and every browsed listing in the sequence, no matter how far apart:

$$\underset{\theta }{\mathrm{argmax}}\sum _{(l,c)\in {\mathcal{D}}_p}\log \frac{1}{1+e^{-v_c^T{v}_l}}+\sum _{(l,c)\in {\mathcal{D}}_n}\log \frac{1}{1+e^{v_c^T{v}_l}}+\log \frac{1}{1+e^{-v_{l_b}^T{v}_l}}$$

The first two terms are standard Skip-Gram (positive/negative samples); the third term $\log \frac{1}{1+e^{-v_{l_b}^T{v}_l}}$ is the innovation — the booked listing provides an extra learning signal for every listing in the sequence, letting the model capture "what kinds of listing combinations ultimately lead to a booking".

Market-Aware Negative Sampling

Users typically only book within the same market (city/region). If negative samples come from other regions, the model easily learns the easy "geographic location" feature while ignoring differences between the listings themselves. Airbnb draws some negative samples from the same market:

$$\sum _{(l,l_m^{-})\in {\mathcal{D}}_m}\log \frac{1}{1+e^{v_{l_m^{-}}^T{v}_l}}$$

This forces the model to learn fine-grained differences between listings within the same region, improving discrimination.

Analysis: Airbnb writes "business conversion" and "geographic constraints" directly into the training objective, a model example of "business-objective-driven sequence construction"; but it is highly domain-customized (session thresholds and market partitions need business-specific tuning), less general than EGES.

⚠️ Common Mistakes in 2.2

Chapter Summary

📌 Key Takeaways

❓ FAQ

Q1: What is the essential difference between Item2Vec and Word2Vec?

A: The architecture and objective function are identical; the only difference is the corpus — Item2Vec treats user interaction history as "sentences" and item IDs as "words", and by default treats history as an unordered set (losing temporal order). Word2Vec processes real text sequences.

Q2: Are EGES's attention weights the same thing as Transformer attention?

A: Not exactly. EGES's attention is a weighted aggregation across multiple attribute sources of the same item (static, per-item), used to obtain a single item vector; Transformer attention is dynamic interaction between tokens within a sequence. Both are called attention, but they operate at different levels.

Q3: Why does Airbnb add global context instead of relying only on the sliding window?

A: The sliding window only sees local neighbors, missing the strongest positive signal — the final booking (which may be far from the browsed listings). Global context pairs the booked listing with every browsed listing, reinforcing the learning of "what combination leads to conversion".

🔗 Connections to Later Chapters

• 2.3 (Two-Tower Model) uses deep networks to encode user/item vectors, upgrading I2I's "item vectors" to "user-item joint vectors" for U2I retrieval.
• 2.4 (Sequential Retrieval) explicitly models temporal order (LSTM/capsules), fixing Item2Vec's loss of order.
• 2.5 (Streaming Index) organizes massive vector indexes with clustering and streaming VQ, carrying forward the item vectors learned in this chapter.

Practice Problems

Work through all problems in order — they get progressively harder. Each has a complete solution you can reveal after trying it yourself.

Problem 2.2.1 — Negative Sampling Intuition 🟢 Easy

The Softmax denominator of Skip-Gram requires traversing the entire vocabulary $|V|$. In recommendation, the item corpus can reach hundreds of millions. In one sentence, explain what problem negative sampling solves, and what form of task it converts the original objective into.

Problem 2.2.2 — The Item2Vec Mapping 🟢 Easy

Map the following text-world concepts to the recommendation world: (a) word (b) sentence (c) word co-occurrence. Also explain how Item2Vec is used for I2I retrieval after training.

Problem 2.2.3 — EGES Attention 🟡 Medium

EGES gives item $v$ a total of $n+1$ vectors ($n$ attributes plus ID), with attention weights $a_v^j$. Write the formula for the final vector $H_v$, and explain: if a phone's "brand" weight is far higher than its "price" weight, what would mean pooling (GES) lose?

Problem 2.2.4 — Airbnb Global Context 🔴 Hard

In the third term of Airbnb's objective, $\log \frac{1}{1+e^{-v_{l_b}^T{v}_l}}$, $l_b$ is the booked listing and $l$ is some browsed listing in the sequence. Explain: when $v_{l_b}^T{v}_l$ is large (semantically close), how does this term's contribution to the loss change? How does this help the model learn "combinations that lead to bookings"?

🏆 Challenge: Design a Cold-Start I2I Solution

A platform adds 100,000 new products daily, 80% of which receive fewer than 5 interactions in their first week. Write about 150 words explaining how you would use this chapter's methods (any combination of Item2Vec / EGES / Airbnb) to build an I2I retrieval pipeline where new products can be retrieved despite minimal interactions, and identify what kind of data must be paired with it.

Two-Tower Model (U2I)

📝 Before You Continue: Please first read matrix factorization in 2.1 and Item2Vec in 2.2. This chapter upgrades the "inner product" idea from item–item to user–item, encoding both sides with deep networks — the backbone of industrial U2I retrieval.

The item vectors you learned in the previous three sections (Item2Vec / EGES) perform I2I retrieval — first a seed item, then similar items. But online retrieval often starts not from an "item" but from a "user": given a user, pull out what they might like directly from a corpus of hundreds of millions of items. This requires a user representation and an item representation encoded separately in the same space, then retrieved by inner product — this is the two-tower model.

Its engineering charm lies in "divide and conquer": the item tower can be precomputed offline and stored in an approximate nearest-neighbor index (ANN), while the user tower computes in real time; online, only a single vector search is needed. From FM's mathematical prototype, to DSSM's deep encoding, to YouTubeDNN's "predict the next watch" — this chapter walks you through the two-tower evolution, with an interactive demo to build intuition for the retrieval process.

After reading this chapter, you will be able to:

• Derive the simplification of the FM second-order interaction term from $O(k{n}^2)$ to $O(kn)$, and show how it reorganizes into a two-tower inner product
• Explain the role of DSSM's extreme multi-class training, vector normalization, and the temperature coefficient
• Describe YouTubeDNN's "asymmetric two-tower" and "temporal split" engineering tricks
• Understand the two-tower retrieval flow of "offline index building / online user query" through the interactive demo
• Complete 5 graded practice problems, consolidating two-tower representation and retrieval

2.3.0 Why Two Towers: From I2I to U2I

I2I retrieval depends on "the user has interacted with some item" as a seed. But in many scenarios, the user just registered, or we want to recommend before they take any action. U2I directly uses the user themselves (profile + behavior) as the query, retrieving candidates from the full corpus:

The two-tower's core convention: the two towers barely interact during training, meeting only at the final inner product. This buys a precious engineering property — item vectors can be computed offline once and indexed, user vectors computed online on the fly, with retrieval cost kept minimal.

2.3.1 FM (Factorization Machines): The Prototype of Two-Tower Models

FM was born before deep learning, yet anticipated the two-tower in spirit. It elegantly decomposes complex user–item interactions into the inner product of two low-dimensional vectors. The full expression:

$$\hat{y}(\mathbf{x})=w_0+\sum _{i=1}^n{w}_i{x}_i+\sum _{i=1}^n\sum _{j=i+1}^n⟨{\mathbf{v}}_i,{\mathbf{v}}_j⟩x_i{x}_j$$

Each feature $i$ corresponds to a $k$-dimensional latent vector ${\mathbf{v}}_i$, with interactions modeled through the inner product $⟨{\mathbf{v}}_i,{\mathbf{v}}_j⟩={\sum }_{f=1}^k{v}_{i,f}{v}_{j,f}$.

Computational Complexity Simplification

The originally $O(n^2)$ second-order term can be rewritten as:

$$\sum _{i=1}^n\sum _{j=i+1}^n⟨{\mathbf{v}}_i,{\mathbf{v}}_j⟩x_i{x}_j=\frac{1}{2}\sum _{f=1}^k\left({\left(\sum _{i=1}^n{v}_{i,f}{x}_i\right)}^2-\sum _{i=1}^n{v}_{i,f}^2{x}_i^2\right)$$

Complexity drops from $O(k{n}^2)$ to $O(kn)$, letting FM handle large-scale sparse data.

🧠 Mental Model: Latent Vectors as Building Blocks

Think of each feature as a block hiding a small pointer (its latent vector). Whether two features "get along" doesn't depend on the blocks themselves, only on whether their pointers point in the same direction (big inner product = compatible). FM's elegance: instead of exhaustively trying all pairs, it computes all pairings at once via "sum of squares minus square of sums".

Decomposition into a Two-Tower Structure

In retrieval scenarios, features fall into two groups: user side $U$ and item side $I$. When recommending different items to the same user, interaction scores among user features are identical across all candidates and can be ignored during ranking. Keep only: intra-item interactions + user–item interactions. After rearrangement:

$${\text{score}}_{FM}=\sum _{t\in I}{w}_t{x}_t+\frac{1}{2}\sum _{f=1}^k\left({\left(\sum _{t\in I}{v}_{t,f}{x}_t\right)}^2-\sum _{t\in I}{v}_{t,f}^2{x}_t^2\right)+\sum _{f=1}^k\left(\sum _{u\in U}{v}_{u,f}{x}_u\sum _{t\in I}{v}_{t,f}{x}_t\right)$$

Look at the last term — it is exactly the inner product of two vectors ${\sum }_{u\in U}{v}_u{x}_u$ and ${\sum }_{t\in I}{v}_t{x}_t$. Hence:

$${\text{score}}_{FM}=V_{item}\cdot V_{user}^T$$

• User vector: $V_{user}=[1;{\sum }_{u\in U}{v}_u{x}_u]$
• Item vector: $V_{item}=[{\sum }_t{w}_t{x}_t+\frac{1}{2}{\sum }_f(({\sum }_t{v}_{t,f}{x}_t{)}^2-{\sum }_t{v}_{t,f}^2{x}_t^2);{\sum }_t{v}_t{x}_t]$

Analysis: FM uses linear algebra to turn feature crossings into a two-tower inner product, with item vectors precomputable offline — the theoretical prototype of the two-tower idea. But it is a linear model with limited expressiveness for complex nonlinear user–item relations — exactly why DSSM took over with deep networks.

2.3.2 DSSM: Deep Structured Semantic Model

DSSM replaces FM's linear transforms with deep neural networks, mapping users and items into a shared semantic space where similarity is measured by vector distance.

The Two-Tower Architecture in Recommendation

DSSM consists of two independent DNN towers: the user tower processes user features (behavior, demographics) and outputs a user embedding; the item tower processes item features (ID, category, attributes) and outputs an item embedding. The embedding dimensions of the two towers must match. Compared to FM's linear combination, DSSM lets each side perform complex nonlinear transforms within its own tower, with the two towers interacting only at the final inner product. Item vectors are precomputed offline, user vectors computed in real time, and retrieval is completed via ANN.

The Multi-Class Training Paradigm

DSSM treats retrieval as extreme multi-class classification: all items in the corpus are classes, and the goal is to maximize the predicted probability of the user's positive sample:

$$P(y|x,\theta )=\frac{e^{s(x,y)}}{{\sum }_{j\in M}{e}^{s(x,y_j)}}$$

$M$ is the entire corpus. Since the corpus is huge, negative sampling approximates it in practice.

Key Details of Two-Tower Models

Vector normalization: L2-normalize both embeddings, $u\leftarrow u/\Vert u{\Vert }_2,v\leftarrow v/\Vert v{\Vert }_2$. The raw dot product doesn't satisfy the triangle inequality, causing inconsistent "distances". After normalization, the dot product is equivalent to Euclidean distance:

$$\Vert u-v\Vert =\sqrt{2-2⟨u,v⟩}$$

The key is train/serve consistency — what training optimizes (normalized dot product) and what online ANN uses (Euclidean distance) are essentially equivalent, avoiding train-serving mismatch.

Temperature coefficient: divide the normalized inner product by $\tau$:

$$s(u,v)=\frac{⟨u,v⟩}{\tau }$$

$\tau <1$ amplifies similarity differences (more "confident"); $\tau >1$ smooths the distribution (more conservative). It essentially scales the logits, reshaping the Softmax output.

Analysis: DSSM's expressiveness comes from deep nonlinearity, and its engineering advantage from "offline item tower + online user tower". Normalization and temperature are the two must-tune knobs before launch — the former guarantees retrieval consistency, the latter controls retrieval "concentration". The cost is that the two-tower's "late interaction" loses some fine-grained feature-crossing signal.

2.3.3 YouTubeDNN: From Matching to Predicting the User's Next Action

YouTubeDNN is a milestone in two-tower evolution. It keeps the two-tower structure but introduces a key shift: defining retrieval as "predicting the user's next watched video", analogous to next-token prediction in NLP.

The Asymmetric Two-Tower Architecture

The user tower integrates multi-modal signals such as watch history, search history, and demographics; video IDs are embedded and aggregated by average pooling, with an Example Age feature introduced to model content freshness. The item tower is comparatively simple — essentially one huge embedding matrix, one learnable vector per video, avoiding complex item feature engineering. The objective is extreme multi-class classification:

$$P(w_t=i|U,C)=\frac{e^{v_i\cdot u}}{{\sum }_{j\in V}{e}^{v_j\cdot u}}$$

Since the video corpus is huge, Sampled Softmax enables efficient training.

Key Engineering Tricks

• Asymmetric temporal split: instead of random validation, use "roll-over" — the prediction target only sees history before it, avoiding future leakage (matching real recommendation, where episodes are watched in order).
• Negative sampling strategy: importance sampling, computing only thousands of negatives each time, speeding training up by over 100x.
• Per-user sample balancing: generate a fixed number of training samples per user, preventing highly active users from dominating learning — critical for long-tail user performance.

Analysis: YouTubeDNN established the "scalable, production-ready" two-tower paradigm: train with a complex multi-class objective + rich user features; at serving time precompute item vectors, compute user vectors in real time, and pair with ANN retrieval. The asymmetric design keeps the item tower lean (easy offline indexing) while the user tower scales flexibly — a balance still widely borrowed today.

2.3.4 Interactive Demo: The Two-Tower Retrieval Process

The interactive demo below lets you feel the core flow of two-tower retrieval: the item tower encodes all items into a vector index offline; online, a user arrives, the user tower encodes their vector in real time, and nearest-neighbor search pulls the most similar Top-K candidates from the index. Click "Next" to observe each step.

Note the third step, "retrieve": it doesn't traverse the full corpus scoring every item, but uses ANN to locate directly in the neighbor space — this is the fundamental reason a two-tower can serve a corpus of hundreds of millions at millisecond latency. Normalization makes the inner product equivalent to Euclidean distance, and the temperature coefficient controls how concentrated retrieval is.

📊 Data Point: On the funrec benchmark, FM retrieval achieves hit_rate@10≈0.047, DSSM≈0.016, YouTubeDNN≈0.013. The numeric differences mainly come from dataset and feature configuration, not model quality — DSSM/YouTubeDNN usually win with richer features and larger scale.

⚠️ Common Mistakes in 2.3

Chapter Summary

📌 Key Takeaways

❓ FAQ

Q1: Why is the two-tower's "late interaction" actually an advantage?

A: Because item vectors can be fully computed offline and indexed; online, only the user vector plus one ANN search is needed. If towers interacted early (e.g., feature crossing), item vectors would depend on the specific user and become impossible to precompute, losing scalability.

Q2: What are the effects of increasing vs. decreasing the temperature τ?

A: τ<1 amplifies similarity differences — the model is more "confident" and retrieval more concentrated (prone to head clustering); τ>1 smooths the distribution, more conservative, with more dispersed candidates. It's the knob balancing precision and diversity.

Q3: Both FM and DSSM use inner products — what's the difference?

A: FM's vectors come from linear combinations + fixed latent vectors (a linear model); DSSM's vectors come from nonlinear transforms of deep networks (more expressive), and it explicitly handles normalization and sampled training.

🔗 Connections to Later Chapters

• 2.4 (Sequential Retrieval) upgrades user representation from "single vector" to "multiple vectors / long-short fusion", fixing the single two-tower vector's loss of temporal order.
• 2.5 (Streaming Index) takes the item vectors produced by two-tower models and organizes them into a streaming index that updates in real time.
• 3.x (Ranking) the ranking side can use more complex "early interaction" models (such as feature crossing), complementing the two-tower's late interaction.

Practice Problems

Work through all problems in order — they get progressively harder. Each has a complete solution you can reveal after trying it yourself.

Problem 2.3.1 — FM Complexity Simplification 🟢 Easy

FM's second-order interaction term in its original form requires pairing $n$ features two by two, with complexity $O(k{n}^2)$. Write the simplified form, and explain why it only needs $O(kn)$.