Wide & Deep

📝 Before You Continue: Please read 1.3 Feature and Embedding Basics first and understand sparse / dense features and lookup vectors; also recommended: finish the retrieval chapters of Part 2 so you know the engineering position of "candidates ready, ranking must score precisely."

When you search for a product in an app and the system recommends it, a ranking model computed scores for hundreds or thousands of candidates in milliseconds behind the scenes. But before 2016, industrial ranking models faced an awkward trade-off: either memorize historical patterns or learn to generalize — having both was hard.

The Wide & Deep model (Google, 2016) offered a plain yet far-reaching answer: since both capabilities are needed, design two components and let them train jointly, each doing its own job. It remains the baseline model for countless recommendation businesses and the starting point of all subsequent deep ranking models. After reading this chapter, you will not only be able to explain its structure, but also understand "why the split was designed this way."

After reading this chapter, you will be able to:

• Distinguish memorization from generalization in one sentence each, with a recommendation scenario example for both
• Write out the linear formula of the Wide part and explain how cross-product features embody memorization
• Explain how the Deep part achieves generalization via Embedding + DNN, and how it fundamentally differs from Wide
• Recite the prediction formula of jointly trained Wide & Deep, and explain why Wide / Deep often use different optimizers
• Work through 4 leveled practice problems to consolidate the "memorization + generalization" design idea

3.1.0 A Seemingly Contradictory Pair of Goals: Memorization and Generalization

When building recommendation models, we often pursue two goals at once: memorization and generalization.

• Memorization means the model learns and remembers feature combinations that frequently co-occur in historical data, e.g., "users who bought A usually also buy B." It precisely captures explicit, high-frequency associations and gives users highly relevant recommendations — but is powerless when facing combinations it has never seen.
• Generalization means the model learns deep relationships between features and can handle combinations rarely seen in training, e.g., "item A and item C belong to the same category; users who like A may also like C." Even if the user has never interacted with C, the model can still make a reasonable recommendation.

💡 Key Insight: Memorization makes recommendations "precise"; generalization makes them "broad." Memorization alone traps users in a filter bubble and cannot handle new items; generalization alone loses those high-value historical strong rules. The essence of Wide & Deep is to give one model both capabilities.

The memorization path on the left captures high-frequency strong rules like "people who buy A also buy B"; the generalization path on the right maps items into a vector space so the model can recommend similar items it has never seen (e.g., new books near The Three-Body Problem).

🧠 Mental Model: Veteran Employee vs Newcomer

Think of the Wide (memorization) part as an employee who has been at the company for twenty years: he remembers every historical "rule" (cross-product feature) — who always shows up with whom, he knows it cold. Think of the Deep (generalization) part as a systematically trained newcomer: he hasn't memorized all the rules, but knows how to reason by analogy and can handle combinations he has never seen. A good team needs both.

3.1.1 The Shortcut of Memorization: The Wide Part

The Wide part is essentially a generalized linear model (such as logistic regression). It is structurally simple, highly interpretable, and good at "memorizing" obvious association rules. Its mathematical form:

$$y={\mathbf{w}}^T\mathbf{x}+b$$

where $y$ is the prediction, $\mathbf{w}$ the weights, $\mathbf{x}$ the feature vector, and $b$ the bias.

The key of the Wide part is that the input $\mathbf{x}$ contains not only raw features but also a large number of manually designed cross-product features. A cross-product feature combines multiple independent features into a new one to capture specific co-occurrence patterns. For example, in app store recommendation we can construct:

AND(installed_app=photo_editor, impression_app=filter_pack)

This stands for "the user has installed a photo editor AND is currently shown a filter pack recommendation." Through such cross features, the Wide part can directly and quickly learn strong associations like "photo editor users have a higher willingness to install filter packs" — a direct embodiment of memorization.

Raw features on the left (installed apps, impression apps) are combined by a cross function into a new feature, which then looks up an independent weight table, directly "remembering" the co-occurrence strength of that pair.

💡 Key Insight: The essence of the Wide part's "memorization" is assigning an independent weight to every feature combination and directly remembering historical co-occurrences via lookup. The cost: these features must be hand-designed by experts, and they cannot generalize to unseen combinations.

3.1.2 Learning Complex Relations: The Deep Part

The Deep part is a standard feedforward neural network (DNN) responsible for the model's "generalization." Unlike Wide, which depends on manual feature engineering, the Deep part can learn high-order, nonlinear relationships between features automatically.

Its workflow has two steps. First, high-dimensional sparse categorical features (user ID, item ID) are mapped by an embedding layer to low-dimensional dense vectors — these vectors capture latent semantics. For example, the IDs of The Wandering Earth and The Three-Body Problem end up closer in the embedding space than The Wandering Earth and Boonie Bears. Then the embedding vectors are concatenated with other numerical features and fed forward through multiple layers:

$$a^{(l+1)}=f(W^{(l)}{a}^{(l)}+b^{(l)})$$

where $a^{(l)}$ is the activation of layer $l$, $W^{(l)}$, $b^{(l)}$ are weights and biases, and $f$ is an activation function (such as ReLU). Layer-by-layer abstraction lets the DNN discover hidden complex patterns and make reasonable predictions for unseen feature combinations.

Sparse categorical features are first embedded into dense vectors (similar items get close in the vector space), then concatenated with numerical features and fed into a multi-layer DNN, automatically learning high-order nonlinear relationships.

Analysis: The Deep part excels at generalization and automatically learning feature interactions, but its interpretability is weak — the high-order combinations it learns are hard to read directly; and for very high-frequency strong rules it may not "stick" as firmly as Wide's explicit crossings. Complexity mainly comes from the deep MLP, with parameter count growing with layer width and depth; embedding lookup is cheap.

3.1.3 Combining the Two: Joint Training

Wide & Deep jointly trains both parts and combines their outputs for the final prediction:

$$P(Y=1|\mathbf{x})=\sigma ({\mathbf{w}}_{wide}^T[\mathbf{x},\varphi (\mathbf{x})]+{\mathbf{w}}_{deep}^T{a}^{(lf)}+b)$$

Here $\sigma$ is the Sigmoid function, $[\mathbf{x},\varphi (\mathbf{x})]$ is Wide's input (raw + cross features), and $a^{(lf)}$ is the output vector of Deep's final layer. During backpropagation, gradients update all parameters of both Wide and Deep at once — this is "joint training," distinct from training separately and then ensembling.

An engineering detail worth noting: because the two parts handle parameters of different natures, they usually use different optimizers.

• The Wide part has sparse inputs, so the FTRL optimizer with L1 regularization is common. L1 produces sparse weights, effectively automatic feature selection, "remembering" only important rules.
• The Deep part has dense parameters, better suited to optimizers like AdaGrad / Adam.

Wide (linear + cross features) and Deep (Embedding + DNN) share the input, each produces a logit, and the sum passes through Sigmoid to output the final click probability. The two parts are jointly optimized during training.

💡 Key Insight: The significance of Wide & Deep goes beyond a new architecture: it established a paradigm — how to combine "memorization" and "generalization" into one end-to-end model. It became the baseline of many ranking models and foreshadows later chapters (replacing manual crossings with FM, replacing fixed user vectors with attention).

⚠️ Common Mistakes in 3.1

Chapter Summary

📌 Key Takeaways

❓ FAQ

Q1: Since Deep is so strong, can we drop Wide and keep only Deep?

A: Pure Deep often "fails to stick" on high-frequency strong rules — it implicitly encodes rules into weights, unlike Wide's direct lookup. For high-frequency historical co-occurrences, explicit memorization is more stable and interpretable. Keeping Wide still has value.

Q2: Do cross features always have to be manually designed?

A: Wide's cross features are manual — that is exactly its shortcoming. FM / DeepFM in Section 3.2 were proposed precisely to learn feature crossings automatically and escape manual feature engineering.

Q3: Why is joint training better than "train Wide first, then Deep"?

A: Joint training uses one loss to update both parts simultaneously, so Wide and Deep calibrate each other during training; training separately and ensembling yields two independent models that cannot optimize end-to-end together.

🔗 Connections to Later Chapters

• 3.2 (Feature Crossing) replaces Wide's manual cross features with automatic FM, evolving into DeepFM with shared embeddings.
• 3.3 (Sequence Modeling) further breaks through the "fixed user vector," introducing attention to dynamically activate history.
• 3.4 (Multi-Objective) extends Wide & Deep's "two-tower / shared-bottom" idea into multi-task shared structures.
• The two-tower use of FM in Part 2 retrieval and FM for crossings in this chapter are two threads of the same technique.

Practice Problems

Work through all problems in order — they get progressively harder. Each has a complete solution you can reveal after trying it yourself.

Problem 3.1.1 — Distinguish Memorization from Generalization 🟢 Easy

For each recommendation behavior below, decide whether it mainly relies on "memorization" or "generalization," and explain why:

• (a) The system recommends the same brand of infant formula again to a user who "just bought infant formula yesterday" — because historically "people who bought formula re-purchase the same kind within 7 days" at a very high rate.
• (b) The system recommends to a fan of The Three-Body Problem a new book the user has never seen, tagged "hard sci-fi" like The Three-Body Problem.

Problem 3.1.2 — Complete a Cross Feature 🟢 Easy

A food delivery app wants its Wide part to memorize a strong rule: "users who browsed fast food on weekdays at lunch (12:00–14:00) are more likely to click afternoon-tea coupons." Write the corresponding cross feature in AND(...) form.

AND(time_slot=weekday_lunch, browse_cate=fast_food, impression=afternoon_tea_coupon)

Problem 3.1.3 — The Joint Training Formula 🟡 Medium

The final prediction of Wide & Deep is $P(Y=1|\mathbf{x})=\sigma ({\mathbf{w}}_{wide}^T[\mathbf{x},\varphi (\mathbf{x})]+{\mathbf{w}}_{deep}^T{a}^{(lf)}+b)$. Answer:

1. In the formula, where do $[\mathbf{x},\varphi (\mathbf{x})]$ and $a^{(lf)}$ each come from?
2. Why do Wide and Deep usually use different optimizers? Give one example of each.

Problem 3.1.4 — Memorization Failure on New Combinations 🔴 Hard

Wide & Deep outputs $P=\sigma (w_{wide}^T[x,\varphi ]+w_{deep}^T{a}^{(lf)}+b)$. Suppose a brand-new feature combination $(a,b)$ (a cross of two high-cardinality IDs) appears online that never occurred in the training set. Analyze: (1) what does the Wide part contribute for this combination; (2) can the Deep part give non-zero generalization; (3) to improve prediction quality on this combination, is it more worthwhile to modify Wide or Deep?

🏆 Challenge: Argue a Design Trade-off

Suppose you own ranking for an e-commerce app with tens of millions of daily active users. The business requires "capturing historical high-frequency hit combinations while giving newly listed long-tail items a chance." In at most 150 words, argue whether Wide & Deep fits, and state what would be lost if only half of it were kept.