Product-based Neural Networks for User Response Prediction【论文记录】

最新推荐文章于 2022-02-06 23:37:53 发布

Novelin

最新推荐文章于 2022-02-06 23:37:53 发布

阅读量244

点赞数

分类专栏：推荐系统文章标签：特征交叉向量内积

本文链接：https://blog.csdn.net/qq_40860934/article/details/110480135

版权

10 篇文章 1 订阅

订阅专栏

1 摘要

Deep models like deep neural networks, on the other hand, cannot be directly applied for the high-dimensional input because of the huge feature space
另一方面，由于巨大的特征空间，像深度神经网络这样的深度模型不能直接应用于高维输入。
Product-based Neural Networks (PNN) with an embedding layer to learn a distributed representation of the categorical data, a product layer to capture interactive patterns between interfield categories, and further fully connected layers to explore high-order feature interactions.
基于乘积的神经网络（PNN），具有一个嵌入层来学习分类数据的分布式表示，一个乘积层来捕获域间类别之间的交互模式，以及全连接层来探索高阶特征交互。

In order to improve the multi-field categorical data interaction, ¹ presented an embedding methodology based on pre-training of a factorization machine. However, the quality of embedding initialization is largely limited by the factorization machine
为了改善多字段分类数据的交互，¹ 提出了一种基于因子分解机预训练的嵌入方法。但是，嵌入初始化的质量在很大程度上受到分解机的限制。
Previous work has shown that local dependencies between features from different fields can be effectively explored by feature vector “product” operations instead of “add” operations.
先前的工作表明，可以通过特征向量“乘积”操作而不是“添加”操作来有效地探索来自不同字段的特征之间的局部依赖性。

Prediction Model (CCPM) was proposed in ² to predict ad click by convolutional neural networks (CNN). However, in CCPM the convolutions are only performed on the neighbor fields in a certain alignment, which fails to model the full interactions among non-neighbor features
² 中提出了预测模型（CCPM），以通过卷积神经网络（CNN）预测广告点击。但是，在 CCPM中，卷积仅以一定的对齐方式在相邻字段上执行，这无法对非相邻要素之间的完整交互进行建模

The task is to build a prediction model to estimate the probability of a user clicking a specific ad in a given context.
任务是建立一个预测模型，以估计用户在给定上下文中点击特定广告的概率。

From a top-down perspective, the output of PNN is a real number $\hat{y}\in(0,1)$ as the predicted CTR
从上到下看，PNN 的输出是 0 或 1 作为 CTR 的预测值
$\hat{y}=\sigma(W_3l_2+b_3) \tag{1}$ 其中， $\sigma(x)=1/(1+e^{-x})$
The output $l_2$ of the second hidden layer is constructed as
第二个隐藏层的输出 $l_2$ 构造为
$l_2 = relu(W_2l_1+b_2) \tag{2}$
The first hidden layer is fully connected with the product layer. The inputs to it consist of linear signals $l_z$ and quadratic signals $l_p$
第一个隐藏层与乘积层完全连接。它的输入包括线性信号 $l_z$ 和二次信号 $l_p$
$l_1 = relu(l_z+l_p+b_1) \tag{3}$

Product-based NN

let us define the operation of tensor inner product:
张量内积操作
$\boldsymbol{A} \odot \boldsymbol{B} \triangleq \sum_{i, j} \boldsymbol{A}_{i, j} \boldsymbol{B}_{i, j} \tag{4}$ 对 A、B 进行逐元素相乘，将乘法结果求和后为标量。
$l_z$ 和 $l_p$ 分别通过 $z$ 和 $p$ 计算得到
$\begin{array}{ll} \boldsymbol{l}_{z}=\left(l_{z}^{1}, l_{z}^{2}, \ldots, l_{z}^{n}, \ldots, l_{z}^{D_{1}}\right), & l_{z}^{n}=\boldsymbol{W}_{z}^{n} \odot \boldsymbol{z} \\ \\\boldsymbol{l}_{p}=\left(l_{p}^{1}, l_{p}^{2}, \ldots, l_{p}^{n}, \ldots, l_{p}^{D_{1}}\right), & l_{p}^{n}=\boldsymbol{W}_{p}^{n} \odot \boldsymbol{p} \end{array} \tag{5}$
线性信号 $z$ 和二次信号 $p$
$\begin{array}{l} \boldsymbol{z}=\left(\boldsymbol{z}_{1}, \boldsymbol{z}_{2}, \ldots, \boldsymbol{z}_{N}\right) \triangleq\left(\boldsymbol{f}_{1}, \boldsymbol{f}_{2}, \ldots, \boldsymbol{f}_{N}\right) \\ \\ \boldsymbol{p}=\left\{\boldsymbol{p}_{i, j}\right\}, i=1 \ldots N, j=1 \ldots N \end{array} \tag{6}$ 其中 $\boldsymbol{f}_i$ 是域 $i$ 的 embedding 向量， $\boldsymbol{p}_{i,j}=g(\boldsymbol{f}_i, \boldsymbol{f}_j)$ 可以定义为一对特征的任何操作

In IPNN, we firstly define the pairwise feature interaction as vector inner product: $\langle\boldsymbol{f}_{i}, \boldsymbol{f}_{j}\rangle$
在 IPNN 中，我们首先将成对特征交互定义为向量内积
Such pairwise connection expands the capacity of the neural network, but also enormously increases the complexity. Inspired by FM, we come up with the idea of matrix factorization to reduce complexity.
这种成对的连接扩展了神经网络的容量，但是也极大地增加了复杂性。受到 FM 的启发，我们提出了矩阵分解的思想以降低复杂性。

假设 $W^n_p=\theta^n\theta^{nT}$ ，这是具有强烈假设的一阶分解。
$\boldsymbol{W}_{p}^{n} \odot \boldsymbol{p}=\sum_{i=1}^{N} \sum_{j=1}^{N} \theta_{i}^{n} \theta_{j}^{n}\left\langle\boldsymbol{f}_{i}, \boldsymbol{f}_{j}\right\rangle=\langle\sum_{i=1}^{N} \boldsymbol{\delta}_{i}^{n}, \sum_{i=1}^{N} \boldsymbol{\delta}_{i}^{n}\rangle \tag{7}$ 这样，我们使用 $\delta^n_i=\theta^n_if_i$

一般的矩阵分解应为
$\boldsymbol{W}_{p}^{n} \odot \boldsymbol{p}=\sum_{i=1}^{N} \sum_{j=1}^{N}\left\langle\boldsymbol{\theta}_{n}^{i}, \boldsymbol{\theta}_{n}^{j}\right\rangle\left\langle\boldsymbol{f}_{i}, \boldsymbol{f}_{j}\right\rangle \tag{8}$

Vector inner product takes a pair of vectors as input and outputs a scalar. Different from that, vector outer product takes a pair of vectors and produces a matrix
向量内积将一对向量作为输入并输出标量。与此不同，向量外积采用一对向量并产生矩阵
The only difference between IPNN and OPNN is the quadratic term $p$ . In OPNN, we define feature interaction as $g(f_i,f_j)=f_if^T_j$
IPNN 和 OPNN 之间的唯一区别是二次项 $p$ 。在 OPNN 中，我们将特征交互定义为 $g(f_i,f_j)=f_if^T_j$

为了降低复杂度，提出了叠加的想法
$\boldsymbol{p}=\sum_{i=1}^{N} \sum_{j=1}^{N} \boldsymbol{f}_{i} \boldsymbol{f}_{j}^{T}=\boldsymbol{f}_{\Sigma}\left(\boldsymbol{f}_{\Sigma}\right)^{T}, \quad \boldsymbol{f}_{\Sigma}=\sum_{i=1}^{N} \boldsymbol{f}_{i} \tag{9}$

PNN*: This model has a product layer, which is a concatenation of inner product and outer product.
PNN *：此模型具有一个乘积层，该乘积层是内乘积和外乘积的连接。

We designed two types of PNN: IPNN based on inner product and OPNN based on outer product. We also discussed solutions to reduce complexity, making PNN efficient and scalable.
我们设计了两种类型的PNN：基于内乘积的 IPNN 和基于外乘积的 OPNN。我们还讨论了降低复杂性，使 PNN 高效且可扩展的解决方案。

W. Zhang, T. Du, and J. Wang, “Deep learning over multi-field categorical data: A case study on user response prediction,” ECIR, 2016. ↩︎ ↩︎
Q. Liu, F. Yu, S. Wu et al., “A convolutional click prediction model,” in CIKM. ACM, 2015, pp. 1743–1746. ↩︎ ↩︎