Deep Crossing模型是由微软提出,在微软的搜索引擎bing的搜索广告场景当中,用户除了会返回相关的结果,还会返回相应的广告,因此尽可能的增加广告的点击率,是微软所考虑的重中之重。
因此才设计出了Deep Crossing模型来解决这个问题。这个模型的结构如下所示:
最下面的各种feature是我们输入层,表示了针对特定的应用背景,微软使用的特征如下:
- Query(搜索):用户搜索的关键词;
- Keyword(广告关键词):广告商对自己的产品广告打的标签,用于匹配用户的搜索词;
- Title(标题):广告的标题;
- Landing Page(落地网页):点击广告后跳转的网页;
- Match Type(匹配类型):广告商可选择的关键字与用户查询的匹配程度,通常有四种:精确匹配、短语匹配、宽泛匹配和上下文相关匹配;
- Campaign(广告计划):广告商投放的计划;
- Imression(曝光样例):记录了该广告实际曝光场景的相关信息;
- Click(点击样例):记录了该广告实际点击场景的相关信息;
- Click Through Rate(点击率):广告的历史点击率
- click prediction(预估点击率):另一个CTR模型的预估值;
Embedding层
几乎所有基于深度学习的推荐、CTR预估模型都离不开Embedding层,它的作用是将离散高维的稀疏特征转化为低维的密集型特征。Embedding矩阵的参数通过神经网络的反向传播进行训练。在模型结构中发现Feature #2并没有使用Embedding,因为文章提到“维度小于256的特征“不需要进行Embedding转化。
Stacking层
Stacking层的工作特别简单,就是将所有的Embedding向量、或者未进行Embedding操作的原生特征进行拼接。
Multiple Residual Units层
Deep Crossing模型中的Crossing就是多个残差单元层来实现。该层使用了残差网络的基本单元。也就是我们通常所说的ResNet.
Scoring层
Scoring层就也是输出层。一般情况下对于CTR预估模型,往往是一个二分类问题,因此采用逻辑回归来对点击进行预测,正好逻辑回归模型将我们的CTR,也就是点击率,投射到一个从0-1的空间内,形成一个概率,其意义正好和CTR相同。如果要考虑是一个多分类问题,则可以使用softmax进行预测。
代码用tensorflow2实现如下:
1.model.py
import tensorflow as tf
from tensorflow.keras import Model
from tensorflow.keras.regularizers import l2
from tensorflow.keras.layers import Embedding, Dense, Dropout, Input
from modules import Residual_Units
class Deep_Crossing(Model):
def __init__(self, feature_columns, hidden_units, res_dropout=0., embed_reg=1e-6):
"""
Deep&Crossing
:param feature_columns: A list. sparse column feature information.
:param hidden_units: A list. Neural network hidden units.
:param res_dropout: A scalar. Dropout of resnet.
:param embed_reg: A scalar. The regularizer of embedding.
"""
super(Deep_Crossing, self).__init__()
self.sparse_feature_columns = feature_columns
self.embed_layers = {
'embed_' + str(i): Embedding(input_dim=feat['feat_num'],
input_length=1,
output_dim=feat['embed_dim'],
embeddings_initializer='random_uniform',
embeddings_regularizer=l2(embed_reg))
for i, feat in enumerate(self.sparse_feature_columns)
}
# the total length of embedding layers
embed_layers_len = sum([feat['embed_dim'] for feat in self.sparse_feature_columns])
self.res_network = [Residual_Units(unit, embed_layers_len) for unit in hidden_units]
self.res_dropout = Dropout(res_dropout)
self.dense = Dense(1, activation=None)
def call(self, inputs):
sparse_inputs = inputs
sparse_embed = tf.concat([self.embed_layers['embed_{}'.format(i)](sparse_inputs[:, i])
for i in range(sparse_inputs.shape[1])], axis=-1)
r = sparse_embed
for res in self.res_network:
r = res(r)
r = self.res_dropout(r)
outputs = tf.nn.sigmoid(self.dense(r))
return outputs
def summary(self):
sparse_inputs = Input(shape=(len(self.sparse_feature_columns),), dtype=tf.int32)
Model(inputs=sparse_inputs, outputs=self.call(sparse_inputs)).summary()
2.modules.py
import tensorflow as tf
from tensorflow.keras.layers import Dense, ReLU, Layer
class Residual_Units(Layer):
"""
Residual Units
"""
def __init__(self, hidden_unit, dim_stack):
"""
:param hidden_unit: A list. Neural network hidden units.
:param dim_stack: A scalar. The dimension of inputs unit.
"""
super(Residual_Units, self).__init__()
self.layer1 = Dense(units=hidden_unit, activation='relu')
self.layer2 = Dense(units=dim_stack, activation=None)
self.relu = ReLU()
def call(self, inputs, **kwargs):
x = inputs
x = self.layer1(x)
x = self.layer2(x)
outputs = self.relu(x + inputs)
return outputs
3.train.py
import tensorflow as tf
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras.losses import binary_crossentropy
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.metrics import AUC
from model import Deep_Crossing
from data_process.criteo import create_criteo_dataset
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
if __name__ == '__main__':
# =============================== GPU ==============================
# gpu = tf.config.experimental.list_physical_devices(device_type='GPU')
# print(gpu)
# If you have GPU, and the value is GPU serial number.
os.environ['CUDA_VISIBLE_DEVICES'] = '4'
# ========================= Hyper Parameters =======================
# you can modify your file path
file = '../dataset/Criteo/train.txt'
read_part = True
sample_num = 5000000
test_size = 0.2
embed_dim = 8
dnn_dropout = 0.5
hidden_units = [256, 128, 64]
learning_rate = 0.001
batch_size = 4096
epochs = 10
# ========================== Create dataset =======================
feature_columns, train, test = create_criteo_dataset(file=file,
embed_dim=embed_dim,
read_part=read_part,
sample_num=sample_num,
test_size=test_size)
train_X, train_y = train
test_X, test_y = test
# ============================Build Model==========================
mirrored_strategy = tf.distribute.MirroredStrategy()
with mirrored_strategy.scope():
model = Deep_Crossing(feature_columns, hidden_units)
model.summary()
# =========================Compile============================
model.compile(loss=binary_crossentropy, optimizer=Adam(learning_rate=learning_rate),
metrics=[AUC()])
# ============================model checkpoint======================
# check_path = 'save/deep_crossing_weights.epoch_{epoch:04d}.val_loss_{val_loss:.4f}.ckpt'
# checkpoint = tf.keras.callbacks.ModelCheckpoint(check_path, save_weights_only=True,
# verbose=1, period=5)
# ===========================Fit==============================
model.fit(
train_X,
train_y,
epochs=epochs,
callbacks=[EarlyStopping(monitor='val_loss', patience=2, restore_best_weights=True)], # checkpoint
batch_size=batch_size,
validation_split=0.1
)
# ===========================Test==============================
print('test AUC: %f' % model.evaluate(test_X, test_y, batch_size=batch_size)[1])