#Home Depot 产品相关性预测 kaggle竞赛：https://www.kaggle.com/c/home-depot-product-search-relevance HomeDepot是美国一家家具建材商品网站，用户通过在搜索框中输入关键词，得到相关商品和服务，如输入floor，得到不同材料的地板商品、地板清洗商品、地板安装服务等。kaggle竞赛目的是通过设计一种模型，能够更好的匹配用户搜索关键词，得到相关性更高的产品和服务。 ##导入所需
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor, BaggingRegressor
df_train = pd.read_csv('train.csv',encoding='ISO-8859-1')
df_test = pd.read_csv('test.csv',encoding='ISO-8859-1')#除了train test数据外，还有一个商品描述数据
df_desc = pd.read_csv('product_descriptions.csv')#看一下各数据的样子
df_train.head(3)
.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }
| id | product_uid | product_title | search_term | relevance | |
|---|---|---|---|---|---|
| 0 | 2 | 100001 | Simpson Strong-Tie 12-Gauge Angle | angle bracket | 3.0 |
| 1 | 3 | 100001 | Simpson Strong-Tie 12-Gauge Angle | l bracket | 2.5 |
| 2 | 9 | 100002 | BEHR Premium Textured DeckOver 1-gal. #SC-141 … | deck over | 3.0 |
df_test.head(3)
.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }
| id | product_uid | product_title | search_term | |
|---|---|---|---|---|
| 0 | 1 | 100001 | Simpson Strong-Tie 12-Gauge Angle | 90 degree bracket |
| 1 | 4 | 100001 | Simpson Strong-Tie 12-Gauge Angle | metal l brackets |
| 2 | 5 | 100001 | Simpson Strong-Tie 12-Gauge Angle | simpson sku able |
df_desc.head(3)
.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }
| product_uid | product_description | |
|---|---|---|
| 0 | 100001 | Not only do angles make joints stronger, they … |
| 1 | 100002 | BEHR Premium Textured DECKOVER is an innovativ… |
| 2 | 100003 | Classic architecture meets contemporary design… |
train中relevance是我们要在test上预测的目标，relevance 1-3代表相关程度，3最高，1最低；search_term是搜索词，即该产品在某一搜索词下的相关度是多少；product discription里是对应产品id的产品介绍。
对train和test数据进行合并方便处理，同时在描述数据中product_uid是共同特征，也可合并进去。
df_all = pd.concat((df_train, df_test), axis=0, ignore_index=True)
#两个表的index都没有实际含义，选择忽视，axis=0按行合并df_all.head(3)
.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }
| id | product_title | product_uid | relevance | search_term | |
|---|---|---|---|---|---|
| 0 | 2 | Simpson Strong-Tie 12-Gauge Angle | 100001 | 3.0 | angle bracket |
| 1 | 3 | Simpson Strong-Tie 12-Gauge Angle | 100001 | 2.5 | l bracket |
| 2 | 9 | BEHR Premium Textured DeckOver 1-gal. #SC-141 … | 100002 | 3.0 | deck over |
df_all.shapedf_all = df_all.merge(df_desc,on='product_uid',how='left')df_all.head(3)
.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }
##文本预处理
| id | product_title | product_uid | relevance | search_term | product_description | |
|---|---|---|---|---|---|---|
| 0 | 2 | Simpson Strong-Tie 12-Gauge Angle | 100001 | 3.0 | angle bracket | Not only do angles make joints stronger, they … |
| 1 | 3 | Simpson Strong-Tie 12-Gauge Angle | 100001 | 2.5 | l bracket | Not only do angles make joints stronger, they … |
| 2 | 9 | BEHR Premium Textured DeckOver 1-gal. #SC-141 … | 100002 | 3.0 | deck over | BEHR Premium Textured DECKOVER is an innovativ… |
from nltk.stem.snowball import SnowballStemmer
import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from sklearn.feature_extraction.text import TfidfVectorizer
#去掉停止词
stop = stopwords.words('english')
#去掉数字
import re
def hasnumber(input_str):
return bool(re.search(r'\d',input_str))
#整合在一起
def check(string):
if string in stop:
return False
elif hasnumber(string):
return False
else:
return True#清洁文本内容
stemmer = SnowballStemmer('english')#提取词干
def text_stemmer(s):
return ' '.join([stemmer.stem(word) for word in s.lower().split() if check(word)])#应用
df_all['search_term'] = df_all['search_term'].map(lambda x: text_stemmer(x))
df_all['product_title'] = df_all['product_title'].map(lambda x:text_stemmer(x))
df_all['product_description'] = df_all['product_description'].map(lambda x:text_stemmer(x))df_all.head()
.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }
###处理训练数据
| id | product_title | product_uid | relevance | search_term | product_description | |
|---|---|---|---|---|---|---|
| 0 | 2 | simpson strong-ti angl | 100001 | 3.00 | angl bracket | angl make joint stronger, also provid consiste… |
| 1 | 3 | simpson strong-ti angl | 100001 | 2.50 | l bracket | angl make joint stronger, also provid consiste… |
| 2 | 9 | behr premium textur deckov tugboat wood concre… | 100002 | 3.00 | deck | behr premium textur deckov innov solid color c… |
| 3 | 16 | delta vero shower faucet trim kit chrome (valv… | 100005 | 2.33 | rain shower head | updat bathroom delta vero single-handl shower … |
| 4 | 17 | delta vero shower faucet trim kit chrome (valv… | 100005 | 2.67 | shower faucet | updat bathroom delta vero single-handl shower … |
#对训练数据构造全部单词合集
train = df_all[:df_train.shape[0]]
test = df_all[df_test.shape[0]:]train['all_text']=train['product_title'] + ' . ' + train['product_description'] + ' . '
test['all_text'] = test['product_title'] + ' . ' + test['product_description'] + ' . 'train['all_text'][0:5]from gensim.utils import tokenize
#使用的是gensim库中的tokenize方法，所以后续对test的处理要一致
from gensim.corpora.dictionary import Dictionary
dictionary = Dictionary(list(tokenize(x, errors='ignore')) for x in train['all_text'].values)
print(dictionary)class corpus:
def __iter__(self):
for x in train['all_text'].values:
yield dictionary.doc2bow(list(tokenize(x, errors='ignore')))
#dictionary.doc2bow是获得语料的向量表达式train_corpus = corpus()count=0
for c in train_corpus:
print(c)
count+=1
if count >2:
breakfrom gensim.models.tfidfmodel import TfidfModel
tfidf_g = TfidfModel(train_corpus)#保存模型
tfidf_g.save("./gensim_tfidf.tfidf")tfidf_g[dictionary.doc2bow(list(tokenize('morning yellow flower', errors='ignore')))]#进行封装
def to_tfidf(text):
res = tfidf_g[dictionary.doc2bow(list(tokenize(text, errors='ignore')))]
return res
from gensim.similarities import MatrixSimilarity
def cos_sim(text1,text2):
tf1 = to_tfidf(text1)
tf2 = to_tfidf(text2)
index = MatrixSimilarity([tf1],num_features=len(dictionary))
sim = index[tf2]
return float(sim[0])train['tfidf_cos_sim_in_title'] = train.apply(lambda x: cos_sim(x['search_term'], x['product_title']), axis=1)test['tfidf_cos_sim_in_title'] = test.apply(lambda x: cos_sim(x['search_term'], x['product_title']), axis=1)train['tfidf_cos_sim_in_desc'] = train.apply(lambda x: cos_sim(x['search_term'], x['product_description']), axis=1)test['tfidf_cos_sim_in_desc'] = test.apply(lambda x: cos_sim(x['search_term'], x['product_description']), axis=1)train.head(2)
.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }
| id | product_title | product_uid | relevance | search_term | product_description | all_text | tfidf_cos_sim_in_title | tfidf_cos_sim_in_desc | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | 2 | simpson strong-ti angl | 100001 | 3.0 | angl bracket | angl make joint stronger, also provid consiste… | simpson strong-ti angl . angl make joint stron… | 0.287958 | 0.188301 |
| 1 | 3 | simpson strong-ti angl | 100001 | 2.5 | l bracket | angl make joint stronger, also provid consiste… | simpson strong-ti angl . angl make joint stron… | 0.000000 | 0.000000 |
test.head(2)
.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }
| id | product_title | product_uid | relevance | search_term | product_description | all_text | tfidf_cos_sim_in_title | tfidf_cos_sim_in_desc | |
|---|---|---|---|---|---|---|---|---|---|
| 166693 | 138080 | winix freshom model true hepa air cleaner plas… | 149579 | NaN | winix air purifi | winix freshom true hepa air cleaner plasmawav … | winix freshom model true hepa air cleaner plas… | 0.413249 | 0.146256 |
| 166694 | 138082 | ge rv outlet box amp volt ring type meter amp … | 149580 | NaN | gcfi outlet | ring-typ meter surfac mount factory-assembl fa… | ge rv outlet box amp volt ring type meter amp … | 0.530571 | 0.022509 |
可以看出我们增加了2个特征，即我们通过TF-IDF模型将单词转换为向量表示，计算两个向量的余弦值作为相似性度量，度量search item与title和product description的相似性。
###word2vec模型 ####tokenize分词tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')#一段文章分成各个句子
tokenizer.tokenize(train['all_text'].values[0])#应用到train样本
sentences = [tokenizer.tokenize(x) for x in train['all_text'].values]type(sentences)#得到list of句子sentences[:2]sentences = [y for x in sentences for y in x]len(sentences)#各个句子分成单词
words = [word_tokenize(x) for x in sentences]from gensim.models.word2vec import Word2Vec
w2c = Word2Vec(words, size=128, window=5, min_count=5, workers=4)w2c['door'].shapevocab = w2c.wv.vocab
#注：对于gensim 1.0.0后的版本，要想获得w2c向量字典，需要使用方法model.wv.vocab
#对于以前的版本，使用model.vocab即可type(vocab)# 得到任意text的vector
def get_vector(text):
# 全是0的,大小是128的array
res =np.zeros([128])
count = 0
for word in word_tokenize(text):
if word in vocab:
res += w2c[word]
count += 1
return res/countprint(get_vector('this is a door'))from scipy import spatial
def w2c_cos_sim(text1,text2):
try:
w1 = get_vector(text1)
w2 = get_vector(text2)
sim = 1 - spatial.distance.cosine(w1, w2)
return float(sim)
except:
return float(0)
#spatial.distance.cosine定义余弦距离为1-cosw2c_cos_sim('hello world','hello from the other side')train['w2v_cos_sim_in_title'] = train.apply(lambda x: w2c_cos_sim(x['search_term'], x['product_title']), axis=1)
train['w2v_cos_sim_in_desc'] = train.apply(lambda x: w2c_cos_sim(x['search_term'], x['product_description']), axis=1)
test['w2v_cos_sim_in_title'] = test.apply(lambda x: w2c_cos_sim(x['search_term'], x['product_title']), axis=1)
test['w2v_cos_sim_in_desc'] = test.apply(lambda x: w2c_cos_sim(x['search_term'], x['product_description']), axis=1)train.head(2)
.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }
| id | product_title | product_uid | relevance | search_term | product_description | all_text | tfidf_cos_sim_in_title | tfidf_cos_sim_in_desc | w2v_cos_sim_in_title | w2v_cos_sim_in_desc | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2 | simpson strong-ti angl | 100001 | 3.0 | angl bracket | angl make joint stronger, also provid consiste… | simpson strong-ti angl . angl make joint stron… | 0.287958 | 0.188301 | 0.531925 | 0.530175 |
| 1 | 3 | simpson strong-ti angl | 100001 | 2.5 | l bracket | angl make joint stronger, also provid consiste… | simpson strong-ti angl . angl make joint stron… | 0.000000 | 0.000000 | 0.279708 | 0.303249 |
test.head(2)
.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }
| id | product_title | product_uid | relevance | search_term | product_description | all_text | tfidf_cos_sim_in_title | tfidf_cos_sim_in_desc | w2v_cos_sim_in_title | w2v_cos_sim_in_desc | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 166693 | 138080 | winix freshom model true hepa air cleaner plas… | 149579 | NaN | winix air purifi | winix freshom true hepa air cleaner plasmawav … | winix freshom model true hepa air cleaner plas… | 0.413249 | 0.146256 | 0.694583 | 0.591168 |
| 166694 | 138082 | ge rv outlet box amp volt ring type meter amp … | 149580 | NaN | gcfi outlet | ring-typ meter surfac mount factory-assembl fa… | ge rv outlet box amp volt ring type meter amp … | 0.530571 | 0.022509 | 0.676237 | 0.369881 |
#去掉文本列
train = train.drop(['search_term','product_title','product_description','all_text'],axis=1)
test = test.drop(['search_term','product_title','product_description','all_text'],axis=1)#保留id，提取目标
ids = test['id']
y_train = train['relevance'].values
X_train = train.drop(['id','relevance'],axis=1).values
X_test = test.drop(['id','relevance'],axis=1).valuesmodel
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import cross_val_scoreparams = [1,3,5,6,7,8,9,10]
test_scores = []
for param in params:
clf = RandomForestRegressor(n_estimators=30, max_depth=param)
test_score = np.sqrt(-cross_val_score(clf, X_train, y_train, cv=10, scoring='neg_mean_squared_error'))
test_scores.append(np.mean(test_score))import matplotlib.pyplot as plt
%matplotlib inline
plt.plot(params, test_scores)
plt.title("Param vs CV Error")<matplotlib.text.Text at 0x26281610c88>
可以看出，max_depth在6的时候效果最好，大约在0.49左右。这里是增加4个特征，选择的模型是随机森林，下一步，可以构造新的特征，如简单的search item是否被包含，还可以使用别的模型，如LR，然后对模型进行ensemble等。
新手学习，欢迎指教！
2885
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
