推荐系统实战

coding: utf-8

## 音乐推荐系统系列

by @寒小阳

### 0.数据获取

任何的机器学习算法解决问题，首先要考虑的是数据，数据从何而来？

* 对于酷狗音乐/网易音乐这样的企业而言，用户的收藏和播放数据是可以直接获得的

* 我们找一个取巧的方式，包含用户音乐兴趣信息，同时又可以获取的数据是什么？

* 对的，是歌单信息

### 1.数据说明

自己手动爬取了网易云音乐中80w首歌400w+次收藏的歌单，数据获取的过程不赘述，感兴趣的宝宝出门右拐爬虫班，存储格式为json格式，数据大小为3.59G，格式说明如下：

#### 1）每个歌单的格式

“`

{

“result”: {

“id”: 111450065,

“status”: 0,

“commentThreadId”: “A_PL_0_111450065”,

“trackCount”: 120,

“updateTime”: 1460164523907,

“commentCount”: 227,

“ordered”: true,

“anonimous”: false,

“highQuality”: false,

“subscribers”: [],

“playCount”: 687070,

“trackNumberUpdateTime”: 1460164523907,

“createTime”: 1443528317662,

“name”: “带本书去旅行吧,人生最美好的时光在路上。”,

“cloudTrackCount”: 0,

“shareCount”: 149,

“adType”: 0,

“trackUpdateTime”: 1494134249465,

“userId”: 39256799,

“coverImgId”: 3359008023885470,

“coverImgUrl”: “http://p1.music.126.net/2ZFcuSJ6STR8WgzkIi2U-Q==/3359008023885470.jpg“,

“artists”: null,

“newImported”: false,

“subscribed”: false,

“privacy”: 0,

“specialType”: 0,

“description”: “现在是一年中最美好的时节，世界上很多地方都不冷不热，有湛蓝的天空和清冽的空气，正是出游的好时光。长假将至，你是不是已经收拾行装准备出发了？行前焦虑症中把衣服、洗漱用品、充电器之类东西忙忙碌碌地丢进箱子，打进背包的时候，我打赌你肯定会留个位置给一位好朋友：书。不是吗？不管是打发时间，小读怡情，还是为了做好攻略备不时之需，亦或是为了小小地装上一把，你都得有一本书傍身呀。读大仲马，我是复仇的伯爵；读柯南道尔，我穿梭在雾都的暗夜；读村上春树，我是寻羊的冒险者；读马尔克斯，目睹百年家族兴衰；读三毛，让灵魂在撒哈拉流浪；读老舍，嗅着老北京的气息；读海茵莱茵，于科幻狂流遨游；读卡夫卡，在城堡中审判……读书的孩子不会孤单，读书的孩子永远幸福。”,

“subscribedCount”: 10882,

“totalDuration”: 0,

“tags”: [

“旅行”,

“钢琴”,

“安静”]

“creator”: {

“followed”: false,

“remarkName”: null,

“expertTags”: [

“古典”,

“民谣”,

“华语”

],

“userId”: 39256799,

“authority”: 0,

“userType”: 0,

“gender”: 1,

“backgroundImgId”: 3427177752524551,

“city”: 360600,

“mutual”: false,

“avatarUrl”: “http://p1.music.126.net/TLRTrJpOM5lr68qJv1IyGQ==/1400777825738419.jpg“,

“avatarImgIdStr”: “1400777825738419”,

“detailDescription”: “”,

“province”: 360000,

“description”: “”,

“birthday”: 637516800000,

“nickname”: “有梦人生不觉寒”,

“vipType”: 0,

“avatarImgId”: 1400777825738419,

“defaultAvatar”: false,

“djStatus”: 0,

“accountStatus”: 0,

“backgroundImgIdStr”: “3427177752524551”,

“backgroundUrl”: “http://p1.music.126.net/LS96S_6VP9Hm7-T447-X0g==/3427177752524551.jpg“,

“signature”: “漫无目的的乱听，听着，听着，竟然灵魂出窍了。更多精品音乐美图分享请加我微信hu272367751。微信是我的精神家园，有我最真诚的分享。”,

“authStatus”: 0}

“tracks”: [{歌曲1},{歌曲2}, …]

}

}

“`

#### 2）每首歌曲的格式为：

“`

{

“id”: 29738501,

“name”: “跟着你到天边 钢琴版”,

“duration”: 174001,

“hearTime”: 0,

“commentThreadId”: “R_SO_4_29738501”,

“score”: 40,

“mvid”: 0,

“hMusic”: null,

“disc”: “”,

“fee”: 0,

“no”: 1,

“rtUrl”: null,

“ringtone”: null,

“rtUrls”: [],

“rurl”: null,

“status”: 0,

“ftype”: 0,

“mp3Url”: “http://m2.music.126.net/vrVa20wHs8iIe0G8Oe7I9Q==/3222668581877701.mp3“,

“audition”: null,

“playedNum”: 0,

“copyrightId”: 0,

“rtype”: 0,

“crbt”: null,

“popularity”: 40,

“dayPlays”: 0,

“alias”: [],

“copyFrom”: “”,

“position”: 1,

“starred”: false,,

“starredNum”: 0

“bMusic”: {

“name”: “跟着你到天边 钢琴版”,

“extension”: “mp3”,

“volumeDelta”: 0.0553125,

“sr”: 44100,

“dfsId”: 3222668581877701,

“playTime”: 174001,

“bitrate”: 96000,

“id”: 52423394,

“size”: 2089713

},

“lMusic”: {

“name”: “跟着你到天边 钢琴版”,

“extension”: “mp3”,

“volumeDelta”: 0.0553125,

“sr”: 44100,

“dfsId”: 3222668581877701,

“playTime”: 174001,

“bitrate”: 96000,

“id”: 52423394,

“size”: 2089713

},

“mMusic”: {

“name”: “跟着你到天边 钢琴版”,

“extension”: “mp3”,

“volumeDelta”: -0.000265076,

“sr”: 44100,

“dfsId”: 3222668581877702,

“playTime”: 174001,

“bitrate”: 128000,

“id”: 52423395,

“size”: 2785510

},

“artists”: [

{

“img1v1Url”: “http://p1.music.126.net/6y-UleORITEDbvrOLV0Q8A==/5639395138885805.jpg“,

“name”: “群星”,

“briefDesc”: “”,

“albumSize”: 0,

“img1v1Id”: 0,

“musicSize”: 0,

“alias”: [],

“picId”: 0,

“picUrl”: “http://p1.music.126.net/6y-UleORITEDbvrOLV0Q8A==/5639395138885805.jpg“,

“trans”: “”,

“id”: 122455

}

],

“album”: {

“id”: 3054006,

“status”: 2,

“type”: null,

“tags”: “”,

“size”: 69,

“blurPicUrl”: “http://p1.music.126.net/2XLMVZhzVZCOunaRCOQ7Bg==/3274345629219531.jpg“,

“copyrightId”: 0,

“name”: “热门华语248”,

“companyId”: 0,

“songs”: [],

“description”: “”,

“pic”: 3274345629219531,

“commentThreadId”: “R_AL_3_3054006”,

“publishTime”: 1388505600004,

“briefDesc”: “”,

“company”: “”,

“picId”: 3274345629219531,

“alias”: [],

“picUrl”: “http://p1.music.126.net/2XLMVZhzVZCOunaRCOQ7Bg==/3274345629219531.jpg“,

“artists”: [

{

“img1v1Url”: “http://p1.music.126.net/6y-UleORITEDbvrOLV0Q8A==/5639395138885805.jpg“,

“name”: “群星”,

“briefDesc”: “”,

“albumSize”: 0,

“img1v1Id”: 0,

“musicSize”: 0,

“alias”: [],

“picId”: 0,

“picUrl”: “http://p1.music.126.net/6y-UleORITEDbvrOLV0Q8A==/5639395138885805.jpg“,

“trans”: “”,

“id”: 122455

}

],

“artist”: {

“img1v1Url”: “http://p1.music.126.net/6y-UleORITEDbvrOLV0Q8A==/5639395138885805.jpg“,

“name”: “”,

“briefDesc”: “”,

“albumSize”: 0,

“img1v1Id”: 0,

“musicSize”: 0,

“alias”: [],

“picId”: 0,

“picUrl”: “http://p1.music.126.net/6y-UleORITEDbvrOLV0Q8A==/5639395138885805.jpg“,

“trans”: “”,

“id”: 0

}

}

}

“`

### 2.数据解析

给大家原始数据和这份数据说明的原因是：里面包含非常多的信息(风格，歌手，歌曲播放次数，歌曲时长，歌曲发行时间…)，大家思考后一定会想到如何使用它们进一步完善推荐系统

我们这里依旧使用最基础的音乐信息，我们认为同一个歌单中的歌曲，有比较高的相似性，同时都是做单的同学喜欢的。

### 2.1 原始数据=>歌单数据

抽取 歌单名称，歌单id，收藏数，所属分类 4个歌单维度的信息

抽取 歌曲id，歌曲名，歌手，歌曲热度 等4个维度信息歌曲的信息

组织成如下格式：

“`

漫步西欧小镇上##小语种,旅行##69413685##474 18682332::Wäg vo dir::Joy Amelie::70.0 4335372::Only When I Sleep::The Corrs::60.0 2925502::Si Seulement::Lynnsha::100.0 21014930::Tu N’As Pas Cherché…::La Grande Sophie::100.0 20932638::Du behöver aldrig mer vara rädd::Lasse Lindh::25.0 17100518::Silent Machine::Cat Power::60.0 3308096::Kor pai kon diew : ชอไปคนเดียว::Palmy::5.0 1648250::les choristes::Petits Chanteurs De Saint Marc::100.0 4376212::Paddy’s Green Shamrock Shore::The High Kings::25.0 2925400::A Todo Color::Las Escarlatinas::95.0 19711402::Comme Toi::Vox Angeli::75.0 3977526::Stay::Blue Cafe::100.0 2538518::Shake::Elize::85.0 2866799::Mon Ange::Jena Lee::85.0 5191949::Je M’appelle Helene::Hélène Rolles::85.0 20036323::Ich Lieb’ Dich Immer Noch So Sehr::Kate & Ben::100.0

“`

# 下面这段代码大家先不要跑，我全量的数据还没放上来，再往下的代码是可以跑的。

In[ ]:

coding: utf-8

import json
import sys

def parse_song_line(in_line):
data = json.loads(in_line)
name = data[‘result’][‘name’]
tags = “,”.join(data[‘result’][‘tags’])
subscribed_count = data[‘result’][‘subscribedCount’]
if(subscribed_count<100):
return False
playlist_id = data[‘result’][‘id’]
song_info = ”
songs = data[‘result’][‘tracks’]
for song in songs:
try:
song_info += “\t”+”:::”.join([str(song[‘id’]),song[‘name’],song[‘artists’][0][‘name’],str(song[‘popularity’])])
except Exception, e:
#print e
#print song
continue
return name+”##”+tags+”##”+str(playlist_id)+”##”+str(subscribed_count)+song_info

def parse_file(in_file, out_file):
out = open(out_file, ‘w’)
for line in open(in_file):
result = parse_song_line(line)
if(result):
out.write(result.encode(‘utf-8’).strip()+”\n”)
out.close()

parse_file(“./playlist_detail_all.json”, “./163_music_playlist.txt”)

### 2.2 歌单数据=>推荐系统格式数据

主流的python推荐系统框架，支持的最基本数据格式为movielens dataset，其评分数据格式为 user item rating timestamp，为了简单，我们也把数据处理成这个格式。

In[ ]:

import surprise
import lightfm

## project = offline modelling + online prediction

### 1）offline

python脚本语言

### 2）online

效率至上 C++/Java

### 原则：能离线预先算好的，都离线算好，最优的形式：线上是一个K-V字典

### 1.针对用户推荐 网易云音乐(每日30首歌/7首歌)

### 2.针对歌曲 在你听某首歌的时候，找“相似歌曲”

In[1]:

coding: utf-8

解析成userid itemid rating timestamp行格式

import json
import sys

def is_null(s):
return len(s.split(“,”))>2

def parse_song_info(song_info):
try:
song_id, name, artist, popularity = song_info.split(“:::”)
#return “,”.join([song_id, name, artist, popularity])
return “,”.join([song_id,”1.0”,’1300000’])
except Exception,e:
#print e
#print song_info
return “”

def parse_playlist_line(in_line):
try:
contents = in_line.strip().split(“\t”)
name, tags, playlist_id, subscribed_count = contents[0].split(“##”)
songs_info = map(lambda x:playlist_id+”,”+parse_song_info(x), contents[1:])
songs_info = filter(is_null, songs_info)
return “\n”.join(songs_info)
except Exception, e:
print e
return False

def parse_file(in_file, out_file):
out = open(out_file, ‘w’)
for line in open(in_file):
result = parse_playlist_line(line)
if(result):
out.write(result.encode(‘utf-8’).strip()+”\n”)
out.close()

In[ ]:

parse_file(“./163_music_playlist.txt”, “./163_music_suprise_format.txt”)

In[2]:

parse_file(“./popular.playlist”, “./popular_music_suprise_format.txt”)

### 2.3 保存歌单和歌曲信息备用

我们需要保存 歌单id=>歌单名 和 歌曲id=>歌曲名 的信息后期备用。

In[3]:

coding: utf-8

import cPickle as pickle
import sys

def parse_playlist_get_info(in_line, playlist_dic, song_dic):
contents = in_line.strip().split(“\t”)
name, tags, playlist_id, subscribed_count = contents[0].split(“##”)
playlist_dic[playlist_id] = name
for song in contents[1:]:
try:
song_id, song_name, artist, popularity = song.split(“:::”)
song_dic[song_id] = song_name+”\t”+artist
except:
print “song format error”
print song+”\n”

def parse_file(in_file, out_playlist, out_song):
#从歌单id到歌单名称的映射字典
playlist_dic = {}
#从歌曲id到歌曲名称的映射字典
song_dic = {}
for line in open(in_file):
parse_playlist_get_info(line, playlist_dic, song_dic)
#把映射字典保存在二进制文件中
pickle.dump(playlist_dic, open(out_playlist,”wb”))
#可以通过 playlist_dic = pickle.load(open(“playlist.pkl”,”rb”))重新载入
pickle.dump(song_dic, open(out_song,”wb”))

In[4]:

parse_file(“./163_music_playlist.txt”, “playlist.pkl”, “song.pkl”)

In[4]:

parse_file(“./popular.playlist”, “popular_playlist.pkl”, “popular_song.pkl”)