TF-IDF探秘

1.TF:词频，IDF:逆文本概率。存在的文本个数越多，值越小。 用户量改成pv？
2.IDF计算：IDF(x)=log(N/N(x)),N为文本总数，N(x)为包含x的文本数 未出现的话，N(x)=0 ----》IDF(x)=log(N/(N(x)+1)

工具调研：
1）结巴分词 ok。同时可以进行关键词提取
2）nlpc-wordrank：Term重要性和主干分析。
Term重要性对组成query的term进行打分，并作归一化处理，所有term重要性之和为1。 与idf不同的是，单term重要性利用了许多统计信息和query迭代算法，使同一query内部term间相对权值更具有可比性。单term重要性可用于检索系统的基础归并权值计算、query中心词提取、query省略等领域。

同一文本测试：

{
    "ret": 0, 
    "msg": "", 
    "content": {
        "nlpc_trunks_pb": [
            {
                "weight": 0.0760907, 
                "buffer": "易", 
                "rank": 1, 
                "reduce_type": 7, 
                "reserve": 0, 
                "term_level": 0, 
                "type": 0
            }, 
            {
                "weight": 0.0730944, 
                "buffer": "烊", 
                "rank": 1, 
                "reduce_type": 7, 
                "reserve": 0, 
                "term_level": 0, 
                "type": 0
            }, 
            {
                "weight": 0.128541, 
                "buffer": "千玺", 
                "rank": 2, 
                "reduce_type": 7, 
                "reserve": 0, 
                "term_level": 0, 
                "type": 0
            }, 
            {
                "weight": 0.0192157, 
                "buffer": "的", 
                "rank": 0, 
                "reduce_type": 2, 
                "reserve": 0, 
                "term_level": 0, 
                "type": 0
            }, 
            {
                "weight": 0.105014, 
                "buffer": "粉丝", 
                "rank": 1, 
                "reduce_type": 7, 
                "reserve": 0, 
                "term_level": 0, 
                "type": 0
            }, 
            {
                "weight": 0.0380482, 
                "buffer": "参加", 
                "rank": 1, 
                "reduce_type": 7, 
                "reserve": 0, 
                "term_level": 0, 
                "type": 0
            }, 
            {
                "weight": 0.017015, 
                "buffer": "了", 
                "rank": 0, 
                "reduce_type": 2, 
                "reserve": 0, 
                "term_level": 0, 
                "type": 0
            }, 
            {
                "weight": 0.107997, 
                "buffer": "生日", 
                "rank": 1, 
                "reduce_type": 7, 
                "reserve": 0, 
                "term_level": 0, 
                "type"