开发者日志-下游项目分析

Yifanan

已于 2022-12-13 11:14:58 修改

阅读量258

点赞数 1

文章标签： python 开发语言爬虫

于 2022-12-10 19:15:06 首次发布

本文链接：https://blog.csdn.net/Nana7mii/article/details/128267984

版权

下面简单介绍一下在OpenEuler组成分析中的下游项目分析。

爬虫

数据爬取就是利用python的爬虫简单将OpenEuler库中的软件进行爬取即可。

一些需要注意的点：

访问限制
Gitee平台的访问限制很奇怪，在我对https://gitee.com/organizations/src-openeuler/projects进行爬取的时候，调整到page=21的时候，会访问失败，需要你进行登录。
此时我们只需要调整headers即可，在里面加入浏览器的Cookies，就可以继续爬取了。
数据处理
为了方便后续分析和重新爬取每一个项目的仓库，将第一步爬取下来的库，按照title-herf进行存储，随后通过遍历访问，对每一个herf进行爬取，就可以得到仓库的具体属性

主题分类

主题分类采用LDA模型开展。

首先对描述集进行分词，分词后读入数据：

def load_data():
    file = open("OpenEuler_description.csv", encoding="utf-8", errors='ignore').read().split("\n")
    data_set = []
    for i in range(len(file)):
        result = []
        seg_list = file[i].split()
        for w in seg_list:
            result.append(w)
        filtered_words = [word for word in result if word not in stopwords.words('english')]
        data_set.append(filtered_words)
    dictionary = corpora.Dictionary(data_set)
    corpus = [dictionary.doc2bow(text) for text in data_set]# 表示为第几个单词出现了几次
    return data_set, dictionary, corpus

随后根据coherence值来判断应该将主题数确定为几，通过画出主题-coherence图，最高峰即为主题数：

def coherence(num_topics):
    data_set, dictionary, corpus = load_data()
    ldamodel = LdaModel(corpus, num_topics=num_topics, id2word=dictionary, passes=30, random_state=1)
    ldacm = CoherenceModel(model=ldamodel, texts=data_set, dictionary=dictionary, coherence='c_v')
    print(ldacm.get_coherence())
    return ldacm.get_coherence()

def draw():
    x = range(1,15)
    y = [coherence(i) for i in x]
    plt.plot(x, y)
    plt.xlabel("主题数目")
    plt.ylabel('coherence大小')
    plt.rcParams['font.sans-serif'] = ['SimHei']
    matplotlib.rcParams['axes.unicode_minus'] = False
    plt.title('主题-coherence变化情况')
    plt.show()

本项目得到的主题数为7，将7作为参数输入lda模型之中，生成可视化网页：

def build():
    data_set, dictionary, corpus = load_data()
    lda = LdaModel(corpus=corpus, id2word=dictionary, num_topics=7, passes=30, random_state=1)
    topic_list = lda.print_topics()
    print(topic_list)

    for i in lda.get_document_topics(corpus)[:]:
        listj = []
        for j in i:
            listj.append(j[1])
        bz = listj.index(max(listj))
        print(i[bz][0])
    data = pyLDAvis.gensim.prepare(lda, corpus, dictionary, mds='mmds')
    pyLDAvis.save_html(data, '7topic.html')

完整代码如下：

# coding: utf-8
import gensim
from gensim import corpora
import matplotlib.pyplot as plt
import matplotlib
import numpy as np
import warnings
import pyLDAvis.gensim

warnings.filterwarnings('ignore')

from gensim.models.coherencemodel import CoherenceModel
from gensim.models.ldamodel import LdaModel
from nltk.corpus import stopwords

def load_data():
    file = open("OpenEuler_description.csv", encoding="utf-8", errors='ignore').read().split("\n")
    data_set = []
    for i in range(len(file)):
        result = []
        seg_list = file[i].split()
        for w in seg_list:
            result.append(w)
        filtered_words = [word for word in result if word not in stopwords.words('english')]
        data_set.append(filtered_words)
    dictionary = corpora.Dictionary(data_set)
    corpus = [dictionary.doc2bow(text) for text in data_set]# 表示为第几个单词出现了几次
    return data_set, dictionary, corpus


def coherence(num_topics):
    data_set, dictionary, corpus = load_data()
    ldamodel = LdaModel(corpus, num_topics=num_topics, id2word=dictionary, passes=30, random_state=1)
    ldacm = CoherenceModel(model=ldamodel, texts=data_set, dictionary=dictionary, coherence='c_v')
    print(ldacm.get_coherence())
    return ldacm.get_coherence()

def draw():
    x = range(1,15)
    y = [coherence(i) for i in x]
    plt.plot(x, y)
    plt.xlabel("主题数目")
    plt.ylabel('coherence大小')
    plt.rcParams['font.sans-serif'] = ['SimHei']
    matplotlib.rcParams['axes.unicode_minus'] = False
    plt.title('主题-coherence变化情况')
    plt.show()

def build():
    data_set, dictionary, corpus = load_data()
    lda = LdaModel(corpus=corpus, id2word=dictionary, num_topics=7, passes=30, random_state=1)
    topic_list = lda.print_topics()
    print(topic_list)

    for i in lda.get_document_topics(corpus)[:]:
        listj = []
        for j in i:
            listj.append(j[1])
        bz = listj.index(max(listj))
        print(i[bz][0])
    data = pyLDAvis.gensim.prepare(lda, corpus, dictionary, mds='mmds')
    pyLDAvis.save_html(data, '7topic.html')


if __name__ == '__main__':
    #draw()
    build()