东方财富网-股吧论坛帖子信息采集

参考网址：http://guba.eastmoney.com/

 

采集目标：帖子标题，阅读数、评论数、链接、发布时间

代码如下：

import requests
from bs4 import BeautifulSoup
import time
import csv
import re
#复制请求头
head ={'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'Accept-Encoding':'gzip,deflate',
'Accept-Language':'zh-CN,zh;q=0.9',
'Cache-Control':'max-age=0',
'Connection':'keep-alive',
'Cookie':'st_pvi=87732908203428;st_si=12536249509085;qgqp_b_id=9777e9c5e51986508024bda7f12e6544;_adsame_fullscreen_16884=1',
'Host':'guba.eastmoney.com',
'Referer':'http://guba.eastmoney.com/list,600596,f_1.html',
'Upgrade-Insecure-Requests':'1',
'User-Agent':'Mozilla/5.0(WindowsNT6.1;Win64;x64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/65.0.3325.181Safari/537.36'}

#设置数据存储方式，csv表格写入
f = open('dfcw.csv','a',newline='')
w = csv.writer(f)

#获取帖子详细时间，列表也没有年份，可以作为获取帖子其他详细内容的通用方法
def get_time(url):
    try:
        q = requests.get(url,headers=head)
        soup = BeautifulSoup(q.text,'html.parser')
        ptime = soup.find('div',{'class':'zwfbtime'}).get_text()
        ptime = re.findall(r'\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}',ptime)[0]
        print(ptime)
        return ptime
    except:
        return ''

#获取列表页第n页的具体目标信息，由BeautifulSoup解析完成
def get_urls(url):
    baseurl = 'http://guba.eastmoney.com/'
    q = requests.get(url,headers=head)
    soup = BeautifulSoup(q.text,'html.parser')
    urllist = soup.findAll('div',{'class':'articleh'})
    print(len(urllist))
    for i in urllist:
        if i.find('a') != None:
            try:
                detailurl = i.find('a').attrs['href'].replace('/','')
                print(detailurl)
                titel = i.find('a').get_text()
                yuedu = i.find('span',{'class':'l1'}).get_text()
                pinlun = i.find('span', {'class': 'l2'}).get_text()
                ptime = get_time(baseurl+detailurl)
                w.writerow([detailurl,titel,yuedu,pinlun,ptime])
                print(baseurl + detailurl)
            except:
                pass
#循环所有页数
for i in range(1,101):
    print(i)
    get_urls('http://guba.eastmoney.com/list,600596,f_'+str(i)+'.html')

测试结果：

小结：

总的来说是一个比较简单的例子，帖子详情并没有采集，但是更新具体时间时已经进行了子链接的请求，可以顺便输出帖子的内容数据。

转自yqxmf.top