爬虫爬取基金数据（天天基金）

m0_53944811

已于 2024-08-27 20:02:02 修改

阅读量693

点赞数 3

文章标签： python 爬虫

于 2021-08-20 16:55:13 首次发布

本文链接：https://blog.csdn.net/m0_53944811/article/details/119826731

版权

本文介绍如何使用Python的request库获取天天基金网站的数据，再结合re和lxml模块进行数据分析，最后通过pylab库进行数据可视化展示，实现基金数据的爬取与分析。

摘要由CSDN通过智能技术生成

import requests
from apscheduler.schedulers.blocking import BlockingScheduler
import datetime
from pylab import *
import re
from lxml import html
etree=html.etree

def menu():
    print("///")
    number=input("请输入你的基金代码:")
    print("///")
    web="http://fundf10.eastmoney.com/jjjz_"+number+".html"
    return web
#获取基金代码整合成(天天基金的http）的函数

def find_web(fun_path):
    header={"Use-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36 Edg/92.0.902.67"}
    fun_web=requests.get(url=fun_path,headers=header).text
    tree=etree.HTML(fun_web)
    return tree
#运用request模块获取网页数据

def deal_text(tree):
    text = ''.join(tree.xpath('//*[@id="bodydiv"]/div[8]/div[3]/div[1]/div[1]/div[2]/p[1]//text()'))
    name=''.join(tree.xpath('//*[@id="bodydiv"]/div[8]/div[3]/div[1]/div[1]/div[1]/h4/a//text()'))
    name=re.findall(r'(.+)\(',name)
    print(name[0])
    today_text=text.replace("\n","").replace("\r","")
    data=today_text.split()
    print(data)
    return data,name[0]
#得到网页初始数据开始解析数据
def find_photo(tree):
    header={"Use-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36 Edg/92.0.902.67"}
    photo_path="http:"+tree.xpath('//*[@id="jzpng"]/@src')[0]
    photo_web=requests.get(url=photo_path,headers=header).content
    with open("today2.jpg","wb+") as fp:
        fp.write(photo_web)
#根据此网页分析可以获得本基金波动图照片
def write(data,name):
    with open (name+".txt","a+",encoding="utf-8") as fp:
        fp.write("\n")
        fp.writelines(data)
#将分析数据写入文件
def draw():
    data=[]
    with open("today2.txt","r",encoding="utf-8") as fp :
        lines=fp.readlines()
    for line in lines:
        k=re.findall(r"）：(.+)\(", line)
        k="".join(k)
        if k=="":
            continue
        data.append(k)
    x_axis_data=[i for i in range(len(data))]
    plt.plot(x_axis_data, data, 'ro-', alpha=0.8, label='波动图')
    plt.show()
    
#此函数用于画基金波动图像

def chance(name):
    try:
        data=[]
        with open(name+".txt","r",encoding="utf-8") as fp :
            lines=fp.readlines()
            for lines in lines:
                data.append(lines)
                pass
        return data[-1]
    except:
        return 0
#判断数据是否重复写入文件
def main():
    path=menu()
    tree = find_web(path)
    find_photo(tree)
    data ,name= deal_text(tree)
    
    
    k = " ".join(data).replace(" ", "")
    if k == chance(name):
        pass
    else:
        write(data,name)
        #draw()
#判断写入文件内容是否重复

if __name__ == '__main__':
   main()

运用知识

python基础知识

数据获取：request库应用

数据分析：模块re lxml

数据展示：pylab库