selenium结合接口获取数据

最新推荐文章于 2024-09-13 19:50:02 发布

醒时风拂衣，情动忽提笔

最新推荐文章于 2024-09-13 19:50:02 发布

阅读量207

点赞数 3

文章标签： selenium python

本文链接：https://blog.csdn.net/bigdatasyr/article/details/141360688

版权

起因是网站登录使用selenium输入账号密码后还需要点击图片验证
于是手动输入账号密码获取cookie绕过登录
这里用到selenium获取cookie保存到本地，然后接下来使用request读取本地cookie绕过登录
参考：https://blog.csdn.net/qq_42348937/article/details/85065104

def Get_hisnav(fund_id,cookie_dict):
    j=1

    for i in range(1,len(fund_id)):
        while True:
            url='https://simu.eastmoney.com/PinzhongF10DataApi.aspx?type=lsjz&fc='+str(fund_id[i])+'&pageindex='+str(j)+'&pagesize=10&lsjzSDate=&lsjzEDate=&'
            payload = {}
            headers = {

                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36',
                'Accept': '*/*',
                'Host': 'simu.eastmoney.com',
                'Connection': 'keep-alive'
            }

            response = requests.request("GET", url,  headers=headers, data=payload,cookies=cookie_dict)
            res=response.text
            if len(res) > 100:
                data_dict = eval(res[11:])
                nav_datas = data_dict['Datas']
                df = pd.DataFrame(nav_datas)

                filePathstd1 = os.path.join(os.getcwd(), r"D:\历史净值数据_new.csv")
                if os.path.exists(filePathstd1):
                    df.to_csv(filePathstd1, mode='a+', encoding='gbk', index=False, header=None)
                else:
                    df.to_csv(filePathstd1, encoding='gbk', index=False)
                print(" "+str(fund_id[i])+"基金的第"+str(j)+"页数据获取完成 ,索引为"+str(i)+" ")
                time.sleep(6)
                j += 1
            else:
                print("翻页结束")
                j = 1
                break
        time.sleep(2)