python网络爬虫抓取金融分析师名单

最新推荐文章于 2022-05-11 19:41:15 发布

呆萌的代Ma

最新推荐文章于 2022-05-11 19:41:15 发布

阅读量307

点赞数

分类专栏： python 爬虫文章标签：网络爬虫数据分析 Python requests lxml

本文为CSDN博主"呆萌的代Ma"原创文章，转载请注明博客链接：https://blog.csdn.net/weixin_35757704/

本文链接：https://blog.csdn.net/weixin_35757704/article/details/121608341

版权

python 同时被 2 个专栏收录

593 篇文章 41 订阅

订阅专栏

爬虫

51 篇文章 4 订阅

订阅专栏

分析师名单网站：https://www.xiadun.net/rank/authorlist.php

代码

import requests
import random
from lxml.html import etree


def get_analyst():
    url = "https://www.xiadun.net/rank/authorlist.php"
    timeout = 60
    user_agent = {
        'User-Agent': random.choice(
            [
                "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",
                "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",
                "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0",
                "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; InfoPath.3; rv:11.0) like Gecko",
            ]
        ),
    }
    response = requests.get(url=url, headers=user_agent, timeout=timeout)
    return response


if __name__ == '__main__':
    write_loc = "../data/analyst-1.txt"

    response: requests.Response = get_analyst()
    html = etree.HTML(response.text)
    for user in html.cssselect(".contentMiddle a"):
        name = user.get("title")
        with open(write_loc, 'a') as file:
            file.write(name)
            file.write("\n")