Web Scraper爬取谷歌学术

最新推荐文章于 2024-09-12 07:45:27 发布

从零开始的机器学习

最新推荐文章于 2024-09-12 07:45:27 发布

阅读量177

点赞数 2

文章标签：爬虫

本文链接：https://blog.csdn.net/weixin_45177899/article/details/138807615

版权

Web Scraper翻页+爬取标题和引用量，不废话直接给出sitemap：

{"_id":"googlescholar","startUrl":["https://scholar.google.com/scholar?hl=zh-CN&as_sdt=0%2C5&q=summarization&oq="],"selectors":[{"id":"title","multiple":false,"parentSelectors":["wrapper_for_name_cite"],"regex":"","selector":"h3","type":"SelectorText"},{"id":"next_page","linkType":"linkFromHref","multiple":true,"parentSelectors":["_root"],"selector":"td[align] a","type":"SelectorLink"},{"id":"cite","multiple":false,"parentSelectors":["wrapper_for_name_cite"],"regex":"被引用次数：[0-9]+","selector":"div.gs_flb","type":"SelectorText"},{"id":"wrapper_for_name_cite","multiple":true,"parentSelectors":["_root","next_page"],"selector":"div.gs_or","type":"SelectorElement"}]}

你只需把startUrl改成你要开始爬的网址就可以，例如我这里搜索的关键词是summarization。
最后导出的csv包含一列标题，一列引用量。