python爬虫 xpath使用问题整理

JSON_L

已于 2022-10-11 14:05:51 修改

阅读量2.9k

点赞数 1

分类专栏： Python 文章标签： python 开发语言

于 2022-10-11 14:04:47 首次发布

本文链接：https://blog.csdn.net/json_ligege/article/details/127261829

版权

Python 专栏收录该内容

59 篇文章 5 订阅

订阅专栏

1. module 'lxml' has no attribute 'html'

代码：

import requests
import lxml

web = requests.get('https://www.xxx.com/paihang.html', timeout=7)
selector = lxml.html.fromstring(web)
print(selector)

解决：

import lxml

改为

import lxml.html

2.ValueError: can only parse strings

代码：

web = requests.get('https://www.xxx.com/paihang.html', timeout=7)
selector = lxml.html.fromstring(web)
print(selector)

解决：

selector = lxml.html.fromstring(web)

改为

selector = lxml.html.fromstring(web.text)

3.AttributeError: 'NoneType' object has no attribute 'xpath'

代码：

import requests
from lxml import etree

web = requests.get('https://www.xxx.com/top250/')
web.encoding = 'utf-8'
selector = etree.HTML(web.text)

title = selector.xpath('//head/title/text()')
print(title)

解决：

缺少请求头
web = requests.get('https://www.xxx.com/top250/')

改为：

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0'}
web = requests.get('https://www.xxx.com/top250/', headers=headers)