爬虫的步骤解析内容xpath介绍_爬虫-数据解析-xpath

提取数据

取文本:

/text():取直系的文本内容

//text():取所有的文本内容

取属性直:

tag/@attrName

# 需求:爬取boss的招聘信息

from lxml import etree

headers = { # 反扒策略

'User-Agent':',

'cookie':''

}

url = 'https://www.zhipin.com/job_detail/?query=python%E7%88%AC%E8%99%AB&city=101010100&industry=&position='

page_text = requests.get(url,headers=headers).text

#数据解析

tree = etree.HTML(page_text)

li_list = tree.xpath('//div[@class="job-list"]/ul/li')

for li in li_list:

# 需要将li表示的局部页面源码数据中的相关数据进行提取

# 如果xpath表达式被作用在了循环中,表达式要以./或者.//开头

detail_url = 'https://www.zhipin.com'+li.xpath('.//div[@class="info-primary"]/h3/a/@href')[0]

job_title = li.xpath('.//div[@class="info-primary"]/h3/a/div/text()')[0]

salary = li.xpath('.//div[@class="info-primary"]/h3/a/span/text()')[0]

company = li.xpath('.//div[@class="info-company"]/div/h3/a/text()')[0]

#对详情页的url发请求解析出岗位职责

detail_page_text = requests.get(detail_url,headers=headers).text

tree = etree.HTML(detail_page_text)

job_desc = tree.xpath('//div[@class="text"]//text()')

job_desc = ''.join(job_desc)

print(job_title,salary,company,job_desc)

乱码处理

对获取到的内容先编码再转码:

img_name = img_name.encode('iso-8859-1').decode('gbk')