爬取过程分析:
1、链家网url:https://bj.lianjia.com/ershoufang/pg2/
显示的二手房信息每个url递增一
2、访问时要加头信息,否则无法访问
3、用beautifulsoup解析网页,找到二手房信息
4、将信息简单处理后保存成csv文件
具体代码如下:
import requests
import re
from bs4 import BeautifulSoup
i=0
with open('./lianjia/lianjia.csv','a') as file:
for j in range(1,100,1):
i+=1
url = 'https://bj.lianjia.com/ershoufang/pg+str(i)'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36'}
resp = requests.get(url,headers = headers).text
# print(resp)
soup = BeautifulSoup(resp,'lxml')
infos = soup.find('ul',{'class':'sellListContent'}).find_all('li')
# print(infos)
for info in infos:
name = info.find('div',{'class':'title'}).find('a').get_text()
price = info.find('div',{'class':'totalPrice'}).find('span').get_text()
address = info.find('div',{'class':'address'}).get_text()
print(address)
area = re.split(r'\|',address)[2]
print(area)
file.write('{},{},{},{}\n'.format((name),(price),(address),area))