爬取链家网房源信息 线性回归做房价预测（一）

爬取链家网房源信息：

1 选择天津市的红桥区二手房信息 https://tj.lianjia.com/ershoufang/hongqiao/mw1dp1sf1。F12获取界面信息，查找所需房源信息对应的页面标签位置

2 按照30页的分页，爬取所有的房源信息，借助pandas写入excel表格中

# !/usr/bin/python
# -*- coding:utf-8 -*-

import warnings
import requests,sys,bs4,re
import pandas as pd
warnings.filterwarnings("ignore")
# 访问时加header,以免爬虫无法访问
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36'}

uinfo = []
# 链家的分页url格式不同 首页是/hongqiao/mw1dp1sf1  后面的页面从pg2开始
for i in range(1,31):
    if i == 1:
        url = 'https://tj.lianjia.com/ershoufang/hongqiao/mw1dp1sf1'
    else:
        url = 'https://tj.lianjia.com/ershoufang/hongqiao/pg'+str(i)+'mw1dp1sf1'

    res=requests.get(url=url, headers=headers)
    soup=bs4.BeautifulSoup(res.text,'lxml')
# F12查找到item标签下记录了每个房源的信息，样例如下：
# <div class="item" data-houseid="101104504632"> 
# <a class="img" data-bl="list" data-el="ershoufang" data-housecode="101104504632" data-is_focus="" data-log_index="1" href="https://tj.lianjia.com/ershoufang/101104504632.html" target="_blank"> 
# <img class="lj-lazy" data-original="https://image1.ljcdn.com/120000-inspection/prod-affe9ba8-f2b8-4845-88d2-aa7beeeb3346.jpg.296x216.jpg.437x300.jpg" src="https://s1.ljcdn.com/feroot/pc/asset/img/blank.gif?_v=20190625170329"/> 
# <div class="btn-follow follow" data-hid="101104504632"><span class="star"></span><span class="follow-text">关注</span></div><div class="leftArrow"><span></span></div><div class="rightArrow"><span></span></div> 
# <div class="price"><span>338</span>万</div></a> 
# <a class="title" data-bl="list" data-el="ershoufang" data-housecode="101104504632" data-is_focus="" data-log_index="1" href="https://tj.lianjia.com/ershoufang/101104504632.html" target="_blank"> 
# 此房满五年 后期税费低看房方便</a> 
# <div class="info">三条石<span>/</span>2室1厅<span>/</span>98平米<span>/</span>南<span>/</span>精装<span>/</span>有电梯</div> 
# <div class="tag"><span class="subway">近地铁</span><span class="good">必看好房</span><span class="vr">VR房源</span><span class="taxfree">房本满五年</span></div></div>

    linkTitle = soup.select('.item')
    print(len(linkTitle))
# 遍历每个分页的所有房源
    for m in range(0,len(linkTitle)-1):
# 判断该房源信息基本信息和房价信息是否完整
        if len(linkTitle[m].select('.info')) >0 and len(linkTitle[m].select('.price')) >0:
#将房源基本信息转化为字符串去除不需要的部分
            infoList =str(linkTitle[m].select('.info')[0]).replace('<div class="info">','').replace('<span>/</span>','^').replace('</div>','').split('^')
##'户型','平米','朝向','装修','电梯' 判断是否信息完整
            if len(infoList) == 6:
#获得房价信息
                intprice = (str(linkTitle[m].select('.price')[0]).replace('<div class="price">','').replace('<span>','').replace('</div>','').replace('</span>万',''))
#添加该信息数组
                uinfo.append([intprice,infoList[1],infoList[2],infoList[3],infoList[4],infoList[5]])
#将所有爬取的数据用pandas封装，写入excel
path = "lianjia-info.xlsx"
uinfo = pd.DataFrame(uinfo, columns=['房价', '户型','平米','朝向','装修','电梯'])
uinfo.to_excel(path, index=False)