pycharm中贝壳网租房数据采集

最新推荐文章于 2024-04-24 22:50:08 发布

qaanlova

最新推荐文章于 2024-04-24 22:50:08 发布

阅读量757

点赞数

文章标签： pycharm python

本文链接：https://blog.csdn.net/qaanlova/article/details/133840413

版权

（一)获取贝壳网页网址

1.新建python文件

2.获取贝壳网租房的网页地址

https://cd.zu.ke.com/zufang/pg

3.复制网页的cURL地址。

4.访问解析网页内容网址。

5.获取解析后的内容。

6.复制解析的内容。
执行代码

cookies = { 'BAIDUID_BFESS': 'FF2250A43CDC71162CE1E25B094F4F9E:FG=1', 'ab_jid': '31fcb631c26627f95fb562fe7e61767bc799', 'ab_jid_BFESS': '31fcb631c26627f95fb562fe7e61767bc799', 'ZFY': 'XW2eyU09Pfaot6:B7ERmRKMtvl9HOPYdBR6:AJ7LVaXA8:C', 'ab_bid': '4fcb5bffc192f94efc9817fb10156133652f', 'ab_sr': '1.0.1_Y2IyNjBiOTBjNjUyZjhiYjFmNWI0Njc5NGU4NzI4OGM4YzA2MWRlN2VkMTIxZjczYmUwOTJjNDgzZDAxMDZmOWQzNmExOTY1NjQ1ZGVmYjFiMjlmMDBkNmY4YjkwMGYyOWVlZTU3ZGU5Y2ZkY2Y2YWY5Y2ZjZDY0NWFiNGVlZjEzNGM2OGNhM2FhZDlmMDE3MGZhZjljYTZmZTdiMzEzMQ==', } headers = { 'Accept': '*/*', 'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6', 'Connection': 'keep-alive', 'Content-Type': 'text/plain;charset=UTF-8', # 'Cookie': 'BAIDUID_BFESS=FF2250A43CDC71162CE1E25B094F4F9E:FG=1; ab_jid=31fcb631c26627f95fb562fe7e61767bc799; ab_jid_BFESS=31fcb631c26627f95fb562fe7e61767bc799; ZFY=XW2eyU09Pfaot6:B7ERmRKMtvl9HOPYdBR6:AJ7LVaXA8:C; ab_bid=4fcb5bffc192f94efc9817fb10156133652f; ab_sr=1.0.1_Y2IyNjBiOTBjNjUyZjhiYjFmNWI0Njc5NGU4NzI4OGM4YzA2MWRlN2VkMTIxZjczYmUwOTJjNDgzZDAxMDZmOWQzNmExOTY1NjQ1ZGVmYjFiMjlmMDBkNmY4YjkwMGYyOWVlZTU3ZGU5Y2ZkY2Y2YWY5Y2ZjZDY0NWFiNGVlZjEzNGM2OGNhM2FhZDlmMDE3MGZhZjljYTZmZTdiMzEzMQ==', 'Origin': 'https://cd.zu.ke.com', 'Referer': 'https://cd.zu.ke.com/', 'Sec-Fetch-Dest': 'empty', 'Sec-Fetch-Mode': 'cors', 'Sec-Fetch-Site': 'cross-site', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36 Edg/117.0.2045.60', 'sec-ch-ua': '"Microsoft Edge";v="117", "Not;A=Brand";v="8", "Chromium";v="117"', 'sec-ch-ua-mobile': '?0', 'sec-ch-ua-platform': '"Windows"', } data = { 'eyJkYXRhIjoiNGU2OTViMTViNzA0NjI2ODhlMzk2MmQzNzg0NGFlNGI1MWYyMjY3NzFiZDcxYmQ3MzVlYmUxOTBmYmE0ZTU3ODNhNzcyNzg1Y2Q2OTQ2NjM0M2U0MzhhMGYwMWYzNjVmN2ZjMmJhZmQzY2IxNzEyOTY1ZTcyY2E3ZDdmZGNhMDZkOTFlZjRmZjI2NzYzMGU4ZjUzYjE2NTBjOWYwMDA3NWUwODJmMmI4YzhmOWRmOWVjODI0OTJkMmFiOTYyZWMwOTFhMzZiYThlZTg5NTQ4YTM0ODY2NTgyMThiM2Y4NjMxMjMzYmFiMDMxNzEwYjNjZTZlMmJlNDgwMTljNjVjOGRjNGEyYWFhODkwOGRlOTdiMDYzNzk3MmFiYzc3Njg5ODdkYmY0NjIwZDczNzUyYWIxMmQxNDIyOWEwYTMxM2NmZTc3NTdlM2NhNDExNmQ5Y2M4Yjk4Zjk3MDk3OWNlZjg0MTY2ZDc2OTkzOTdhN2RmNGYxOTExYWE1MzNjZmEwNmY2YzJjNDkxZGUzYTE1NjgxMzMzMjRhYWQ3NTViODFlMzk4MDVmN2FiYTJlYWQ0ZGY0ZmM0ODdjODU1MjFjOTk5ZTRjYmI5Y2Y4OTMyOWU0NmIwNTJjMDU5OGNlZjViYjhiM2EwYTE1ODNkNWE0ZWNmNTU4NGQ4ZDlhOTUwMDAwNDQzNmNiMzg5YmEyZDc0MWYxNmM1ZGZiNWM4NWVmZDY3OWVjYjFmNmNmMWI5ZWQ2MzAxYWVjNzViN2Y2OTAwNDcyNTQwOWZmNTQzYjk5MzI1ODM2ODAwODA3YmE5YTljYmNjZmFjMDQyZWQxMjQzZGM5NGM1MjgwM2VkNzEzMTM5YWZlZGQ0MDhiYWY3ZjhkZjc1OWFlMTk3NzY5MzYzMGRlOWYwN2UzOTZiOWE0NWVjOWM0NjI5OTUzYjI2YzJjNDU5NGQxN2ZkODFjZTAyM2I2OTBiZTkxZjIyMmIzMGU3ZDljOGUzMDlmOGVkMzMxZjE0NDllN2FjZDE3ZjJjNDI5MDk5ZTcxYjQ4YzNkNTc2Njk2NTNkNjk0MGFkODNlZTllNDgwZDdhNzMzODg0NTBkOGQxODdmMzJjNGEwYmM2MGM0NGNiYzlhZGUxZGM1OWFmODlkNjhlYWU3MzgxMTI0MjAwYjA4ZWE3OTc0MmI2MTA4MGFmZWMzNDYzM2NkMzhjN2FjMDZiMWYxZjUyOThiZTA0YWQxZjJhMDJjOWQ4MjMxMGY1N2Q5YTk3OWFhZjg1Yzg2YjBlOGJkMzc0NzllMzZiNDVmYTI1YjE0NTNiYjEyMGNhN2FhZjIwMTZjYWVlOGNkYTg3NDRhMWQwZmQ0MWExZjIzZDg3YmE0YzM3N2RiMThjZmZhZGY4YzYzZmVkZjljYjA1Y2RkZjg5MmVjZjE1YzBkNDhkNmZjMjViMWNhYzAxZWY0MjQ2ZDQ5ZmFlZDlkZmI3ZTA3MGQ0ZTE3MmZmZjBmNjgzZjAxOWYwNWQxNTMxMTBlZGM4Y2Q5ZTMxYmE5OTEyODFjYTMwNzY2OTc5MzAwYTFjMTNhYzNjNmQwNDA3ODFjMjJkYmZlNTgzZWFmZGI0YmUxODMxODM1MThiNjk4NjVmZDE0ZmFiNTEwZWEyODIxM2FjNGU2YzA0YzBlMTkzOWY1YTg5ZDE0ODY0NjM1OWQ0NTExYWUyZjIyYjlkZjE5NDU4Y2Q0NTI2YjkxYTgzMjllYzY3MzcwZjI5OTU4MjgyY2U5ZDY1NjUyZGQzYTdlOTZhM2FiMzhlMjVkZjYwYzYzMTRjNWE4ZDI1MjkxZTIzNTIzNTM2YTQwYjEwZGNmOTNkYjJhYjY5NWEzNGIxYzQyM2ZiMTIwMWUyY2Y3M2MxMTUyMWY1OWY1YWMwYjZkNGFkODgxZDEyZWQ4MDY3ZDBjMzJlYmFiYmEzMjc0ODdkNDliMjVlMGViY2UyYTA4ODE5MzI0MTMzOTQ0ZjA4OWZmM2ZmNjgzNzI4NGUxM2M3ZTIwMmJkNGQ0OTFlMDNjZjc5NmIzZGU2ZmIwNGExYmZmZjdkODljOTMxYTY4NDgwY2IxY2ExOWEwYzg3OGExZDFiOWI4MjFmMzExNWQ3YzIwNDc3YTUxMWU0Zjk0YzlmYmMwMzMzYzUyNmMxOTRmMWNmMGJjMDExNTk5YjcxM2U2OGY4NGQ4NzU5ZDRjNmFjNzU2M2YxMGU5MTdmYTUwZDhjOTIyZGZlY2Y3ZTNjMTIxM2NjYWVkYTE1YmViM2U4ZTAzMThkNTYzZTg2OWE0N2Y5NmI3NTgwZTc1MWM1YjQ3YzI3YTNhNmRhZGU4N2E2ZTA2NTBiNDM0M2M1MGJmZjU0ZDM2ZjExMzJmNDlkNzU4MWMxZWIxZGRlNjhmNTI4OWRhNGUzYmUxYzM4NzdjODA0NzAyMmMxYWM4ODM1NjMzNTgyZTQ2YjdlZjQyNjE0ZTA5M2QzODg2MTU2NzM3NzJjZDQzMTdiZDAwZGE4Mjk1OGZlOTAzZWU2MWNmYzM0ZWIxNzFlNzA4YzUxMzQ3NjA1ODAwOGVlMzEyMjI1NDQ3NDc4Njk5ODExNjRhMzIxNDM4NTNlMzUxYzY1YzIzZDNjZTFjMzNiMmI2ZjJlOGQyZGZhNDQxZDM2MTEzMmJjZWNmNThhOWViMzFhMWJhOWYyZjQ2Y2NlMDg3YjI2M2ZkZmFiN2FhN2RlNzg1NTExNTE1YjgwMGEwOTUzN2JhMzI3ZGJkNTdhZjc5NjRlNjI5NTkxOWE2OTNlOTZhNDFkMzNmMjQyMjc4NzQzODAxZGI5MTFhYmUwNWZkNjQ3Y2NlYzU2MTQ0YjQ3OTYwMzg2YTcyNGRmZDc0ZWVjZmM3Yzg4NjI5YTllNWMwNmM1MDgyNDZlNGVkYzIwNzkyNGYzODc4Y2FlODY4OTg0OTAzZjU5NDFjYjU5NjQ4MjY3MThjMTA0M2QzNjYzNDhkOGM4MGJjZTg5ZjBlZjRhOWZmMGJlMjI4ZWI2MzQ4ZGQwOGRiZWNkYmNhMTliNjUwNzRhYzJkYjIzOTNmOWRiOGY5Mjg2Y2EyMzQxZGVmNjUwY2Y4ZWUyN2FkYjQ0YzA3MzQ0Y2U4MTQ0MjU2NTZjNDkwMDE4YzMwODBhMjk3YjVlYjIwMGVlYjNlYjY1YTMzYTllYzMwYjBmZDJmZWJmOGRkZWM4NzFjYzRjYWEzNjQzNjVhYjMxOWJhZjYzNzY5YmJhMTYzYzA5ODIzOWM4ZTcwOWQ0MTJhNTgyNzViYTI2YjAwM2U1NjAxZDA0MGM5NWFjMDE1ZjljMmI3YzUwMDQyODVlMTUwOWE1NmQyYWIzODYxNjVmMTJhOTI4ZmRkMTI0MWE3YThkMDg3ODk3MTlkMWRjN2I0MDVkMGJiNDk2NDlmMzQyNDliMjU1OWJhODJiYWU0Mzc1MTAzMTRlZDc3MTMxOGFhNjk1ZTRjYTU1ODQzNWJmNDU2MDJhYmZhM2E5N2M5YTg3MGU1ZGQyMjI2NWU1NTY1NWVlYzYzNzE1MzgxMDA5NTE0MGIwMTRmOWRkODMwNDU3ZTBkMzU0MmIyZGQ5MjhlNTVjNjJmNDNjMjBlN2JiZjZlODNhNjgwOTBkYzE1ZmQwMzIzZGNkODA1NzYyMWUzNDdjNGZkZmRiN2U4MGY4ODFmYTRiN2EzNzNlNzU2OGQ5NmE1ZTE3Njc5M2ZiYzhkZDYwYmM2YmYwN2JhMDM3MDM3Y2M2NzdjNjYxYTMxOGJiMzJlNTg1NjE4ZDg3NmE4MTE0NmQ0M2ExNzkyYjdiNzkzZTM5ZmJiNWI3NzlhZDcyZjlhOGY1N2Y0MGYyNzMzOTExMTZkYzI5OTVmODE4OWJhNmZmM2EyMDNmYzIzNzQ4MDBlNjU1YmVkZTQyOWMiLCJrZXlfaWQiOiIyZDk2M2E0NmUzZWQ0NjQ5In0': '', } response = requests.post('https://miao.baidu.com/abdr', cookies=cookies, headers=headers, data=data)

（二）解析网页的内容

1.添加贝壳网租房网页地址。

执行代码
1 #贝壳地址
2 beike_address='https://cd.zu.ke.com/zufang/pg'

2.拼接地址，实现(1-6)翻页。
执行代码
1 #拼接地址，实现翻页
2 for i in range(1,6) :
3 beike_url=beike_address+str(i)
4 print(beike_url)

3.贝壳租房地址打印成功。

4.复制网址进行验证。

5.使用POST获取请求内容。
执行代码

1 #使用POST获取请求头
2 response=requests.post(beike_url,cookies=cookies,headers=headers,data=data)

6.打印状态码。
执行代码

1 #打印请求状态码
2 print(response.status_code)

7.状态码打印成功。

8.查看网页内容。
执行代码

1 #查看网页内容
2 print(response.text)

9.网页内容。

（三）贝壳租房信息下载

1.使用etree解析网页。
执行代码

1 #使用etree解析网页
2 selector=etree.HTML(response.text)

2.定位租房信息。

3.复制租房信息的xpath路径。

4.使用xpath解析。
执行代码

1 all_zufang=selector.xpath('')

5.复制xpath路径。
执行代码

1 #小区xpath地址//*[@id="content"]/div[1]/div[1]/div[1]/div

2 allzufang=selector.xpath('//*[@id="content"]/div[1]/div[1]/div[1]/div')

6.完善xpath路径。
执行代码

1 #小区xpath地址//*[@id="content"]/div[1]/div[1]/div[1]/div
2 allzufang=selector.xpath('//*[@id="content"]/div[1]/div[1]/div/div')

7.使用for循环遍历地址内容。
执行代码

1 #使用for循环遍历地址内容
2 forxiaoquinall_zufang:

8.获取小区名称，并打印小区名称。
执行代码

1 #小区名称//*[@id="content"]/div[1]/div[1]/div[1]/div/p[1]/a
2 xiaoqu_name=xiaoqu.xpath('p[1]/a/text()')
3 print(xiaoqu_name)

9.小区名称结果。

10.获取小区地址的xpath路径。

11.获取小区地址，并打印小区地址。
执行代码

1 #小区地址//*[@id="content"]/div[1]/div[1]/div[1]/div/p[2]/a[1]
2 xiaoqu_address=xiaoqu.xpath('p[2]/a/text()')
3 print(xiaoqu_address)

12.返回小区地址结果

13.获取小区价格的xpath路径。

14.获取小区价格，并打印租房价格。
执行结果
1 #小区价格//*[@id="content"]/div[1]/div[1]/div[1]/div/span
2 xiaoqu_price=xiaoqu.xpath('span/em/text()')
3 print(xiaoqu_price)