二、抓取网络上的资源信息
一、获取到网络上的网页
from bs4 import BeautifulSoup import requests url = 'https://www.tripadvisor.cn/Attractions-g60763-Activities-New_York_City_New_York.html' web_data = requests.get(url) soup = BeautifulSoup(web_data.text,'lxml') print(soup)
二、获取想要的数据
from bs4 import BeautifulSoup import requests url = 'https://www.tripadvisor.cn/Attractions-g60763-Activities-New_York_City_New_York.html' web_data = requests.get(url) soup = BeautifulSoup(web_data.text,'lxml') titles = soup.select('#taplc_attraction_coverpage_attraction_0 > div:nth-of-type(1) > div > div > div.shelf_item_container > div:nth-of-type(1) > div.poi > div > div.item.name > a') print(titles)
但不是所有的,使用下面的方式获取所有
from bs4 import BeautifulSoup import requests url = 'https://www.tripadvisor.cn/Attractions-g60763-Activities-New_York_City_New_York.html' web_data = requests.get(url) soup = BeautifulSoup(web_data.text,'lxml') titles = soup.select('div.item.name') imgs = soup.select('img[width="200"]') cates = soup.select('div.poi > div > div:nth-of-type(3)') #taplc_attraction_coverpage_attraction_0 > div:nth-child(1) > div > div > div.shelf_item_container > div:nth-child(4) > div.poi > div > div:nth-child(3) # print(titles,imgs,cates,sep='\n-----------\n') #验证下 # for title in titles: # print(title.get_text()) # for img in imgs: # print(img.get('src')) # for cate in cates: # print(cate.get_text()) for title,img,cate in zip(titles,imgs,cates): data={ 'title':title.get_text(), 'img':img.get('src'), 'cate':list(cate.stripped_strings), } print(data)
二、伪造登陆
三、爬取多个网页
四、应对js-爬取手机端
posted on 2017-10-28 21:04 Michael2397 阅读(...) 评论(...) 编辑 收藏