Python爬虫笔记

• 1: Pycharm无法引入bs4库
  

• ① 点击Preference —— ② 选择Project Reference ——
• ③ 点击左下角 + 标号 —— ④ 搜索BeautifulSoup4 —— ⑤点击安装

• 2.BeautifulSoup的理解
  

   # 选取中间部分div_listbox
    div_listbox = soup.find_all(name='div', attrs={"class": "text fl"})
    div_listbox_str = str(div_listbox)
    div_listbox_soup = BeautifulSoup(div_listbox_str, "html.parser")
  

• 3.标签树的下行遍历
  

• 4.做好一锅粥

   # 做好一锅汤soup
    r = requests.get(index, headers=kv)
    r.encoding = r.apparent_encoding
    demo = r.text
    soup = BeautifulSoup(demo, "html.parser")
    soup.prettify()
  

• 5.查看某tag标签
  
  
  
  

• 6.find_all()
  
  
  

• 6.获取所有的a标签
  

• 7.爬取JS加载的数据

• https://blog.csdn.net/qq_26200629/article/details/86141131 # 安装selinium库后，安装chrome_driver

• 使用driver

   src = list((index+"{}.html".format(i) for i in range(1,5)))
    #创建模拟浏览器浏览的driver
    driver = webdriver.Chrome(executable_path=driver_path)
    time.sleep(3)
    driver.maximize_window()    #最大化浏览器
    driver.get(index)
    data = driver.page_source
    soup = BeautifulSoup(data,'lxml')