python爬虫进阶（三）：微博的抓取

说明：仅做学习之用

下面用常用的两种方法来爬取微博：使用selenium+phantomjs和API解析

一、使用selenium+phantomjs

最重要的是设置user_agent，否则无法跳转链接

当然，还可以设置其它更多参数

from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities


user_agent = (
    'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36 QIHU 360SE'
)

dcap = dict(DesiredCapabilities.PHANTOMJS)
dcap['phantomjs.page.settings.userAgent'] = user_agent

driver = webdriver.PhantomJS(desired_capabilities=dcap)

在登录时需要输入用户名与密码，然后点击登录按钮，还好这里不需要输入验证码

一般的，我们现在浏览器console控制台找到我们想要的东西

document.getElementById('loginname').value = '123'

document.getElementsByName('password')[0].value = '123'

下面是两种登录方式，但我在登录过程中遇到了验证码问题，后来在输入用户名和密码后各自休息了几秒钟，居然登录成功了，难道是程序输入太快而被阻止了吗？？？

#driver = webdriver.PhantomJS(desired_capabilities=dcap)
driver = webdriver.Chrome()
#driver.set_window_size(1280,2400)
driver.get('https://www.weibo.com/')

time.sleep(10)

driver.find_element_by_id('loginname').send_keys(username)
time.sleep(5)
driver.find_element_by_name('password').send_keys(password)
time.sleep(2)
driver.find_element_by_xpath('//div[contains(@class,"login_btn")][1]/a').click()

# wait = WebDriverWait(driver, 10)
# u_id = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR,'#loginname')))
# p_word = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR,'#pl_login_form > div > div:nth-child(3) > div.info_list.password > div > input')))
# login = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,'#pl_login_form > div > div:nth-child(3) > div.info_list.login_btn > a')))
# u_id.send_keys(username)
# time.sleep(5)
# p_word.send_keys(password)
# time.sleep(2)
# login.click()

time.sleep(15)

html = driver.page_source

print(html)