使用（python&selenium）爬取资料

问题：怎么样模仿人工点击浏览器获取资料

网址：http://www.kaikaidai.com/Lend/Black.aspx

• 获取每页的10个人的信息（每个人的信息包含8项），总376个名单，每页10个，共38页。
  
• 将数据保存在txt，一行包含一个人的八条信息，中间以分号”;”分隔，保存格式为如下：

冯介伦；370724197209050338；山东省潍坊市临朐县冶源镇车家沟村107号；临朐县腾达天然大理石加工厂；临朐县冶源镇红光村南(无门牌号）；lqfengjielun@163.com；05363332678；150656810891

实现方法

使用爬虫利器selenium获取页面每个元素的xpath内容（XPath是路径表达式，可以定位HTML中的元素，并获取元素的内容），最后模拟在页号框输入页码，模拟点击回车跳转指定页面。
获取元素xpath的方法（谷歌浏览器）：
页面选中元素—-右击—-检查—-右击—copy

还有一点需要注意：在获取一页10人信息时，每个人xpath的路径会有联系，需要自己总结。

代码实现

import unittest
import time
import re
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
url = "http://www.kaikaidai.com/Lend/Black.aspx"
path = "D:\Drivers\chromedriver.exe"
driver = webdriver.Chrome(executable_path=path)


#打开请求的url
driver.get(url)
f = open("res.txt",'w')
a = 1
m = 1
x = 2
y = 12
while m <=38:
    elem = driver.find_element_by_name("rpMessage")
    elem.send_keys(m)
    elem.send_keys(Keys.RETURN)#模拟点击回车

    if m == 38:
        x=2
        y=7
    for n in range(x,y):
        name_xpath = '//*[@id="form1"]/div[3]/div/div[2]/div[4]/table[' + str(n) + ']/tbody/tr[1]/td[3]/a'
        name = driver.find_element_by_xpath(name_xpath).text

        id_xpath = '//*[@id="form1"]/div[3]/div/div[2]/div[4]/table[' + str(n) + ']/tbody/tr[2]/td[2]'
        id = driver.find_element_by_xpath(id_xpath).text

        adress_xpath = '//*[@id="form1"]/div[3]/div/div[2]/div[4]/table[' + str(n) + ']/tbody/tr[3]/td[2]'
        adress = driver.find_element_by_xpath(adress_xpath).text

        company_name_xpath = '//*[@id="form1"]/div[3]/div/div[2]/div[4]/table[' + str(n) + ']/tbody/tr[4]/td[2]'
        company_name = driver.find_element_by_xpath(company_name_xpath).text

        company_adress_xpath = '//*[@id="form1"]/div[3]/div/div[2]/div[4]/table[' + str(n) + ']/tbody/tr[5]/td[3]'
        company_adress = driver.find_element_by_xpath(company_adress_xpath).text

        email_xpath = '//*[@id="form1"]/div[3]/div/div[2]/div[4]/table[' + str(n) + ']/tbody/tr[1]/td[5]'
        email= driver.find_element_by_xpath(email_xpath).text

        call_xpath = '//*[@id="form1"]/div[3]/div/div[2]/div[4]/table[' + str(n) + ']/tbody/tr[2]/td[4]'
        call = driver.find_element_by_xpath(call_xpath).text

        phone_xpath = '//*[@id="form1"]/div[3]/div/div[2]/div[4]/table[' + str(n) + ']/tbody/tr[3]/td[4]'
        phone = driver.find_element_by_xpath(phone_xpath ).text
        try:
            save = name  + ";" + id + ";" + adress + ";" + company_name + ";" + company_adress + ";" + email + ";" + call + ";" + phone + "\n"
            f.write(save)
            a = a + 1
        except:
            print (name,id,company_name,email,call,phone)
            save = name  + ";" + id + ";" + "**" + ";" + company_name + ";" + "**" + ";" + email + ";" + call + ";" + phone + "\n"
            f.write(save)
            a = a + 1

    m = m + 1

driver.close()

数据爬取结果