正方系统的爬取

项目目的:

输入学号密码，模拟登录正方系统，并爬取学生个人信息和成绩和课程表保存至数据库

项目环境：

python版本:3.5;数据库:mysql;库包:requests、BeautifulSoup、pymysql、urllib、re、PIL；数据库可视化工具:Navicat；浏览器:Chrome；IDE:Pycharm

项目开发：

数据库设计：

SET FOREIGN_KEY_CHECKS=0;
-- ----------------------------
-- Table structure for course
-- ----------------------------
DROP TABLE IF EXISTS `course`;
CREATE TABLE `course` (
  `cid` int(11) NOT NULL AUTO_INCREMENT,
  `cname` varchar(15) DEFAULT NULL,
  PRIMARY KEY (`cid`),
  KEY `cname` (`cname`)
) ENGINE=InnoDB AUTO_INCREMENT=63 DEFAULT CHARSET=utf8mb4;
-- ----------------------------
-- Table structure for score
-- ----------------------------
DROP TABLE IF EXISTS `score`;
CREATE TABLE `score` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `sid` varchar(15) DEFAULT NULL,
  `cname` varchar(15) DEFAULT NULL,
  `garde` float(4,0) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `cid` (`cname`),
  KEY `sid` (`sid`),
  CONSTRAINT `score_ibfk_1` FOREIGN KEY (`sid`) REFERENCES `student` (`sid`),
  CONSTRAINT `score_ibfk_2` FOREIGN KEY (`cname`) REFERENCES `course` (`cname`)
) ENGINE=InnoDB AUTO_INCREMENT=132 DEFAULT CHARSET=utf8mb4;
-- ----------------------------
-- Table structure for student
-- ----------------------------
DROP TABLE IF EXISTS `student`;
CREATE TABLE `student` (
  `sid` varchar(15) NOT NULL,
  `name` varchar(15) DEFAULT NULL,
  `password` varchar(15) DEFAULT NULL,
  `sex` varchar(15) DEFAULT NULL,
  `bnumber` varchar(15) DEFAULT NULL,
  `birthday` varchar(15) DEFAULT NULL,
  `IdCord` varchar(15) DEFAULT NULL,
  `profession` varchar(15) DEFAULT NULL,
  `class` varchar(15) DEFAULT NULL,
  `Examinee number` varchar(15) DEFAULT NULL,
  PRIMARY KEY (`sid`)

) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

首先用浏览器访问正方系统的网址http://210.38.162.116/(wk5amj45g5b5dh55roqgh445)/default2.aspx，同时打开开发者工具，输入一次登录，观察Network捕获到的default2.ajpx

根据Headers构造相应的表头和数据data

headers = {
    "Accept-Encoding": "gzip, deflate",
    "Accept-Language": "zh-CN,zh;q=0.9",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36',
    "Host": "210.38.162.116",
    "Upgrade-Insecure-Requests": "1",
    "Cache-Control": "max-age=0"
}

data = {
 

        'TextBox1': username,
        'TextBox2' : password,
        'TextBox3' : ycode,
        '__VIEWSTATE' : __VIEWSTATE,
        'RadioButtonList1' : '%D1%A7%C9%FA',
        'Button1' : '',
        'lbLanguage' : '',
 }

RadioButtonList1的值为学生的gb2312解码

用request.get先请求一次网页的源码，发现其中有

<input type="hidden" name="__VIEWSTATE" value="dDwtMTg3MTM5OTI5MTs7PjJ06Q8x0sjwFTIugwoEgtaDQze7">

用soup解析并将相应的值填充至data

观察Network发现里面有Checkcode.aspx，这里使用第三库将验证码下载至本地在弹出

imgresponse = s.get(imgUrl, stream=True)
image = imgresponse.content
DstDir = os.getcwd()+"\\"
print("保存验证码到："+DstDir+"code.jpg"+"\n")
try:
    with open(DstDir+"code.jpg" ,"wb") as jpg:
        jpg.write(image)
except IOError:
        print("IO Error\n")
finally:
        jpg.close

数据全部封装好后，首先开启一个Session,并获得相应的cookies，然后为header添加cookies，利用session提交post请求模拟登录

response = s.post(url,headers=headers,data=data)

后面的操作就是模仿前面的样子，观察Network捕获的aspx，构造相应的表头和data，然后用session去post或者get

def xxLoading(kburl,headers_code):
    db = pymysql.connect(host="localhost", user="root", passwd="123456", db="pystu", charset='utf8')
    cursor = db.cursor()
    response = s.get(kburl, headers=headers_code)
    html = response.content.decode('gb2312')
    soup = bs4.BeautifulSoup(html, 'lxml')
    id = soup.find('span', attrs={
 'id': 'xh'}).string
    name = soup.find('span', attrs={
 'id': 'xm'}).string
    sex = soup.find('span', attrs={
 'id': 'lbl_xb'}).string
    phone = soup.find('span', attrs={
 'id': <