项目目的:
输入学号密码,模拟登录正方系统,并爬取学生个人信息和成绩和课程表保存至数据库
项目环境:
项目开发:
SET FOREIGN_KEY_CHECKS=0;
-- ----------------------------
-- Table structure for course
-- ----------------------------
DROP TABLE IF EXISTS `course`;
CREATE TABLE `course` (
`cid` int(11) NOT NULL AUTO_INCREMENT,
`cname` varchar(15) DEFAULT NULL,
PRIMARY KEY (`cid`),
KEY `cname` (`cname`)
) ENGINE=InnoDB AUTO_INCREMENT=63 DEFAULT CHARSET=utf8mb4;
-- ----------------------------
-- Table structure for score
-- ----------------------------
DROP TABLE IF EXISTS `score`;
CREATE TABLE `score` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`sid` varchar(15) DEFAULT NULL,
`cname` varchar(15) DEFAULT NULL,
`garde` float(4,0) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `cid` (`cname`),
KEY `sid` (`sid`),
CONSTRAINT `score_ibfk_1` FOREIGN KEY (`sid`) REFERENCES `student` (`sid`),
CONSTRAINT `score_ibfk_2` FOREIGN KEY (`cname`) REFERENCES `course` (`cname`)
) ENGINE=InnoDB AUTO_INCREMENT=132 DEFAULT CHARSET=utf8mb4;
-- ----------------------------
-- Table structure for student
-- ----------------------------
DROP TABLE IF EXISTS `student`;
CREATE TABLE `student` (
`sid` varchar(15) NOT NULL,
`name` varchar(15) DEFAULT NULL,
`password` varchar(15) DEFAULT NULL,
`sex` varchar(15) DEFAULT NULL,
`bnumber` varchar(15) DEFAULT NULL,
`birthday` varchar(15) DEFAULT NULL,
`IdCord` varchar(15) DEFAULT NULL,
`profession` varchar(15) DEFAULT NULL,
`class` varchar(15) DEFAULT NULL,
`Examinee number` varchar(15) DEFAULT NULL,
PRIMARY KEY (`sid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
headers = { "Accept-Encoding": "gzip, deflate", "Accept-Language": "zh-CN,zh;q=0.9", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8", 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36', "Host": "210.38.162.116", "Upgrade-Insecure-Requests": "1", "Cache-Control": "max-age=0" }
data = {
'TextBox1': username, 'TextBox2' : password, 'TextBox3' : ycode, '__VIEWSTATE' : __VIEWSTATE, 'RadioButtonList1' : '%D1%A7%C9%FA', 'Button1' : '', 'lbLanguage' : '', }
RadioButtonList1的值为学生的gb2312解码
用request.get先请求一次网页的源码,发现其中有
<input type="hidden" name="__VIEWSTATE" value="dDwtMTg3MTM5OTI5MTs7PjJ06Q8x0sjwFTIugwoEgtaDQze7">
用soup解析并将相应的值填充至data
观察Network发现里面有Checkcode.aspx,这里使用第三库将验证码下载至本地在弹出
imgresponse = s.get(imgUrl, stream=True) image = imgresponse.content DstDir = os.getcwd()+"\\" print("保存验证码到:"+DstDir+"code.jpg"+"\n") try: with open(DstDir+"code.jpg" ,"wb") as jpg: jpg.write(image) except IOError: print("IO Error\n") finally: jpg.close
数据全部封装好后,首先开启一个Session,并获得相应的cookies,然后为header添加cookies,利用session提交post请求模拟登录
response = s.post(url,headers=headers,data=data)
后面的操作就是模仿前面的样子,观察Network捕获的aspx,构造相应的表头和data,然后用session去post或者get
def xxLoading(kburl,headers_code): db = pymysql.connect(host="localhost", user="root", passwd="123456", db="pystu", charset='utf8') cursor = db.cursor() response = s.get(kburl, headers=headers_code) html = response.content.decode('gb2312') soup = bs4.BeautifulSoup(html, 'lxml') id = soup.find('span', attrs={ 'id': 'xh'}).string name = soup.find('span', attrs={ 'id': 'xm'}).string sex = soup.find('span', attrs={ 'id': 'lbl_xb'}).string phone = soup.find('span', attrs={ 'id': <