python 爬虫 | 检查网站情况

这里的笔记来源于对《用python写网络爬虫》的总结，写作以记录。
版本：python2.7

1、网站大小估计
在谷歌或百度中输入site:域名
例如

显示这个网站有1亿0720万个网页。

2、识别网站所用的技术
在爬去网站之前，了解网站使用的技术，会对爬去数据有一定的印象。这里使用builtwith模块来探测网上搭建的技术。

import builtwith
builtwith.parse("http://www.jianshu.com")

结果：
{u’javascript-frameworks’: [u’Prototype’, u’RequireJS’],
u’programming-languages’: [u’Ruby’],
u’web-frameworks’: [u’Twitter Bootstrap’, u’Ruby on Rails’]}

3、查看网站的拥有者
采用python-whois包，需要下载。

pip install python-whois

查看所有者

import whois
print(whois.whois("http://www.jianshu.com"))

结果：
{
“updated_date”: [
“2016-04-06 00:00:00”,
“2016-04-06 10:24:47”
],
“status”: [
“clientTransferProhibited https://icann.org/epp#clientTransferProhibited“,
“clientTransferProhibited https://www.icann.org/epp#clientTransferProhibited”
],
“name”: “Shanghai Bai Ji Information Technology Inc. Ltd,”,
“dnssec”: “unSigned”,
“city”: “Shanghai”,
“expiration_date”: [
“2020-03-20 00:00:00”,
“2020-03-20 18:28:58”
],
“zipcode”: “200433”,
“domain_name”: “JIANSHU.COM”,
“country”: “CN”,
“whois_server”: “whois.name.com”,
“state”: “Shanghai”,
“registrar”: “Name.com, Inc.”,
“referral_url”: “http://www.name.com“,
“address”: “Innospace 2, B1, Building #5, KIC, No.316 Songhu Road , Yangpu District”,
“name_servers”: [
“F1G1NS1.DNSPOD.NET”,
“F1G1NS2.DNSPOD.NET”,
“f1g1ns1.dnspod.net”,
“f1g1ns2.dnspod.net”
],
“org”: “Shanghai Bai Ji Information Technology Inc. Ltd,”,
“creation_date”: [
“2008-03-20 00:00:00”,
“2008-03-20 18:28:58”
],
“emails”: [
“contact@jianshu.com”,
“abuse@name.com”
]
}
In [ ]:

​