在网上看到下面这个问题:
某个url的重要参数都在“#”后面,网络请求的时候“#”后面的参数会被忽略,这种情况该怎么处理?
在scrapy爬虫框架中会自动过滤掉#后面的内容,这时我们我需要将#进行字符转化,将#转化为%23进行处理.
问题如下:当不改变#为 %23时,输出结果入下截图
class netcoreSpider(Spider):
name = "netcore"
timeout = 20
trytimes = 3
start_urls = ["http://www.netcoretec.com/companyfile/2/"]
# must be lower character
typefilter = ["txt", "pdf"]
allsuffix = Set()
def parse(self, response):
for i in range(1,14): #13+1
url_list = "http://www.netcoretec.com/companyfile/2/#c_companyFile_list-15086754191809347-%s" % i
print url_list
request = scrapy.Request(url_list, callback=self.parse_list)
request.meta["prototype"] = MI.FirmcrawlerItem(