本次练习网站为智联招聘网,要先注册登录。 传送门
看到Ajax:登录网站–>F12–>network–>XHR–>F5–>Preview
展开result,可以看到很多的招聘数据,这就是我们需要的。
现在开始写代码
一、创建scrapy项目。
scrapy startproject CrawPosition
二、使用pycharm来编辑代码
open项目之后可以看到scrapy框架已经帮我们创建好一些py文件
CrawPosition/
scrapy.cfg
CrawPosition/
__init__.py
items.py
pipelines.py
settings.py
spiders/
__init__.py
...
三、在spiders目录下创建一个 crawl.py爬虫文件 ,
-
编写爬虫文件
crawl.py ----------------------------------------------------------- from scrapy import Spider import time import scrapy from crawposition.items import CrawpositionItem import json class Craw(Spider): name='Craw' def start_requests(self): headers = { 'User-Agent': ' Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36', 'Connection': 'keep-alive', 'Cookie':'你的cookies' } for i in range(1, 2): #这里注意,不能使用中文,要改变的变量只有&kw。 url = 'https://fe-api.zhaopin.com/c/i/sou?start={}&pageSize=90&cityId=489&salary=0,0&workExperience=-1&education=-1&companyType=-1&kw=%E5%A4%A7%E6%95%B0%E6%8D%AE&kt=3&=0'.format(i) #只是单纯的学习,不想给其服务器造成负担。嘿嘿嘿~~ time.sleep(5) print(url) yield scrapy.Request(url, self.parse, headers=headers) def parse(self, response): html=response.text item=CrawpositionItem() if json.loads(html).get('data'): data = json.loads(html).get('data') results = data.get('results') for result in results: item['companyName'] = result['company']['name'] item['city'] = result['city']['display'] item['extractSkillTag']=result['extractSkillTag'] item['recruitCount']=result['recruitCount'] item['salary']=result['salary'] item['jobName']=result['jobName'] yield item
-
编写items.py文件
items.py ------------------------------------------------------ import scrapy class CrawpositionItem(scrapy.Item): # define the fields for your item here like: # name = scrapy.Field() companyName=scrapy.Field() city=scrapy.Field() extractSkillTag=scrapy.Field() recruitCount=scrapy.Field() salary=scrapy.Field() jobName=scrapy.Field()
-
编写pipelines.py文件
class CrawpositionPipeline(object):
def __init__(self):
self.file=open('ans.json','a',encoding='utf-8')
def process_item(self, item, spider):
lines=dict(item)
line={"name":lines['jobName'],"detail":{lines['companyName'],lines['city'],lines['recruitCount'],lines['salary']}}
print(line)
self.file.write(str(line))
self.file.write('\n')
return item
def spider_closed(self,spider):
self.file.close()
- 编写settings.py文件
BOT_NAME = 'CrawPosition'
SPIDER_MODULES = ['CrawPosition.spiders']
NEWSPIDER_MODULE = 'CrawPosition.spiders'
#这里的CrawpositionPipeline为pipelines.py文件的class类名。300是优先级
ITEM_PIPELINES={
'CrawPosition.pipelines.CrawpositionPipeline':300
}
LOG_LEVEL='INFO'
ROBOTSTXT_OBEY = True
运行项目
scrapy crawl Crawl