Python爬虫+可视化分析技术实现招聘网站岗位数据抓取与分析推荐系统_基于网络爬虫的求职大数据获取及分析系统

“”"
插入数据到数据库

create table jobpost(
j_salary float(3, 1),
j_locate text,
j_post text
);
“”"
conn = pymysql.connect(
host=“",
port=,
user=“root”,
paswd=”****",
db=“AAAA”,
charset=“utf8”,
)
cur = conn.cursor()
with open(os.path.join(“data”, “post_salary.csv”), “r”, encoding=“utf-8”) as f:
f_csv = csv.reader(f)
sql = “insert into jobpost(j_salary, j_locate, j_post) values(%s, %s, %s)”
for row in f_csv:
value = (row[0], row[1], row[2])
try:
cur.execute(sql, value)
conn.commit()
except Exception as e:
logger.error(e)
cur.close()

def run(self):
“”"
多线程爬取数据
“”"
self.job_spider()
self.execute_more_tasks(self.post_require)
self.desc_url_queue.join() # 主线程阻塞,等待队列清空

def execute_more_tasks(self, target):
“”"
协程池接收请求任务,可以扩展把解析,存储耗时操作加入各自队列,效率最大化

:param target: 任务函数
:param count: 启动线程数量
“”"
for i in range(POOL_MAXSIZE):
self.pool.apply_async(target)

if name == “main”:
spider = JobSpider()

start = time.time()
spider.run()
logger.info(“总耗时 {} 秒”.format(time.t