计算机专业本科毕业论文爬虫,面向电子商务网站的专业网络爬虫设计与实现_计算机专业论文...

最新推荐文章于 2023-10-29 10:41:36 发布

猎奇症候群

最新推荐文章于 2023-10-29 10:41:36 发布

阅读量342

点赞数

文章标签：计算机专业本科毕业论文爬虫

本文探讨了一款面向电子商务网站的专业网络爬虫的实现，该爬虫通过分析和过滤网页，专注于获取与商品信息相关的内容。利用多线程技术，爬虫能有效地抓取和存储大量网页信息，帮助用户查找所需商品。设计过程中，考虑了URL筛选策略和网页解析，以提高爬取效率和数据质量。

摘要由CSDN通过智能技术生成

面向电子商务网站的专业网络爬虫设计与实现_计算机专业论文

发布时间：2015-01-22

来源：人大经济论坛

计算机专业论文范文

摘要

网络爬虫是一个自动下载网页的程序，是搜索引擎的重要组成。传统爬虫从一个或若干初始网页的URL开始，获得初始网页上的URL，在抓取网页的过程中，不断从当前页面上抽取新的URL放入队列，直到该URL对列为空为止。

本文设计的这款面向电子商务网站的专业网络爬虫，只对电子商务网站进行信息搜索，让用户可以尽可能多的找到自己关心的商品信息。面向电子商务网站的专业网络爬虫的工作流程十分复杂，需要根据一定的网页分析过滤与电子商务商品信息无关的链接，保留有用的链接并将其放入等待抓取的URL队列。然后，它将根据一定的搜索策略从队列中选择下一步要抓取的网页URL，并重复上述过程，直到达到保存URL的队列为空为止。另外，所有被爬虫抓取的网页将会被系统存贮。文章在分析网络爬虫的工作原理的基础上，结合多线程技术，设计了这个网络爬虫程序。

关键字：搜索引擎，网络爬虫，电子商务

The Topic-Specific Web Crawler of Oriented e-commerce website Design and Implementation

Abstract

Web Crawler is a procedure of automatically downloading website pages, it downloads website pages from the World Wide Web for search engine, and works as an important component of search engine. Traditional Web Crawler starts from one or several of the initial URL of a website, and get some new URLs from the website pages, in the process of continuously downloading website html pages, it finds some new URLs and determine which URLs will be added into a queue, it works until the URL Queue is empty.

The Web Crawler, which is designed by me, is to collect information on the e-commerce websites, so that users can find as much information as they concerned.

The Web Crawler which downloads e-commerce websites, has a very complicated workflow, and needs doing an analysis for the website and filter links which are unrelated to e-commerce website, then keeps the useful links and places them into the URL queue. Then, under certain searching strategy, it would choose the next URL from the queue to download the website page, and repeat this process until the URL queue is empty. In addition, all the pages are stored on the local driver.

Based on the analysis of the principle of the Web Crawler, and the multithreading technology, this article designs this Web Crawler procedure.

Key Words: Search engine, Web Crawler, E-commerce

摘要I

AbstractII

目录III

1 绪论4

1.1 课题背景及意义4

1.2 国内外研究现状2

1.3 爬虫程序在电子商务的应用3

1.4 本文所要完成的工作4

2 网络爬虫5

2.1 搜索引擎概述5