PHP同步爬虫

最新推荐文章于 2023-02-01 16:56:43 发布

huizhang.

最新推荐文章于 2023-02-01 16:56:43 发布

阅读量516

点赞数

分类专栏： php Swoole 文章标签：同步爬虫 php Swoole

本文链接：https://blog.csdn.net/qq_34908844/article/details/79156657

版权

php 同时被 2 个专栏收录

132 篇文章 0 订阅

订阅专栏

Swoole

22 篇文章 1 订阅

订阅专栏

主文件

<?php
/**
 * 同步爬虫
 */
require_once 'Crawler.php';

$start = microtime(true);//unix微妙数

$url = 'http://www.swoole.com/';
$ins = new Crawler($url);
$ins->visitOneDegree();

$timeUsed = microtime(true) - $start;
echo "time used: " . $timeUsed;

爬虫类

<?php

/**
 * Class Crawler
 * Path: /Sync/Crawler.php
 */
class Crawler
{
    private $url;
    private $toVisit = [];
    public function __construct($url)
    {
        $this->url = $url;
    }

    public function visitOneDegree()
    {
        $this->loadPageUrls();
        $this->visitAll();
    }

    private function loadPageUrls()
    {
        $content = $this->visit($this->url);
        $pattern = '#((http|ftp)://(\S*?\.\S*?))([\s)\[\]{},;"\':<]|\.\s|$)#i';
        preg_match_all($pattern, $content, $matched);//正则匹配
        foreach ($matched[0] as $url) {
            if (in_array($url, $this->toVisit)) {
                continue;
            }
            echo $url;echo "<br/>";
            $this->toVisit[] = $url;
        }
    }

    private function visitAll()
    {
        foreach ($this->toVisit as $url) {
            $this->visit($url);
        }
    }

    private function visit($url)
    {
        return @file_get_contents($url);//读取地址内容
    }
}

爬取结果

http://wiki.swoole.com/"
http://compiler.swoole.com/"
http://wiki.swoole.com/wiki/page/prid-1-p-project/change_log.html"
http://group.swoole.com"
http://wiki.swoole.com/wiki/page/prid-1-p-author.html"
http://wiki.swoole.com/wiki/page/p-case.html"
http://wiki.swoole.com/wiki/page/p-donate.html"
http://127.0.0.1:
time used: 275.22750902176