php扫描网站所有url,使用php如何扫描url死链接

最新推荐文章于 2024-05-13 15:06:24 发布

顾扯淡

最新推荐文章于 2024-05-13 15:06:24 发布

阅读量616

点赞数

文章标签： php扫描网站所有url

这篇文章主要介绍了关于使用php如何扫描url死链接，有着一定的参考价值，现在分享给大家，有需要的朋友可以参考一下

* 从Packagist上搜索需要的包

https://packagist.org/

* 通过composer下载依赖包composer require guzzlehttp/guzzle

composer require league/csv

* 使用composer自动加载器, 编写scan.php<?php

// 1. 使用composer自动加载器

require 'vendor/autoload.php';

use GuzzleHttp\RequestOptions;

// 2. 实例Guzzle HTTP客户端

$client = new \GuzzleHttp\Client();

$options = [

RequestOptions::TIMEOUT => 3,

RequestOptions::DECODE_CONTENT => false,

RequestOptions::HEADERS => [

'User-Agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36',

]

];

// 3. 打开迭代处理csv

// 或者传用户自定义命令行参数指定输入文件 $argv[1]

// $file = new SplFileObject('../data/t_video.csv'); /* $csvRow[4], $csvRow[5] */

$file = new SplFileObject('../data/urls.csv');

$csv = \League\Csv\Reader::createFromFileObject($file);

foreach ($csv as $csvRow) {

$url = $csvRow[0];

echo 'scanning ',$url,'... ';

try {

// 4. 发送http options请求

$httpResponse = $client->request('GET', $url, $options);

// 5. 检查http相应的状态码

$code = $httpResponse->getStatusCode();

if ($code === 200) {

echo "\033[32m[OK]\033[0m",PHP_EOL;

} else {

throw new \Exception();

}

} catch (\Exception $e) {

// 6. 把死链发给标准输出

// echo $url.PHP_EOL;

echo "\033[31m[ERROR]\033[0m ".$e->getMessage().PHP_EOL;

}

* input csv:

../data/urls.csvhttps://www.baidu.com

https://mail.qq.com/cgi-bin/frame_html?sid=CYcBjsDbOqznWhVO&r=375cccc57697ed7d00ae5d751663a71c

https://pan.baidu.com/disk/home?errno=0&errmsg=Auth%20Login%20Sucess&&bduss=&ssnerror=0&traceid=#/all?vmode=list&path=%2F05.php%2F25K%20PHP%E9%9D%A2%E8%AF%95%E8%A7%86%E9%A2%91%E6%95%99%E7%A8%8B

http://dict.youdao.com/w/eng/components/#keyfrom=dict2.index

http://php.net/manual/en/splfileobject.fwrite.php

https://www.baidu.com/s?ie=utf-8&f=8&rsv_bp=1&rsv_idx=2&tn=baiduhome_pg&wd=ansi%20%E7%BB%88%E7%AB%AF%E9%A2%9C%E8%89%B2%20%5B%5C33&rsv_spt=1&oq=ansi%2520%25E7%25BB%2588%25E7%25AB%25AF%25E9%25A2%259C%25E8%2589%25B2&rsv_pq=8b17bd6e0027882b&rsv_t=fcf6oR2SbHi9Cpu2eThdv3AQvGwSDf7ecjv7QBvjXoZ3SMpBem3pdNzlNRNmuOW%2BEowe&rqlang=cn&rsv_enter=1&inputT=2640&rsv_sug3=68&rsv_sug2=0&rsv_sug4=3243

https://blog.csdn.net/SLASH_24/article/details/54846392

https://www.jb51.net/article/42358.htm

https://www.cnblogs.com/xudong-bupt/p/3721210.html

http://www.cnblogs.com/mingzhanghui/p/9314906.html

https://packagist.org/packages/maatwebsite/excel

https://www.phptherightway.com/#use_the_current_stable_version

https://doc.phpspider.org/methods.html

http://nosuchurl

http://deadurl

output: