python写爬虫的优势-各种语言写网络爬虫有什么优点缺点？

用PHP做爬虫相当简单,稍微学习下simple_html_dom.php这个库就可以分析DOM了.

比如抓取PHP官网首页新闻插入数据库并在Web上显示只需几行代码:

<?php

// php -S 127.0.0.1:8181 -t /home/eechen/www/

// http://127.0.0.1:8181/index.php

$file = '/home/eechen/data.db3';

$ddl = 'BEGIN; CREATE TABLE IF NOT EXISTS test (id INTEGER PRIMARY KEY AUTOINCREMENT, title VARCHAR(200), content TEXT, time VARCHAR(20)); COMMIT;';

if(!file_exists($file)) { $db = new PDO('sqlite:'.$file); $db->exec($ddl); } else { $db = new PDO('sqlite:'.$file); }

require dirname(__FILE__).'/simple_html_dom.php';

$html = file_get_html('http://cn2.php.net');

$db->beginTransaction();

foreach($html->find('article.newsentry') as $v) {

$title = trim($v->find('h2.newstitle', 0)->plaintext);

$content = trim($v->find('div.newscontent', 0)->plaintext);

$time = trim($v->find('time', 0)->plaintext);

$stmt = $db->prepare("INSERT INTO test (title, content, time) VALUES (?, ?, ?)");

$stmt->execute(array($title, $content, $time));

$stmt = null;

}

$db->commit();

header('Content-Type: text/plain; charset=utf-8');

var_export($db->query("SELECT * FROM test")->fetchAll(PDO::FETCH_ASSOC));

$dbh = null;另外PHP用libcurl库提供的curl_multi也能并行发出多个HTTP请求:

https://my.oschina.net/eechen/blog/745224

接口1: php -S 127.0.0.1:8080 -t /home/eechen/www

接口2: php -S 127.0.0.2:8080 -t /home/eechen/www

/home/eechen/www/index.php:

<?php

header('Content-Type: application/json; charset=utf-8');

echo json_encode(array('SERVER_NAME' => $_SERVER['SERVER_NAME']));

//串行访问需要sum(2,1)秒,并行访问需要max(2,1)秒.

($_SERVER['SERVER_NAME'] == '127.0.0.1') ? sleep(2) : sleep(1);

?>

并行发出多个请求:

<?php

$url[] = 'http://127.0.0.1:8080';

$url[] = 'http://127.0.0.2:8080';

$mh = curl_multi_init();

foreach($url as $k => $v) {

$ch[$k] = curl_init($v);

curl_setopt($ch[$k], CURLOPT_HEADER, 0); //不输出头

curl_setopt($ch[$k], CURLOPT_RETURNTRANSFER, 1); //exec返回结果而不是输出,用于赋值

curl_multi_add_handle($mh, $ch[$k]); //决定exec输出顺序

}

$running = null;

$starttime = microtime(true);

//执行批处理句柄(类似pthreads多线程里的start开始和join同步)

do {

//CURLOPT_RETURNTRANSFER如果为0,这里会直接输出获取到的内容.

//如果为1,后面可以用curl_multi_getcontent获取内容.

curl_multi_exec($mh, $running);

//阻塞直到cURL批处理连接中有活动连接,不加这个会导致CPU负载超过90%.

curl_multi_select($mh);

} while ($running > 0);

echo microtime(true) - $starttime." "; //耗时约2秒

foreach($ch as $v) {

$json[] = curl_multi_getcontent($v);

curl_multi_remove_handle($mh, $v);

}

curl_multi_close($mh);

var_export($json);

//输出:

2.0015449523926

array (

0 => '{"SERVER_NAME":"127.0.0.1"}',

1 => '{"SERVER_NAME":"127.0.0.2"}',

)当然,也可以使用PHP的多线程扩展pthreads用多线程实现并行发起多个请求,不过必须要求使用线程安全版本PHP.另外鸟哥的Yar扩展也支持并行调用,不过是用作PHP RPC的.

因为PHP内置了SQLite数据库,所以存储起来很方便.

又因为PHP内置了HTTP服务器,所以展示起来也很方便.

建立HTTP服务:

php -S 127.0.0.1:8080 -t /www

打开浏览器访问:

xdg-open http://127.0.0.1:8080