关于消息队列——ZeroMQ的pipeline模型

官方给的那个pipeline例子很直观，当我们有些一大堆任务（如分析日志）需要用多个worker来工作时，可以不用hadoop这么重量级的产品，使用pipeline完全可以做到。这里做点点修改如下：

 

pipeline模型图如下：

1，首先需要个任务生成器：taskcreate.php

任务生成器负责PUSH任务内容到套接口，需要指定类型为SOCKET_PUSH。生成一堆随机数模拟每个任务所需时间

如，这里运行的结果为需要花费5s

<?php

$context = new ZMQContext();

//push message mode
$sender = new ZMQSocket($context, ZMQ::SOCKET_PUSH);
$sender->bind("tcp://*:5557");

echo "Press Enter when the workers are ready: ";
$fp = fopen('php://stdin', 'r');
$line = fgets($fp, 512);
fclose($fp);
echo "Sending tasks to workers...", PHP_EOL;

// The first message is "0" and signals start of batch
$sender->send(0);

// Send 100 tasks
$total_msec = 0; // Total expected cost in msecs
for ($task_nbr = 0; $task_nbr < 100; $task_nbr++) {
    // Random workload from 1 to 100msecs
    $workload = mt_rand(1, 100);
    $total_msec += $workload;
    $sender->send($workload);
}

printf ("Total expected cost: %d msec\n", $total_msec);
sleep (1); // Give 0MQ time to deliver
?>

2，需要worker.php来具体处理任务

worker负责处理任务，启动多个worker能增强处理能力，缩短处理时间。这里多开启了个套接口订阅广播是为了全部任务处理完毕后汇总结果的程序通知worker.php关闭时用。

同时使用poll封装实现读写监听，区分不同的套接口。

<?php

$context = new ZMQContext();

// pull message from taskcreate.php
$receiver = new ZMQSocket($context, ZMQ::SOCKET_PULL);
$receiver->connect("tcp://localhost:5557");

// push message to result.php
$sender = new ZMQSocket($context, ZMQ::SOCKET_PUSH);
$sender->connect("tcp://localhost:5558");

// Socket for control input
$controller = new ZMQSocket($context, ZMQ::SOCKET_SUB);
$controller->connect("tcp://localhost:5559");
$controller->setSockOpt(ZMQ::SOCKOPT_SUBSCRIBE, "");

// Process messages from receiver and controller
$poll = new ZMQPoll();
$poll->add($receiver, ZMQ::POLL_IN);
$poll->add($controller, ZMQ::POLL_IN);
$readable = $writeable = array();

// Process messages from both sockets
while (true) {
    $events = $poll->poll($readable, $writeable);
    if ($events > 0) {
        foreach ($readable as $socket) {
            if ($socket === $receiver) {
                $message = $socket->recv();
                // Simple progress indicator for the viewer
                echo $message, PHP_EOL;

                // Do the work : times $message by 1000
                usleep($message * 1000);

               // Send results to sink : result=$message*1000
                $sender->send($message * 1000);
            }
            // Any waiting controller command acts as 'KILL'
            else if ($socket === $controller) {
                exit();
            }
        }
    }
}
?>

3，最后汇总结果result.php

完毕后套接口发布消息，订阅了此消息的worker都会关闭。

<?php

$context = new ZMQContext();

// Socket to receive messages on
$receiver = new ZMQSocket($context, ZMQ::SOCKET_PULL);
$receiver->bind("tcp://*:5558");

// Socket for worker control
$controller = new ZMQSocket($context, ZMQ::SOCKET_PUB);
$controller->bind("tcp://*:5559");

// Wait for start of batch
$string = $receiver->recv();

// Process 100 confirmations
$tstart = microtime(true);
$total_msec = 0; // Total calculated cost in msecs
$sum = 0;
for ($task_nbr = 0; $task_nbr < 100; $task_nbr++) {
    $string = $receiver->recv();
    echo $string."\n";
}

$tend = microtime(true);

$total_msec = ($tend - $tstart) * 1000;
echo PHP_EOL;
printf ("Total elapsed time: %d msec", $total_msec);
echo PHP_EOL;

// Send kill signal to workers
$controller->send("KILL");

// Finished
sleep (1);

?>

运行方式如下：

1，开启多个worker.php（如开启2个）

2，开启result.php

3，运行taskcreate.php

 

可以看到原本需要花费5s的任务在3s就处理完毕了，如何将任务均匀地分配给多个worker官方文档中有详细建议。