NetMQ（四）：推拉模式 Push-Pull

最新推荐文章于 2024-03-22 09:55:43 发布

weixin_30693683

最新推荐文章于 2024-03-22 09:55:43 发布

阅读量339

点赞数

文章标签：网络爬虫

原文链接：http://www.cnblogs.com/weiqinl/p/5461184.html

版权

ZeroMQ系列之NetMQ

一：zeromq简介

二：NetMQ 请求响应模式 Request-Reply

三：NetMQ 发布订阅模式 Publisher-Subscriber

四：NetMQ 推拉模式 Push-Pull

NetMQ 推拉模式 Push-Pull

1：简介

推拉模式，也叫管道模式”Parallel Pipeline”。想象一下这样的场景，如果需要统计各个机器的日志，我们需要将统计任务分发到各个节点机器上，最后收集统计结果，做一个汇总。PipeLine比较适合于这种场景，他的结构图，如图1所示

图1 官方图

Ventilator，在管道中生产任务；
Worker ,处理任务；
Sink,收集Worker处理的结果。

2：案例

下面有三个对象Ventilator 消息分发者，Worker 消息处理者，Sink 接受Worker处理消息后返回的结果，耗时的计算处理工作是交给Worker的，如果开多个Worker.exe，可以提升处理速度，Worker的最终目的是分布式计算，部署到多台PC上面，把计算工作交给他们去做（在分布式爬虫上面，每个Worker相当于一个爬虫）。
下面案例结构，如图2所示：

图2

源码：

Ventilator

    static void Main(string[] args)
    {
        // Task Ventilator
        // Binds PUSH socket to tcp://localhost:5557
        // Sends batch of tasks to workers via that socket
        Console.WriteLine("====== VENTILATOR ======");


        //socket to send messages on
        using (NetMQSocket sender = new DealerSocket())
        {
            sender.Bind("tcp://*:5557");

            using (var sink = new DealerSocket())
            {
                sink.Connect("tcp://localhost:5558");

                Console.WriteLine("Press enter when worker are ready");
                Console.ReadLine();

                //the first message it "0" and signals start of batch
                //see the Sink.csproj Program.cs file for where this is used
                Console.WriteLine("Sending start of batch to Sink");
                sink.SendFrame("0");

                Console.WriteLine("Sending tasks to workers");

                //initialise random number generator
                Random rand = new Random(0);

                //expected costs in Ms
                int totalMs = 0;

                //send 100 tasks (workload for tasks, is just some random sleep time that
                //the workers can perform, in real life each work would do more than sleep
                for (int taskNumber = 0; taskNumber < 100; taskNumber++)
                {
                    //Random workload from 1 to 100 msec
                    int workload = rand.Next(0, 100);
                    totalMs += workload;
                    Console.WriteLine("Workload : {0}", workload);
                    sender.SendFrame(workload.ToString());
                }
                Console.WriteLine("Total expected cost : {0} msec", totalMs);
                Console.WriteLine("Press Enter to quit");
                Console.ReadLine();
            }
        }
    }

Worker

    static void Main(string[] args)
    {
        // Task Worker
        // Connects PULL socket to tcp://localhost:5557
        // collects workload for socket from Ventilator via that socket
        // Connects PUSH socket to tcp://localhost:5558
        // Sends results to Sink via that socket
        Console.WriteLine("====== WORKER ======");


        //socket to receive messages on
        using (var receiver = new DealerSocket())
        {
            receiver.Connect("tcp://localhost:5557");

            //socket to send messages on
            using (var sender = new DealerSocket())
            {
                sender.Connect("tcp://localhost:5558");

                //process tasks forever
                while (true)
                {
                    //workload from the vetilator is a simple delay
                    //to simulate some work being done, see
                    //Ventilator.csproj Proram.cs for the workload sent
                    //In real life some more meaningful work would be done
                    string workload = receiver.ReceiveString();

                    //simulate some work being done
                    Thread.Sleep(int.Parse(workload));

                    //send results to sink, sink just needs to know worker
                    //is done, message content is not important, just the precence of
                    //a message means worker is done. 
                    //See Sink.csproj Proram.cs 
                    Console.WriteLine("Sending to Sink");
                    sender.SendFrame(string.Empty);
                }
            }
        }
    }

Sink

    static void Main(string[] args)
    {
        // Task Sink
        // Bindd PULL socket to tcp://localhost:5558
        // Collects results from workers via that socket
        Console.WriteLine("====== SINK ======");

        //socket to receive messages on
        using (var receiver = new DealerSocket())
        {
            receiver.Bind("tcp://localhost:5558");

            //wait for start of batch (see Ventilator.csproj Program.cs)
            var startOfBatchTrigger = receiver.ReceiveString();
            Console.WriteLine("Seen start of batch");

            //Start our clock now
            Stopwatch watch = new Stopwatch();
            watch.Start();

            for (int taskNumber = 0; taskNumber < 100; taskNumber++)
            {
                var workerDoneTrigger = receiver.ReceiveString();
                if (taskNumber % 10 == 0)
                {
                    Console.Write(":");
                }
                else
                {
                    Console.Write(".");
                }
            }
            watch.Stop();
            //Calculate and report duration of batch
            Console.WriteLine();
            Console.WriteLine("Total elapsed time {0} msec", watch.ElapsedMilliseconds);
            Console.ReadLine();
        }
    }