MQTT简介之十四--Ubuntu下Mosquitto 性能测试

1、硬件环境

1.1 、网络拓扑图

           

1.2、对各个服务器的要求

2、实验一:验证单机mosquitto能够支持的最大定阅端数量

2.1、Linux 进程级开启最大文件描述符 调优

2.1.1、开启最大文件数

系统可以开启的最大文件描述符(可同时开启最多的文件数),最大开启1024,可根据需求进行调优。

 

2.1.2、查看系统当前可开启最大文件描述符数

ulimit -n
[root@localhost ~]# ulimit -n
65535

 

2.1.3、修改最大文件描述符数

# 在limits.conf文件内添加如下行“*”标识用户,*为所有用户,可改为用户名。
 
echo -ne 

“
* soft nofile 65535
* hard nofile 65535
”

>>/etc/security/limits.conf
#/etc/security/limits.conf允许所有用户打开999,999个
#<domain>      <type>  <item>         <value>

  *               -       nofile         999999

注:该操作在50万并发下可使用。

 

2.1.4、设置登录自动读取限制参数

修改/etc/pam.d/login文件,在文件中添加如下行

session required /lib/security/pam_limits.so

注:这是告诉Linux在用户完成系统登录后,应该调用pam_limits.so模块来设置系统对该用户可使用的各种资源数量的最大限制(包括用户可打开的最大文件数限制),而pam_limits.so模块就会从/etc/security/limits.conf文件中读取配置来设置这些限制值。修改完后保存此文件。

2.1.5、参考网址:

     https://www.cnblogs.com/xiangsikai/p/9496479.html

2.2、修改mosquitto能够支持的最大客户端连接数

   

vi /etc/mosquitto/mosquitto.conf


#添加内容如下
# The maximum number of client connections to allow. This is
# a per listener setting.
# Default is -1, which means unlimited connections.
# Note that other process limits mean that unlimited connections
# are not really possible. Typically the default maximum number of
# connections possible is around 1024.
#max_connections -1
max_connections 65535

2.3、多进程测试实例 shell脚本启动

#!/bin/bash
#python3 mqtt_sub.py 1 &
#python3 mqtt_sub.py 2 &
counter=1
loop=5000
b=0
hostip="10.10.40.242"#broker 服务端IP
while true
do
    echo "counter==" $counter
    b=$(( $counter % 5 ))
    #sleep 1
    #python3 mqtt_sub.py $counter &
    topic="caoft/test/"$counter
    echo $topic
    #mosquitto_pub -h $hostip -t $topic -m "Hello1111 MQTT2"
    mosquitto_sub -h $hostip -t $topic &
    counter=$[counter+1]
    if [ $counter == $loop ] ; then
	break;
    fi
    if [ $b == 0 ] ; then
 	sleep 1
    fi
done

2.4、结论: 

         ①ubuntu默认安装的mosquito 1.4.8 最大支持订阅端的数量为1024左右,我实际测试位1118  

mosquitto -v
1592383135: mosquitto version 1.4.8 (build date Tue, 18 Jun 2019 11:59:34 -0300) starting
1592383135: Using default config.
1592383136: Opening ipv4 listen socket on port 1883.
1592383136: Error: Address already in use

               ubuntu通过apt-get install 安装Mosquittohttps://blog.csdn.net/caofengtao1314/article/details/106686329

        ②ubuntu默认安装的mosquito 1.6.9 最大支持订阅端的数量为10000左右,我实际测试位10000,

mosquitto -v
1592382811: mosquitto version 1.6.9 starting
1592382811: Using default config.
1592382811: Opening ipv4 listen socket on port 1883.
1592382811: Error: Address already in use

             ubuntu通过手动下载编译mosquittohttps://blog.csdn.net/caofengtao1314/article/details/106686329 

 

3、实验二:每个客户端定义唯一的topic,验证发布端发布第一个定阅端到最后一个定阅端 时间差

3.1、多进程测试实例 shell脚本启动

#!/bin/bash
#python3 mqtt_sub.py 1 &
#python3 mqtt_sub.py 2 &
counter=1
loop=5000
b=0
hostip="10.10.40.242"#broker 服务端IP
while true
do
    echo "counter==" $counter
    b=$(( $counter % 5 ))
    #sleep 1
    #python3 mqtt_sub.py $counter &
    topic="caoft/test/"$counter
    echo $topic
    #mosquitto_pub -h $hostip -t $topic -m "Hello1111 MQTT2"
    mosquitto_sub -h $hostip -t $topic &
    counter=$[counter+1]
    if [ $counter == $loop ] ; then
	break;
    fi
    if [ $b == 0 ] ; then
 	sleep 1
    fi
done

3.2、使用发布端服务器MQTTBox发布消息统计时间

 发布主题caoft/test/1

发布主题 caoft/test/9999

订阅端收到的消息没有时间差

 

4、实验三:使用mosquito_sub 定阅1万个不同主题,使用mosquito_pub发布1万个对应的不同主题,所需要的时间消耗

4.1、订阅端服务器start_mosquito_sub.sh

#!/bin/bash

counter=1
loop=10000
b=0
hostip="10.10.40.242"
endT=0
consume=0
startT=$[$(date +%s%N)/1000000]
while true
do
    echo "counter==" $counter
    b=$(( $counter % 5 ))
    #sleep 1
    #python3 mqtt_sub.py $counter &
    topic="caoft/test/"$counter
    echo $topic
    #mosquitto_pub -h $hostip -t $topic -m "Hello1111 MQTT2"
    mosquitto_sub -h $hostip -t $topic &
    counter=$[counter+1]
    if [ $counter == $loop ] ; then
	break;
    fi
    if [ $b == 0 ] ; then
 	sleep 1
    fi
done

endT=$[$(date +%s%N)/1000000]
consume=$[endT-startT]
echo "sub start ===" $startT "end ===" $endT "consume ==" $consume "counter==" $counter

4.2、发布端服务器start_mosquito_pub.sh

#!/bin/bash
counter=1
loop=10000
b=0
hostip="10.10.40.242"
endT=0
consume=0
startT=$[$(date +%s%N)/1000000]
while true
do
    echo "counter==" $counter
    b=$(( $counter % 5 ))
    #sleep 1
    #python3 mqtt_sub.py $counter &
    topic="caoft/test/"$counter
    echo $topic
    mosquitto_pub -h $hostip -t $topic -m $topic
    counter=$[counter+1]
    if [ $counter == $loop ] ; then
	break;
    fi
    #if [ $b == 0 ] ; then
 #	sleep 1
  #  fi
done
mosquitto_pub -h $hostip -t $topic -m $topic
endT=$[$(date +%s%N)/1000000]
consume=$[endT-startT]
echo "start ===" $startT "end ===" $endT "consume ==" $consume "counter==" $counter

4.3、测试方法

     4.3.1 首先启动start_mosquito_sub.sh 至少需要半个小时

     4.3.2 启动start_mosquito_pub.sh的时候启动秒表计数等待订阅端全部接受完数据

     4.3.3 测试结果

                  发布端消耗时间统计  70.142s

                  订阅端消耗时间统计  71.29s

5、实验四:使用mosquito_sub 定阅一个组主题,但是启动1万个定阅客户端,使用mosquito_pub发布1个组主题,所需要的时间消耗

5.1、订阅端服务器start_mosquito_sub.sh

#!/bin/bash

counter=1
loop=10000
b=0

hostip="10.10.40.242"
endT=0
consume=0
startT=$[$(date +%s%N)/1000000]
while true
do
echo "counter==" $counter
b=$(( $counter % 5 ))
#sleep 1
#python3 mqtt_sub.py $counter &
topic="caoft/test/#"
echo $topic
#mosquitto_pub -h $hostip -t $topic -m "Hello1111 MQTT2"
mosquitto_sub -h $hostip -t $topic &
counter=$[counter+1]
if [ $counter == $loop ] ; then
break;
fi
if [ $b == 0 ] ; then
sleep 1
fi
done

endT=$[$(date +%s%N)/1000000]
consume=$[endT-startT]
echo "sub start ===" $startT "end ===" $endT "consume ==" $consume "counter==" $counter

5.2、发布端服务器使用MQTTBox发布一个组主题

MQTTBox发布caoft/test/111

5.3、测试方法

     5.3.1 首先启动start_mosquito_sub.sh 至少需要半个小时

     5.3.2 启动MQTTBox发布caoft/test/111 的时候启动秒表计数等待订阅端全部接受完数据

     5.3.3 测试结果

             发布端消耗时间统计  延迟3左右秒,订阅端才有响应

             订阅端消耗时间统计  7秒

6、实验五:使用mosquito_sub 定阅一个组主题,但是启动1万个定阅客户端,使用mosquito_pub发布10个组主题,所需要的时间消耗

6.1、订阅端服务器start_mosquito_sub.sh

#!/bin/bash

counter=1
loop=10000
b=0

hostip="10.10.40.242"
endT=0
consume=0
startT=$[$(date +%s%N)/1000000]
while true
do
    echo "counter==" $counter
    b=$(( $counter % 5 ))
    #sleep 1
    #python3 mqtt_sub.py $counter &
    #topic="caoft/test/"$counter
    topic="caoft/test/#"
    echo $topic
    #mosquitto_pub -h $hostip -t $topic -m "Hello1111 MQTT2"
    mosquitto_sub -h $hostip -t $topic &
    counter=$[counter+1]
    if [ $counter == $loop ] ; then
	break;
    fi
    if [ $b == 0 ] ; then
 	sleep 1
    fi
done

endT=$[$(date +%s%N)/1000000]
consume=$[endT-startT]
echo "sub start ===" $startT "end ===" $endT "consume ==" $consume "counter==" $counter

6.2、发布端服务器start_mosquito_pub.sh

#!/bin/bash
try_times=0
init_counter=0
loop=10000
loop_try_times=10
b=0
hostip="10.10.40.242"
endT=0
consume=0
startT=$[$(date +%s%N)/1000000]

counter=$init_counter
while true
do
    while true
    do
        echo "counter==" $counter
    	b=$(( $counter % 5 ))
    	#sleep 1
    	#python3 mqtt_sub.py $counter &
    	topic="caoft/test/"$counter
    	echo $topic
    	mosquitto_pub -h $hostip -t $topic -m $topic
    	counter=$[counter+1]
    	if [ $counter == $loop ] ; then
		break;
    	fi
    	#if [ $b == 0 ] ; then
 	#		sleep 1
  	#  fi
    done
    try_times=$[try_times+1]
    counter=$init_counter
    echo "counter==" $counter "try_times==" $try_times
    if [ $try_times == $loop_try_times ] ; then
	break;
    fi
done
mosquitto_pub -h $hostip -t $topic -m $topic
endT=$[$(date +%s%N)/1000000]
consume=$[endT-startT]
echo "start ===" $startT "end ===" $endT "consume ==" $consume "counter==" $counter "try_times==" $try_times

6.3、测试方法

     6.3.1 首先启动start_mosquito_sub.sh 至少需要半个小时

     6.3.2 启动start_mosquito_pub.sh发布caoft/test/111 的时候启动秒表计数等待订阅端全部接受完数据

     6.3.3 测试结果

             发布端消耗时间统计  延迟3左右秒,订阅端才有响应

             订阅端消耗时间统计  预计消耗至少19个小时,实际测试系统已经崩溃

             

7、实验六:订阅端使用mosquito_sub 一个不同的主题,使用python mqtt_pub 一万个对应的订阅端主题的信息,时间测试

7.1、订阅端服务器start_mosquito_sub.sh

#!/bin/bash

counter=1
loop=10000
b=0
hostip="10.10.40.242"
endT=0
consume=0
startT=$[$(date +%s%N)/1000000]
while true
do
    echo "counter==" $counter
    b=$(( $counter % 5 ))
    #sleep 1
    #python3 mqtt_sub.py $counter &
    topic="caoft/test/"$counter
    #topic="caoft/test/#"
    echo $topic
    #mosquitto_pub -h $hostip -t $topic -m "Hello1111 MQTT2"
    mosquitto_sub -h $hostip -t $topic &
    counter=$[counter+1]
    if [ $counter == $loop ] ; then
	break;
    fi
    if [ $b == 0 ] ; then
 	sleep 1
    fi
done

endT=$[$(date +%s%N)/1000000]
consume=$[endT-startT]
echo "sub start ===" $startT "end ===" $endT "consume ==" $consume "counter==" $counter

7.2、订阅端服务器start_mosquito_pub.sh

# -*- coding: utf-8 -*-

import paho.mqtt.client as mqtt
import sys
import psutil
import os
#import datetime
import time
#from datetime import *

MQTTHOST = "10.10.40.242"
MQTTPORT = 1883
mqttClient = mqtt.Client()

#def get_head_info():
# try:
# raise Exception
# except:
# f = sys.exc_info()[2].tb_frame.f_back
# return '%s, %s, %s, %s, %d' % (str(datetime.now()), f.f_code.co_filename, f.f_code.co_name, str(f.f_lineno),os.getpid())


# 连接MQTT服务器
def on_mqtt_connect():
mqttClient.connect(MQTTHOST, MQTTPORT, 60)
mqttClient.loop_start()


# publish 消息
def on_publish(topic, payload, qos):
mqttClient.publish(topic, payload, qos)

# 消息处理函数
def on_message_come(lient, userdata, msg):

print(msg.topic + " " + ":" + str(msg.payload))


# subscribe 消息
def on_subscribe():
mqttClient.subscribe("caoft/test/#", 1)
mqttClient.on_message = on_message_come # 消息到来处理函数


def main():
on_mqtt_connect()
i = 0
#print(get_head_info() + " start publish")
print("start publish")
#on_publish("caoft/test/1","caoftestt111111", 1)
while True:
topic="caoft/test/"+str(i)
on_publish(topic,topic, 1)
i += 1
if i > 10000:
break
#on_mqtt_connect()
#on_publish("caoft/test/1", "Hello Python!", 1)
#print(get_head_info() + " end publish")
print("end publish")

if __name__ == '__main__':
main()
while True:
time.sleep( 5 )
pass

7.3、测试方法

     

    7.3.1 首先启动start_mosquito_sub.sh 至少需要半个小时

     7.3.2 启动start_mosquito_pub.sh发布c的时候启动秒表计数等待订阅端全部接受完数据

     7.3.3 测试结果

         发布端消耗时间统计  消耗时间为1.31秒

              start publish
               end publish

             订阅端消耗时间统计7.30秒 对比 4、实验三:使用mosquito_sub 定阅1万个不同主题,使用mosquito_pub发布1万个对应的不同主题,所需要的时间消耗 71.29s 节省64秒    

8、试验七:使用python 发布一个组主题,开启1万个订阅端

8.1、发布端mqtt_pub.py

# -*- coding: utf-8 -*-

import paho.mqtt.client as mqtt
import sys
import psutil
import os
#import datetime
import time
#from datetime import *

MQTTHOST = "10.10.40.242"
MQTTPORT = 1883
mqttClient = mqtt.Client()

#def get_head_info():
# try:
# raise Exception
# except:
# f = sys.exc_info()[2].tb_frame.f_back
# return '%s, %s, %s, %s, %d' % (str(datetime.now()), f.f_code.co_filename, f.f_code.co_name, str(f.f_lineno),os.getpid())


# 连接MQTT服务器
def on_mqtt_connect():
mqttClient.connect(MQTTHOST, MQTTPORT, 60)
mqttClient.loop_start()


# publish 消息
def on_publish(topic, payload, qos):
mqttClient.publish(topic, payload, qos)

# 消息处理函数
def on_message_come(lient, userdata, msg):

print(msg.topic + " " + ":" + str(msg.payload))


# subscribe 消息
def on_subscribe():
mqttClient.subscribe("caoft/test/#", 1)
mqttClient.on_message = on_message_come # 消息到来处理函数


def main():
on_mqtt_connect()
i = 0
#print(get_head_info() + " start publish")
print("start publish")
#on_publish("caoft/test/1","caoftestt111111", 1)
while True:
topic="caoft/test/"+str(i)
on_publish(topic,topic, 1)
i += 1
if i > 1:
break
#on_mqtt_connect()
#on_publish("caoft/test/1", "Hello Python!", 1)
#print(get_head_info() + " end publish")
print("end publish")

if __name__ == '__main__':
main()
while True:
time.sleep( 5 ) #不加入延时会导致CPU占有率100%
pass

8.2、订阅端start_mosquito_sub.sh

!/bin/bash
#python3 mqtt_sub.py 1 &
#python3 mqtt_sub.py 2 &
counter=1
loop=10000
b=0
#hostip="114.115.223.53"
hostip="10.10.40.242"
endT=0
consume=0
startT=[(date +%s%N)/1000000]
while true
do
echo "counter==" counterb=(( countercounter &
#topic="caoft/test/"You can't use 'macro parameter character #' in math modetopic
#mosquitto_pub -h hostip−t topic -m "Hello1111 MQTT2"
mosquitto_sub -h hostip−t topic &
counter=[counter+1]if[counter == loop];thenbreak;fiif[b == 0 ] ; then
sleep 1
fi
done

endT=[(date +%s%N)/1000000]
consume=[endT−startT]echo"substart===" startT "end ===" endT"consume=="consume "counter==" $counter

8.3、测试方法

     8.3.1 首先启动start_mosquito_sub.sh 至少需要半个小时

     8.3.2 启动MQTTBox发布caoft/test/111 的时候启动秒表计数等待订阅端全部接受完数据

     8.3.3 测试结果

             发布端消耗时间统计  延迟1左右秒,订阅端才有响应 

             订阅端消耗时间统计 7秒左右    

9、结论: 

     ①ubuntu默认安装的mosquito 1.4.8 最大支持订阅端的数量为1024左右,我实际测试位1118

               ubuntu通过apt-get install 安装MosquittoUbuntu安装使用mosquitto

      ②ubuntu默认安装的mosquito 1.6.9 最大支持订阅端的数量为10000左右,我实际测试位10000,

             ubuntu通过手动下载编译mosquittoUbuntu安装使用mosquitto

      ③在局域网内如果每个客户端定义一个唯一的topic,总共启动1万个客户端,通过发布唯一的topic第一个和第1万个topic的时间是相同的。

       ④对比试验三与试验四:对每一个客户端发布一个消息耗时要比对一个组发布消息慢很多,但是订阅端收到的消息要比发布端发送的消息数量多 ,这是Qos=1导致的

       ⑤对比试验四与实验五:发布10万条组消息,没有导致broker死机,但是订阅端收到的消息要比发布端发送的消息数量多 ,这是Qos=1导致的

       ⑥实验六与实验三对比 确少了MQTT的connect阶段节省了64秒

       ⑦实验四,实验六 可以证明发布一个组的效率基本和发布不同的主题实践基本差不多

       ⑧如果要想解决LOT设备的海量连接、海量主题、海量会话的问题,需要引入集群

               MQTT简介之十五--Ubuntu下Mosquitto 集群搭建

               https://blog.csdn.net/caofengtao1314/article/details/106826840

       

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值