XMPP(ejabberd) 集群部署说明（测试为27台集群）

HMJ589119

已于 2024-05-06 11:38:27 修改

阅读量2k

点赞数

文章标签：中间件 bash 数据库

于 2019-10-18 10:37:03 首次发布

本文链接：https://blog.csdn.net/HMJ589119/article/details/102620454

版权

XMPP(ejabberd) 集群部署说明（测试为27台集群）

3.5 确认防火墙关闭,ssh配置互相免密 7

3.6 创建数据库ejabberd 7

3.7 从github下载原始库，并导入 7

3.8 修改配置文件ejabberd.yml 7

3.9 启动第一台作为master 15

3.10 注册用户 15

3.11 用注册账号从浏览器访问ejabberd测试 16

4.1 使用WebSocket压力测试软件进行并发测试 17

4.2 使用tsung软件进行集中注册测试 18

五、总结 24

一、项目简介

1.1背景

目前我司苏州业务的机顶盒app模拟遥控器功能遇到一个瓶颈，xmpp集群的19台服务器无法承载50-60万用户规模，其中长时间在线用户的用户可能达到20多万，客户想通过增加集群节点方式扩容业务，但是当增加到20台的时候，集群会挂掉！

为了解决这个问题，特进行此次超过25台xmpp集群研究，扩容目标50万在线！

1.2项目分析

按照要求，若扩容目标为50万在线，采用25台服务器分担,即每台需承受最高2万压力，牵涉到两问题：1.同时在线，由于我们的遥控器功能采用的是websocket协议长链接，需要达到同时超过2万端口处于并发链接状态;2.注册速率，按最严格的集中注册来说，单台服务器需承载2万用户集中注册。

1.3架构设计

按照超过25台节点方式构建xmpp集群，为了提高数据库性能，采用独立服务器作为数据库服务器，注意：需选用CPU频率和内核数尽量高的主机作为mysql服务器；需选用内存频率足够快的主机作为redis服务器，为了性能最大化集群，最好采用硬件作为调度服务器，综合分析，此次我们内部测试架构按如下架构：

二、系统配置及优化

2.1操作系统要求

CentOS 7 及以上;xmpp节点内存建议大于4G;mysql内存按照每个连接2M配置，磁盘剩余空间大于16G,建议lvm分区,raid5及以上.

2.2基础环境优化

2.2.1 ulimit 限制优化

LMS=`cat /etc/rc.d/rc.local |grep 'ulimit -SHn 655350'|wc -l`

if [ $LMS -eq 0 ];then

echo -e "ulimit -SHn 655350 \n ulimit -s 655350" >> /etc/rc.d/rc.local;fi

ulimit -SHn 655350 ; ulimit -s 655350

LMT=`cat /etc/security/limits.conf |grep 'soft nofile 655350'|wc -l`

if [ $LMT -eq 0 ];then

echo '

* soft nofile 655350

* hard nofile 655350

* soft nproc 655350

* hard nproc 655350

* soft memlock unlimited

* hard memlock unlimited

* soft core unlimited

* hard core unlimited

' >> /etc/security/limits.conf ;fi

2.2.2sysctl限制优化

SCT=`cat /etc/sysctl.conf |grep 'net.core.somaxconn = 65535'|wc -l`

if [ $SCT -eq 0 ];then

echo '

net.nf_conntrack_max = 1000000

net.netfilter.nf_conntrack_max = 1000000

net.netfilter.nf_conntrack_tcp_timeout_time_wait = 10

net.ipv4.tcp_keepalive_time = 120

net.ipv4.tcp_keepalive_probes = 3

net.ipv4.tcp_keepalive_intvl = 3

fs.file-max = 6553500

fs.nr_open = 6553500

net.ipv6.conf.all.disable_ipv6 = 1

net.ipv6.conf.default.disable_ipv6 = 1

net.ipv4.neigh.default.gc_stale_time = 120

net.ipv4.conf.default.arp_announce = 2

net.ipv4.conf.all.arp_announce = 2

net.ipv4.conf.lo.arp_announce = 2

net.ipv4.conf.all.arp_ignore = 1

net.ipv4.conf.lo.arp_ignore = 1

net.ipv4.icmp_echo_ignore_broadcasts = 1

net.ipv4.icmp_ignore_bogus_error_responses = 1

net.ipv4.conf.all.accept_source_route = 0

net.ipv4.conf.default.accept_source_route = 0

kernel.core_uses_pid = 1

net.ipv4.tcp_syncookies = 1

kernel.msgmnb = 655360

kernel.msgmax = 655360

net.ipv4.tcp_max_tw_buckets = 6000

net.ipv4.tcp_sack = 0

net.ipv4.tcp_window_scaling = 1

net.ipv4.tcp_rmem = 4096 87380 4194304

net.ipv4.tcp_wmem = 4096 16384 4194304

net.core.wmem_default = 8388608

net.core.rmem_default = 8388608

net.core.rmem_max = 16777216

net.core.wmem_max = 16777216

net.ipv4.tcp_max_orphans = 3276800

net.ipv4.tcp_max_syn_backlog = 262144

net.ipv4.tcp_timestamps = 0

net.ipv4.tcp_synack_retries = 3

net.ipv4.tcp_syn_retries = 3

net.ipv4.tcp_tw_recycle = 1

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_mem = 94500000 915000000 927000000

net.ipv4.tcp_fin_timeout = 5

net.ipv4.conf.all.rp_filter = 1

net.ipv4.conf.default.rp_filter = 1

kernel.shmmax = 68719476736

kernel.shmall = 4294967296

net.core.netdev_max_backlog = 655350

net.core.somaxconn = 32768

net.core.optmem_max = 16777216

net.ipv4.tcp_mtu_probing = 1

net.ipv4.ip_local_port_range = 1024 65000

' >> /etc/sysctl.conf ;fi;

sysctl -p

2.2.3 关闭selinux 、防火墙，并配置开机不启动

setenforce 0 ;sed -i '/^SELINUX=/s/SELINUX.*/SELINUX=disabled/g' /etc/selinux/config

2.2.4 修改时区,与服务器对时,语言

echo 'ZONE=\"Asia/Shanghai\"

UTC=true

ARC=false' > /etc/sysconfig/clock

timedatectl set-timezone Asia/Shanghai

ntpdate ntp1.aliyun.com

NTP=`cat /etc/crontab |grep 'ntpdate'|wc -l`

if [ $NTP -eq 0 ];then

echo '0 0 * * * root /usr/sbin/ntpdate ntp1.aliyun.com > /dev/null' >> /etc/crontab ;fi

yum reinstall -y kde-l10n-Chinese || yum install -y kde-l10n-Chinese

yum reinstall -y glibc-common || yum install -y glibc-common

echo 'LANG="zh_CN.UTF-8" ' > /etc/locale.conf

source /etc/locale.conf

2.2.5 创建普通权限用户uxmpp (XMPP项目不一定需要)

groupadd -g 1200 uxmpp || groupmod -g 1200 uxmpp

useradd -g 1200 -u 1200 -d /uxmpp -m uxmpp

usermod -g 1200 -u 1200 -d /uxmpp -m uxmpp

echo xmpppass1 | passwd root --stdin

echo xmpppass2 | passwd uxmpp --stdin

2.2.6 优化历史记录

HSFS=`cat /etc/bashrc |grep 'HISTFILESIZE=50000'|wc -l`

if [ $HSFS -eq 0 ];then

echo '

HISTFILESIZE=50000

HISTSIZE=50000

HISTTIMEFORMAT="%Y-%m-%d %H:%M:%S "

export HISTTIMEFORMAT

' >> /etc/bashrc ;fi

source /etc/bashrc

2.2.7卸载不需要的服务(可以直接在后面加更多),去除开机启动

yum remove -y mariadb*

systemctl stop NetworkManager firewalld

systemctl disable NetworkManager firewalld

2.3 mysql数据库部署

2.3.1 mysql部署

由于centos7系统已经完善离线装包服务,可以直接下载依赖包再离线安装,不再建议源码安装,建议下载捆绑包,再根据需要安装,以下以mysql-5.7.26为例安装脚本:

# 下载下载捆绑包:

wget https://dev.mysql.com/get/Downloads/MySQL-5.7/mysql-5.7.26-1.el7.x86_64.rpm-bundle.tar

# 确认包是否存在,并安装

if [ ! -f mysql-5.7.*.rpm-bundle.tar ];then echo mysql.5.7组合tar包不存在 ; exit 4 ;fi

tarbaoshu=$( ls -lh mysql-5.7.*.rpm-bundle.tar | wc -l ) ;echo $tarbaoshu

if [ $tarbaoshu -gt 1 ];then echo mysql.5.7组合tar包 ; exit 5 ;fi

tar -axf mysql-5.7.*.rpm-bundle.tar ;echo > /var/log/mysqld.log

rpm -Uvh mysql-community-*.rpm && cp /etc/my.cnf /etc/my.cnf.original && systemctl restart mysqld || echo "安装失败,请自行检查原因!"

sleep 5

# 修改初始复杂密码

TMPASS=$(awk '/temporary password/{print $NF}' /var/log/mysqld.log)

mysqladmin --user=root --password=$TMPASS password "hmj1314+" >/dev/null

# 修改mysql配置文件

echo '请根据需要修改my.cnf配置文件,建议直接复制别人配置好的,以下为参考'

if [ -f "/etc/my.cnf" ] ;then

cp /etc/my.cnf /etc/my.cnf.pre_one

echo '如果之前有重要配置，请至/etc/my.cnf.pre_one恢复'

echo '# for mysql5.7

[client]

default-character-set=utf8mb4

socket = /var/lib/mysql/mysql.sock

[mysql]

default-character-set=utf8mb4

[mysqld]

explicit_defaults_for_timestamp=true

character-set-server=utf8mb4

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

symbolic-links=0

log-error=/var/log/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid

max_connections = 65535

wait_timeout = 1800

interactive_timeout = 1800

thread_cache_size = 400

max_connect_errors = 200

slow_query_log = ON

slow_query_log_file=mysql_slow.log

max_allowed_packet = 500M

default-storage-engine=INNODB

validate_password_policy=0

validate_password_length=6

expire_logs_days = 7 # 删除多少天前的日志

log_bin_trust_function_creators = 1

server_id=1 #集群中需各不相同

log-bin=xmpp-binlog

binlog_format="mixed"

relay_log_purge=off

lower_case_table_names=1

#CREATE database $dname DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci ;

#show VARIABLES like "%max_allowed_packet%";

' > /etc/my.cnf

# 再次重启修改成简易密码

systemctl restart mysqld;systemctl status mysqld && sleep 5

TMPASS=$(awk '/temporary password/{print $NF}' /var/log/mysqld.log)

mysqladmin --user=root --password=$TMPASS password "hmj314+" >/dev/null

echo +++++++++++++++++++++++++++++++++++++

mysqladmin --user=root --password="hmj1314+" password "hmj1314" >/dev/null

echo -------------------------------------------------------------

rm -rf mysql-community-*.rpm #清理tar包解压文件

echo '安装完毕,现在请进行登陆测试,密码公司默认.否则请自行排错!

mysql -hlocalhost -uroot -p '

2.3.2 mysql主从集群配置

主从都开启半同步

mysql -uroot -phmj1314 #以下进入mysql执行

#> INSTALL PLUGIN rpl_semi_sync_master SONAME "semisync_master.so";

INSTALL PLUGIN rpl_semi_sync_slave SONAME "semisync_slave.so";

SET GLOBAL rpl_semi_sync_master_enabled = 1;

grant replication slave on *.* to 'rep'@'%' identified by 'hmj1314';

flush privileges;

reset master; show master status; #查看”主库”binlog的”File” 和”Position”信息

下面为”从库”配置项,参照上面信息

CHANGE MASTER TO MASTER_HOST='$主库IP', MASTER_PORT=3306,MASTER_USER='rep',\

MASTER_PASSWORD='hmj1314',MASTER_LOG_FILE='$File',MASTER_LOG_POS=’$Position’;

2.4.redis数据库部署

使用redis服务器作为session存放库,以redis5.0.5示例安装,下载包后执行如下命令;

useradd -s /sbin/nologin -d /var/lib/redis redis

yum install -y gcc gcc-c++ make automake pkgconfig

tar -axf redis-5.0.5.tar.gz && cd redis-5.0.5 && make && make install

./utils/install_server.sh ##根据提示全部回车默认安装

启动redis

/etc/init.d/redis_6379 restart && /etc/init.d/redis_6379 status

三、XMPP集群部署

3.1 集群环境配置

所有准备用来做XMPP集群的主机的主机名依次改成XMPP01-XMPPnn,或者根据IP来排序,方便识别,并把所有加入集群的主机名和IP信息对应追加写入/etc/hosts文件，配置域名解析，例如:

...

192.168.1.70 xmpp70

192.168.1.71 xmpp71

192.168.1.72 xmpp72

192.168.1.73 xmpp73

....

3.2官方下载ejabberd

wget https://www.process-one.net/downloads/ejabberd/19.05/ejabberd-19.05-linux-x64.run

3.3安装ejabberd

chmod +x ejabberd-19.05-linux-x64.run

./ejabberd-19.05-linux-x64.run

#域名写hmj.com、启用集群,密码自己设，其他全部回车，至安装完毕！

# ejabberd server domain [xmpp]:hmj.com

# Cluster [y/N]:y

3.4 配置服务和环境

cp /opt/ejabberd-19.05/bin/ejabberd.service /etc/init.d/

VEJB=`grep ejabberd-19.05 /etc/profile` ;if [ $? -ne 0 ];then echo 'export PATH=$PATH:/opt/ejabberd-19.05/bin' >> /etc/profile ;fi && source /etc/profile && tail -2 /etc/profile

3.5 确认防火墙关闭,ssh配置互相免密

systemctl disable firewalld ;systemctl stop firewalld.service ;systemctl status firewalld.service

for targetIP in 192.168.1.{70..98};do ssh-copy-id $targetIP ;done

3.6 创建数据库ejabberd

# 进入mysql创建数据库ejabberd (生产环境建议读写分离):

mysql -uroot -phmj1314

#> CREATE database ejabberd DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci ;

#> show databases;

#> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'hmj1314' WITH GRANT OPTION;

#> flush privileges;

3.7 从github下载原始库，并导入

# 下载网站 ejabberd/sql at master · processone/ejabberd · GitHub

mysql -uroot -phmj1314 ejabberd < mysql.new.sql #导入下载的库

3.8 修改配置文件ejabberd.yml

# 以下为我的配置文件全文及注释说明供参考:

echo ‘###

###' ejabberd configuration file for 19.05 by hmj

###

### The parameters used in this configuration file are explained at

###

### https://docs.ejabberd.im/admin/configuration

###

### The configuration file is written in YAML.

### *******************************************************

### ******* !!! WARNING !!! *******

### ******* YAML IS INDENTATION SENSITIVE *******

### ******* MAKE SURE YOU INDENT SECTIONS CORRECTLY *******

### *******************************************************

### Refer to http://en.wikipedia.org/wiki/YAML for the brief description.

### However, ejabberd treats different literals as different types:

###

### - unquoted or single-quoted strings. They are called "atoms".

### Example: dog, 'Jupiter', '3.14159', YELLOW

###

### - numeric literals. Example: 3, -45.0, .0

###

### - quoted or folded strings.

### Examples of quoted string: "Lizzard", "orange".

### Example of folded string:

### > Art thou not Romeo,

### and a Montague?

###

language: "en"

hosts:

- "hmj.com"

loglevel: 3

log_rotate_size: 10485760

log_rotate_date: ""

log_rotate_count: 1

log_rate_limit: 100

certfiles:

- "/opt/ejabberd/conf/server.pem"

ca_file: "/opt/ejabberd/conf/cacert.pem"

listen:

port: 5222

#监听端口

ip: "::"

#监听IP,::表示IPV4和IPV6

module: ejabberd_c2s

#模块名称

#max_stanza_size: 262144

#此选项指定XML节的最大最大字节数,不写为无穷大

shaper: c2s_shaper

#限流器及对应规则名称

access: c2s

#acl访问限制规则

#backlog: 30000

#设置TCP backlog队列长度，建议设置得大一些。因为移动网络下，连接断开重连相对频繁，backlog队列较大可以保证服务端的连接accept能力

starttls_required: false

#与端口的连接需要STARTTLS加密,true|false

port: 5269

ip: "::"

module: ejabberd_s2s_in

max_stanza_size: 524288

port: 5443

ip: "::"

module: ejabberd_http

tls: false

#连接后使用SSL对流量进行加密,true|false

request_handlers:

#HTTP请求路径及提供模块

"/admin": ejabberd_web_admin

"/api": mod_http_api

"/bosh": mod_bosh

"/captcha": ejabberd_captcha

"/upload": mod_http_upload

"/ws": ejabberd_http_ws

"/oauth": ejabberd_oauth

port: 5280

ip: "::"

module: ejabberd_http

request_handlers:

"/admin": ejabberd_web_admin

"/register": mod_register_web

"/xmpp": ejabberd_http_ws

"/api": mod_http_api

"/bosh": mod_bosh

"/captcha": ejabberd_captcha

"/upload": mod_http_upload

"/ws": ejabberd_http_ws

"/oauth": ejabberd_oauth

port: 1883

ip: "::"

module: mod_mqtt

backlog: 1000

port: 5281

module: ejabberd_http

tls: false

request_handlers:

"/register": mod_register_web

#页面注册

port: 5333

module: ejabberd_http

request_handlers:

"/xmpp": ejabberd_http_ws

#澄视自定义websocket监听端口

s2s_use_starttls: false

#false|optional|required|required_trusted

acl:

local:

user_regexp: ".*"

loopback:

ip:

- "127.0.0.0/8"

- "::1/128"

- "::FFFF:127.0.0.1/128"

- "192.168.1.0/24"

- "0.0.0.0/0"

admin:

user:

- "admin@hmj.com"

- "adminhmj@hmj.com"

- "hmj@hmj.com"

access_rules:

#对应上面ACL规则

local:

- allow: local

c2s:

# - deny: blocked

- allow

announce:

- allow: admin

configure:

- allow: admin

muc_create:

- allow: local

pubsub_createnode:

- allow: local

trusted_network:

# - allow: loopback

- allow

api_permissions:

#API应用程序接口权限

"console commands":

from:

- ejabberd_ctl

who: all

what: "*"

"admin access":

who:

- access:

- allow:

- acl: loopback

- acl: admin

- oauth:

- scope: "ejabberd:admin"

- access:

- allow:

- acl: loopback

- acl: admin

what:

- "*"

- "!stop"

- "!start"

"public commands":

who:

- ip: "127.0.0.1/8"

what:

- "status"

- "connected_users_number"

shaper:

#限制连接流量,字节每秒

# normal: 1000

# fast: 50000

normal: 100000

fast: 5000000

shaper_rules:

#对应限流器规则

max_user_sessions: 100

# max_user_sessions: 10

max_user_offline_messages:

- 5000: admin

- 100

c2s_shaper:

- none: admin

- normal

s2s_shaper: fast

max_fsm_queue: 10000

#控制出队消息队列长度，建议不要较大

acme:

#为ejabberd服务的域自动获取SSL证书,要求在端口80上发送,Automatic Certificate Management Environment

contact: "mailto:admin@hmj.com"

ca_url: "https://acme-v01.api.letsencrypt.org"

## Staging environment模拟环境

# ca_url: https://acme-staging-v02.api.letsencrypt.org/directory

## Production environment (the default):生产环境

# ca_url: https://acme-v02.api.letsencrypt.org/directory

# auto: true

# auto: false

# cert_type: rsa

#ejabberdctl request-certificate hmj.com #请求域证书

#ejabberdctl request-certificate all #请求所有域证书

#ejabberdctl list-certificates #查看使用ACME获得的证书

#ejabberdctl revoke-certificate /path/to/cert/file #撤消证书

#tree /var/lib/ejabberd #目录树中account.key是用于标识ACME帐户的EC私钥，

#使用openssl命令检查其内容: openssl ec -text -noout -in /var/lib/ejabberd/acme/account.key

# ca_url: "https://acme-v01.api.letsencrypt.org"

registration_timeout: infinity

# infinity

# registration_timeout: 1

modules:

mod_adhoc: {}

# 临时命令

mod_admin_extra: {}

mod_announce:

# 管理公告

access: announce

mod_avatar: {}

mod_blocking: {}

mod_bosh: {}

mod_caps: {}

mod_carboncopy: {}

mod_client_state: {}

mod_configure: {}

mod_disco: {}

mod_fail2ban: {}

mod_http_api: {}

mod_http_upload:

put_url: "https://@HOST@:5443/upload"

mod_last: {}

mod_mam:

# 消息存档管理

## Mnesia is limited to 2GB, better to use an SQL backend

## For small servers SQLite is a good fit and is very easy

## to configure. Uncomment this when you have SQL configured:

db_type: sql

assume_mam_usage: true

default: always

mod_mqtt: {}

mod_muc:

access:

- allow

access_admin:

- allow: admin

access_create: muc_create

access_persistent: muc_create

default_room_options:

allow_subscription: true # enable MucSub

mam: true

mod_muc_admin: {}

mod_offline:

access_max_user_messages: max_user_offline_messages

mod_ping: {}

mod_privacy: {}

mod_private: {}

mod_proxy65:

# 实现使用SOCKS5进行字节流文件传输

access: local

max_connections: 100

# max_connections: 5

mod_pubsub:

# 发布-订阅服务

access_createnode: pubsub_createnode

plugins:

- "flat"

- "pep"

force_node_config:

## Avoid buggy clients to make their bookmarks public

"storage:bookmarks":

access_model: whitelist

mod_push: {}

# 推送通知

mod_push_keepalive: {}

mod_register:

welcome_message:

subject: "Welcome!"

body: |-

Hi.

Welcome to this Jabber server.

Check http://www.jabber.org

Bye

## 带内注册

## Only accept registration requests from the "trusted"

## network (see access_rules section above).

## Think twice before enabling registration from any

## address. See the Jabber SPAM Manifesto for details:

## https://github.com/ge0rg/jabber-spam-fighting-manifesto

ip_access: trusted_network

registration_watchers:

- "admin1@hmj.com"

- "hmj@hmj.com"

redirect_url: "http://192.168.1.71:5280/register"

mod_register_web: {}

# 帐户注册网站,由listen定义,如https://hmj.com:5281/register/

mod_roster:

# 名册管理

versioning: true

mod_s2s_dialback: {}

mod_shared_roster: {}

mod_stream_mgmt:

resend_on_timeout: if_offline

mod_vcard: {}

mod_vcard_xupdate: {}

mod_version:

# 软件版本

show_os: false

# 允许所有域匿名登入

# auth_method: [anonymous]

# anonymous_protocol: login_anon

### Local Variables:

### mode: yaml

### End:

### vim: set filetype=yaml tabstop=8

new_sql_schema: true

default_db: sql

# sm_db_type: sql

# sm_db_type: mnesia

sm_db_type: redis

sql_type: mysql

sql_server: "192.168.1.71"

sql_port: 3306

sql_database: "ejabberd"

sql_username: "root"

sql_password: "hmj1314"

sql_pool_size: 10

sql_keepalive_interval: 14400

sql_start_interval: 30

redis_server: "192.168.1.101"

redis_port: 6379

redis_db: 0

redis_connect_timeout: 3

' > /opt/ejabberd/conf/ejabberd.yml

3.9 启动第一台作为master

/opt/ejabberd-19.05/bin/start && /opt/ejabberd-19.05/bin/status

3.10 注册用户

# ejabberdctl help

ejabberdctl registered_users hmjcom

ejabberdctl register admin hmj.com hmj1314

ejabberdctl list_cluster

3.11 用注册账号从浏览器访问ejabberd测试

# http://192.168.1.70:5280/admin ，查看是否成功登入，不成功根据jeabberd日志排错。

# 用户名: admin@hmj.com ,密码: hmj1314

3.12 其他节点配置安装

#其他节点安装好后，不用启动,复制master70文件并改成自己主机名,然后启动节点,如下命令：

killall -9 -u ejabberd ;rm -rf /opt/ejabberd/database/ejabberd@*/* ;LOCALIP=`ifconfig|grep 192.168.1.255|awk '{print $2}'|head -1` && echo $LOCALIP && scp 192.168.1.70:/opt/ejabberd/conf/ejabberd.yml /opt/ejabberd/conf/ejabberd.yml && scp 192.168.1.70:/opt/ejabberd/.erlang.cookie /opt/ejabberd/.erlang.cookie && scp 192.168.1.70:/etc/hosts /etc/hosts

/opt/ejabberd-19.05/bin/start && /opt/ejabberd-19.05/bin/status

VEJB=`grep ejabberd-19.05 /etc/profile` ;if [ $? -ne 0 ];then echo 'export PATH=$PATH:/opt/ejabberd-19.05/bin' >> /etc/profile ;fi && source /etc/profile && tail -2 /etc/profile

3.13将启动的新节点都加入集群

#将启动的新节点都加入master，至大于等于25台，并从浏览器确认是否加入成功：

ejabberdctl join_cluster ejabberd@xmpp70

#创建完毕，确认成功连接27个节点

3.14节点错误恢复方法

# 重复执行上面3.12-3.13步骤即可。

四、对xmpp集群进行压力测试

4.1 使用WebSocket压力测试软件进行并发测试

4.1.1 压测软件设置

本次测试软件来源网络，软件名称为：WebSocket压力并发测试by煌v1.0.4.exe，如下图。

使用三台电脑分别连接被压测节点，配置和压测结果如下：

.1.2 WebSocket压测结果分析

对被压测机进行WebSocket处于连接状态的端口进行统计，结果如下：

[root@xmpp71 ~]# netstat -anplut | grep 5333 | grep ESTABLISHED| wc -l

32710

得出结果为32710处于同时连接状态，满足单节点大于2万要求；

4.2 使用tsung软件进行集中注册测试

4.2.1 压测分析

假设要对50万用户注册就行压测,如果采用25台服务器分担,即每台需承受2万压力,考虑到连接时可能丢失部分连接,测试时可以尝试压测单台2.5万-3万.

4.2.2 安装tsung

Centos7安装tsung非常方便,可以直接yum自动解决依赖安装:

# a 备份系统原有yum源,使用阿里云yum源:

yum install wget -y

mkdir -p /etc/yum.repos.d/original

mv -f /etc/yum.repos.d/* /etc/yum.repos.d/original

wget http://mirrors.aliyun.com/repo/Centos-7.repo -O /etc/yum.repos.d/Centos-7.repo

wget http://mirrors.aliyun.com/repo/epel-7.repo -O /etc/yum.repos.d/epel-7.repo

yum repolist

# b 安装tsung:

yum install tsung -y

4.2.3 tsung压力测试

4.2.3.1以下是我配置的注册压力测试文件全文及注释,供参考:

mkdir -p ~/.tsung/

echo '<?xml version="1.0"?>

<!DOCTYPE tsung SYSTEM "/opt/lsmp/openfire/tsung/share/tsung/tsung-1.0.dtd">

</client>

</clients>

</servers>

<load>

</arrivalphase>

</load>

</options>

</request>

<match do="abort" when="match">error</match>

</request>

</request>

</request>

</transaction>

</request>

</request>

</session>

</sessions>

</tsung>

' > ~/.tsung/tsung.xml

4.2.3.2启动压测,并生成压测报告

tsung -f ~/.tsung/tsung.xml start

压测完成后,进入压测日志文件夹,并使用分析工具对日志进行加工生成报告:

cd /root/.tsung/log/2019xxxx-xxxx/ && tsung_stats

4.2.3.3配置nginx，对日子进行浏览查看

为查看日志方便,以nginx为例参考下文修改配置文件:

user root; #暂时以root方式启动,不用时注意切回

...

http { ...

server { ...

location ~ /log/.* {

root /root/.tsung; #定位到日志目录；

index report.html ; #默认显示报告页；

autoindex on; #开启目录浏览功能；

}

使用浏览器访问目录 http://192.168.1.95/log/ ,查看报告.

# a.以下为文字及表格信息:

Main Statistics

Name	highest 10sec mean	lowest 10sec mean	Highest Rate	Mean Rate	Mean	Count
connect	30.91 msec	1.76 msec	657.6 / sec	126.98 / sec	17.01 msec	96002
page	0.16 sec	1.84 msec	461 / sec	98.54 / sec	63.58 msec	74232
request	29.13 msec	13.43 msec	707.9 / sec	98.12 / sec	18.22 msec	74998
session	10mn 2sec	10mn 2sec	240 / sec	166.67 / sec	10mn 2sec	24261

Transactions Statistics

Name	highest 10sec mean	lowest 10sec mean	Highest Rate	Mean Rate	Mean	Count
tr_authenticate	0.18 sec	53.80 msec	236.4 / sec	32.71 / sec	86.19 msec	25000

Network Throughput

Name	Highest Rate	Total
size_rcv	1.95 Mbits/sec	26.60 MB
size_sent	1.05 Mbits/sec	14.53 MB

Counters Statistics

Name	Highest Rate	Mean Rate	Total number
nomatch	235.6 / sec	32.71 / sec	25000
request_noack	461 / sec	98.54 / sec	75000

Name	Max
connected	672
finish_users_count	25000
users	25000
users_count	25000

Errors

Name	Highest Rate	Total number
error_connect_etimedout	0.3 / sec	3
error_connection_closed	432.7 / sec	46767
error_send_einval	0.1 / sec	1

Server monitoring

Name	highest 10sec mean	lowest 10sec mean
cpu:os_mon@tsung_controller@xmpp95	99.18 %	17.86 %
freemem:os_mon@tsung_controller@xmpp95	956.50 MB	387.14 MB
load:os_mon@tsung_controller@xmpp95	5.79	0.33

# b.以下为对应图表信息(更直观,一般先看这个):

4.2.3.4对图标信息进行分析：

根据上面图表可以发现，注册主要集中在前面150秒完成，并成功达到25000用户量，持续连接到约600秒后开始释放，在此期间，CPU一直处于高负荷状态，是需改进方向。

进入数据库确认实际注册量：

mysql -uroot -phmj1314 -e 'select count(*) from ejabberd.users ;'

结果显示为25001，除去admin，刚好2.5万注册用户，表示压测100%成功。

五、总结

根据 3.1.14、4.1.2和4.2.3.4以上测试结果总结为：集群能满足25台以上节点，单节点能满足3万并发，单节点能满足2.5万用户集中注册.判定认为能满足1.2要求的单节点承载2万的压力，项目测试成功！

(备注：由于本公司测试的环境为单金属机上的多虚拟机，性能无法满足实际同时50万真实压测，考虑到与真实生产环境可能不同，真实环境建议根据实际情况再次调整优化)

HMJ589119

关注

0
点赞
踩
4

收藏

觉得还不错? 一键收藏
0
评论
XMPP(ejabberd) 集群部署说明（测试为27台集群）

XMPP(ejabberd) 集群部署说明（测试为27台集群）目录一、项目简介 11.1背景 11.2项目分析 11.3架构设计 1二、系统配置及优化 12.1操作系统要求 12.2基础环境优化 12.3 mysql数据库部署 42.4.redis数据库部署 6三、XMPP集群部署 63.1 集群环境配置 63.2官方下载eja...
复制链接

扫一扫