MongoDB 日志分析工具 mtools

最新推荐文章于 2024-08-24 07:30:30 发布

xiewen99

最新推荐文章于 2024-08-24 07:30:30 发布

阅读量9.5k

点赞数 1

分类专栏：数据库文章标签： mongodb

本文链接：https://blog.csdn.net/xiewen99/article/details/52452328

版权

mtools 是 MongoDB 的一套辅助工具集合，用于解析、过滤、可视化日志，并快速创建测试环境。它包括 mlaunch 用于搭建复制集、分片等，mloginfo 分析日志信息，mlogfilter 过滤日志，mplotqueries 可视化查询分布。本文介绍了 mtools 的使用方法和常见问题解决方案。

摘要由CSDN通过智能技术生成

概述

mtools is a collection of helper scripts to parse and filter MongoDB log files (mongod, mongos), visualize log files and quickly set up complex MongoDB test environments on a local machine.

mtools 是一组非常好用的 MongoDB 日志分析工具
mtools 是由 MongoDB Inc 官方工程师所写，设计之初是为了方便自己的工作，但是随着 MongoDB 用户的增加，越来越多的朋友也开始使用 mtools，也越来越感受到 mtools 带来的便捷。

https://github.com/rueckstiess/mtools

组成

mlogfilter
slices log files by time, merges log files, filters slow queries, finds table scans, shortens log lines, filters by other atributes, convert to JSON
mloginfo
returns info about log file, like start and end time, version, binary, special sections like restarts, connections, distinct view
mplotqueries
visualize logfiles with different types of plots (requires matplotlib)
mlogvis
creates a self-contained html file that shows an interactive visualization in a web browser (as an alternative to mplotqueries)
mlaunch
a script to quickly spin up local test environments, including replica sets and sharded systems (requires pymongo)
mgenerate
generates structured pseudo-random data based on a template for testing and reproduction

安装

安装过程仅供参考

pip 安装

yum -y install glibc-devel python-devel
yum -y install python-pip
yum -y install python-pymongo
pip install mtools

或 setup.py 安装

yum -y install glibc-devel python-devel


wget -O - ftp://ftp.no.sohu.com/pub/software/unix/Mongodb/utils/psutil-4.3.1.tar.gz | tar -zxvf - 
cd psutil-4.3.1
python setup.py install


yum -y install python-{ordereddict,argparse,dateutil,matplotlib,nose,pymongo,psutil}
yum -y install numpy
pip install --upgrade matplotlib


wget -O - ftp://ftp.no.sohu.com/pub/software/unix/Mongodb/utils/mtools-1.2.1.tar.gz | tar -zxvf -
cd mtools-1.2.1
python setup.py install

使用

详细使用说明请参考：https://github.com/rueckstiess/mtools

mlaunch

单机环境下快速搭建MongolDB 测试环境，支持单节点、复制集、分片的架构

初始化复制集

# mlaunch init --replicaset
launching: mongod on port 27017
launching: mongod on port 27018
launching: mongod on port 27019
replica set 'replset' initialized.

# ps -ef |grep mongo
root      2565     1  0 20:11 ?        00:00:00 mongod --replSet replset --dbpath /root/data/replset/rs1/db --logpath /root/data/replset/rs1/mongod.log --port 27017 --logappend --fork
root      2588     1  0 20:11 ?        00:00:00 mongod --replSet replset --dbpath /root/data/replset/rs2/db --logpath /root/data/replset/rs2/mongod.log --port 27018 --logappend --fork
root      2627     1  0 20:11 ?        00:00:00 mongod --replSet replset --dbpath /root/data/replset/rs3/db --logpath /root/data/replset/rs3/mongod.log --port 27019 --logappend --fork
root      3002 38064  0 20:11 pts/0    00:00:00 grep mongo

进程列表

# mlaunch list
PROCESS    PORT     STATUS     PID
mongod     27017    running    2565
mongod     27018    running    2588
mongod     27019    running    2627

停止

# mlaunch stop
3 nodes stopped.
# mlaunch list
PROCESS    PORT     STATUS    PID
mongod     27017    down      -
mongod     27018    down      -
mongod     27019    down      -

启动

# mlaunch start
launching: mongod on port 27017
launching: mongod on port 27018
launching: mongod on port 27019

删除

# mlaunch kill
sent signal 15 to 3 processes.
# mlaunch list
PROCESS    PORT     STATUS    PID
mongod     27017    down      -
mongod     27018    down      -
mongod     27019    down      -
# rm -rf ./data

初始化分片+复制+arbiter

# mlaunch init --sharded 2  --mongos 2 --config 3 --replicaset --nodes 2 --arbiter
launching: mongod on port 27019
launching: mongod on port 27020
launching: mongod on port 27021
launching: mongod on port 27022
launching: mongod on port 27023
launching: mongod on port 27024
launching: config server on port 27025
launching: config server on port 27026
launching: config server on port 27027
replica set 'shard01' initialized.
replica set 'shard02' initialized.
launching: mongos on port 27017
launching: mongos on port 27018
adding shards. can take up to 30 seconds...
# mlaunch list

PROCESS          PORT     STATUS     PID

mongos           27017    running    33593
mongos           27018    running    33680

config server    27025    running    33378
config server    27026    running    33398
config server    27027    running    33418

shard01
    mongod       27019    running    33237
    mongod       27020    running    33260
    mongod       27021    running    33284

shard02
    mongod       27022    running    33309
    mongod       27023    running    33332
    mongod       27024    running    33355

mloginfo

对日志中各类最常出现情况进行统计

基本信息

# mloginfo mongod.log
     source: mongod.log
       host: m202-ubuntu:27017
      start: 2016 Apr 05 06:39:57
        end: 2016 Apr 06 06:27:07
date format: ctime-pre2.4
     length: 58224
     binary: mongod
    version: 2.2.3
    storage: mmapv1

统计客户端连接次数

# mloginfo mongod.log --connections
...

CONNECTIONS
     total opened: 24853
     total closed: 24719
    no unique IPs: 9
socket exceptions: 37

192.0.2.2        opened: 16120     closed: 16113   
127.0.0.1        opened: 2837      closed: 2832    
192.0.2.4        opened: 2677      closed: 2674    
192.0.2.3        opened: 2621      closed: 2620    
192.0.2.8        opened: 133       closed: 107     
192.0.2.5        opened: 126       closed: 122     
192.0.2.7        opened: 122       closed: 98      
192.0.2.10       opened: 110       closed: 74      
192.0.2.6        opened: 107       closed: 79      
...

统计各类信息出现的次数

# mloginfo mongod.log --distinct
...

DISTINCT

   24853    connection accepted from ... # ... ( ... now open)
   24719    end connection ... ( ... now open)
    6137    couldn't connect to ... :
      37    SocketException handling request, closing client connection:
      27    replSet member ... is now in state
      10    replSet member ... is up
       9    closing listening socket:
       9    replSet info ... is down (or slow t