一、MogileFS简介
MogileFS是一个开源的分布式文件存储系统，由LiveJournal旗下的Danga Interactive公司开发。Danga团队开发了包括 Memcached、MogileFS、Perlbal 等多个知名的开源项目。
目前使用MogileFS 的公司非常多，如日本排名先前的几个互联公司及国内的yupoo(又拍)、digg、豆瓣、1号店、大众点评、搜狗和安居客等，分别为所在的组织或公司管理着海量的图片。
1、Mogilefs特性
工作于应用层：http,nfs
无单点故障
自动完成文件复制
传输无需特殊协议
名称空间（完成复制）
不共享任何数据
2、MogileFS组件
tracker：追踪器，追踪元数据（不存储元数据）， http协议
mogilefsd(守护进程），它的主要职责包括：
replication：节点间文件的复制
deletion：删除文件
queryworker：响应客户请求的文件元数据访问请求
reaper：在存储失败后将文件复制请求重新放置于队列中
monitor：监测主机和设备的健康状态
database：存储元数据
一般使用mysql也可以使用pgsql，建议使用冗余方案以保证其可用性
mogilefs专门提供了数据管理（初始化）工具mogdbsetup;
storage：存储数据， http协议，nfs协议
mogstored(进程名），一个准备好的mogstored节点可通过mogadm命令添加至现在的集群中，存储节点需要定义“设备”用作存储空间，每个设备在当前集群中都需要通过唯一的DevID来标识
client：客户端用于与mogilefs建立通信，完成数据存取 #http协议
注意：
分布式系统应该具有机架感知能力，将冗余数据存放在不同机架上的服务器
MogileFS工作图解：
3、常用术语
Domain：域 name space 命名空间 相当于一个目录
一个mogilefs可以有多个Dom，
用来存放不同文件，同一个Domain内key必须唯一，同mogilefs内，key可以相同
class：最小复制单元，文件属性管理
定义文件存储在不同设备上的份数
定位文件：Domain+fid（文件id）
二、安装Mogilefs
1、实验环境
CentOS6.5 x86_64 , 时间同步，epel源
node1：192.168.10.1 tracker，database
node3：192.168.10.3 storage
node4：192.168.10.4 storage
安装包介绍：
[root@Node1 ~]# ls mogilefs/ MogileFS-Server-2.46-2.el6.noarch.rpm #基础组件 Perlbal-doc-1.78-1.el6.noarch.rpm MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm #tracker组件 perl-MogileFS-Client-1.14-1.el6.noarch.rpm #客户端组件 MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm #storage组件 perl-Net-Netmask-1.9015-8.el6.noarch.rpm #基础组件 MogileFS-Utils-2.19-1.el6.noarch.rpm #工具集 perl-Perlbal-1.78-1.el6.noarch.rpm #基础组件 Perlbal-1.78-1.el6.noarch.rpm
2、安装步骤
1）node4上安装mysql-server并配置，安装mogilefsd并配置，初始化数据库
[root@Node1 ~]# yum install mysql-server #在/etc/my.cnf配置文件中[mysqld]段添加skip_name_resolve，跳过mysql对外部连接进行DNS解析 ；在MySQL的授权表中就不能使用主机名了，只能使用IP [root@Node1 mogilefs]# yum install MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm perl-Perlbal-1.78-1.el6.noarch.rpm perl-Net-Netmask-1.9015-8.el6.noarch.rpm [root@Node1 ~]# rpm -ql MogileFS-Server-mogilefsd /etc/mogilefs/mogilefsd.conf /etc/rc.d/init.d/mogilefsd /usr/bin/mogdbsetup /usr/bin/mogilefsd /usr/share/man/man1/mogilefsd.1.gz /usr/share/man/man3/MogileFS::Checksum.3pm.gz /usr/share/perl5/vendor_perl/MogileFS/Checksum.pm /var/run/mogilefsd
数据库化数据库：
[root@Node1 ~]# mysql mysql> GRANT ALL ON *.* TO 'root'@'192.168.10.%' IDENTIFIED BY 'magedu' WITH GRANT OPTION; Query OK, 0 rows affected (0.00 sec) mysql> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec) [root@Node1 mogilefs]# mogdbsetup --help Usage: mogdbsetup [opts] Options: Default Description ============ =========================================== --verbose <off> Be verbose about what's happening. --dbhost= localhost hostname or IP to database server. --dbport= dbd default port number to database server. --dbname= mogilefs database name to create/upgrade. --dbrootuser= root Database administrator username. Only needed for initial setup, not subsequent upgrades. --dbrootpass= <blank> Database administrator password. Only needed for initial setup, not subsequent upgrades. --dbuser= mogile Regular database user to create and/or use for MogileFS database. This is what the mogilefsd trackers connect as. --dbpass= <blank> You should change this, especially if your database servers are accessible to other users on the network. But they shouldn't be if you're running MogileFS, because MogileFS assumes your network is closed. --type= MySQL Which MogileFS::Store implementation to use. Available: MySQL, Postgres --yes Run without questions. [root@Node1 ~]# mogdbsetup --dbhost=192.168.10.1 --dbrootuser=root --dbrootpass=magedu --dbname=mogdb --dbuser=moguser --dbpass=mogpass This will attempt to setup or upgrade your MogileFS database. It won't destroy existing data. Run with --help for more information. Run with --yes to shut up these prompts. Continue? [N/y]: y Create/Upgrade database name 'mogdb'? [Y/n]: y Grant all privileges to user 'moguser', connecting from anywhere, to the mogilefs database 'mogdb'? [Y/n]: y mysql> SHOW DATABASES; +--------------------+ | Database | +--------------------+ | information_schema | | mogdb | | mysql | | test | +--------------------+ 4 rows in set (0.00 sec) mysql>
修改/etc/mogilefs/mogilefsd.conf：
[root@Node1 ~]# cd /etc/mogilefs/ [root@Node1 mogilefs]# ls mogilefsd.conf [root@Node1 mogilefs]# cp mogilefsd.conf mogilefsd.conf.bak [root@Node1 mogilefs]# ls mogilefsd.conf mogilefsd.conf.bak [root@Node1 mogilefs]# grep -v '^#\|^$' mogilefsd.conf #修改前的配置文件 daemonize = 1 pidfile = /var/run/mogilefsd/mogilefsd.pid db_dsn = DBI:mysql:mogilefs:host=127.0.0.1 db_user = username db_pass = password listen = 127.0.0.1:7001 conf_port = 7001 query_jobs = 10 delete_jobs = 1 replicate_jobs = 5 reaper_jobs = 1 [root@Node1 mogilefs]# grep -v '^#\|^$' mogilefsd.conf #修改后的配置文件 daemonize = 1 pidfile = /var/run/mogilefsd/mogilefsd.pid db_dsn = DBI:mysql:mogdb:host=192.168.10.1 db_user = moguser db_pass = mogpass listen = 0.0.0.0:7001 conf_port = 7001 query_jobs = 10 delete_jobs = 3 replicate_jobs = 5 reaper_jobs = 1
2）node3和node4安装mogstored并配置，需要额外安装perl-IO-AIO，
这里node3和node4上的操作都是一样的
[root@Node3 mogilefs]# yum install MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm perl-Perlbal-1.78-1.el6.noarch.rpm perl-Net-Netmask-1.9015-8.el6.noarch.rpm MogileFS-Server-2.46-2.el6.noarch.rpm MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm #被MogileFS-Server所依赖 [root@Node3 mogilefs]# yum install perl-IO-AIO [root@Node3 mogilefs]# rpm -ql MogileFS-Server-mogstored /etc/mogilefs/mogstored.conf /etc/rc.d/init.d/mogstored /usr/bin/mogautomount /usr/bin/mogstored /usr/share/man/man1/mogautomount.1.gz /usr/share/man/man1/mogstored.1.gz /usr/share/perl5/vendor_perl/Mogstored/ChildProcess
创建一个新的分区用来做mogstored：
[root@Node3 ~]# df -TH
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda2 ext4 20G 2.3G 16G 13% /
tmpfs tmpfs 254M 0 254M 0% /dev/shm
/dev/sda1 ext4 508M 34M 448M 7% /boot
[root@Node3 mogilefs]# fdisk -l
Disk /dev/sda: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00037190
Device Boot Start End Blocks Id System
/dev/sda1 * 1 64 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 64 2418 18902016 83 Linux
/dev/sda3 2418 2673 2048000 82 Linux swap / Solaris
[root@Node3 mogilefs]# cat /proc/partitions
major minor #blocks name
8 0 41943040 sda
8 1 512000 sda1
8 2 18902016 sda2
8 3 2048000 sda3
[root@Node3 mogilefs]# fdisk /dev/sda
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n
Command action
e extended
p primary partition (1-4)
e
Selected partition 4
First cylinder (2673-5221, default 2673):
Using default value 2673
Last cylinder, +cylinders or +size{K,M,G} (2673-5221, default 5221): +10G
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
[root@Node3 mogilefs]# partx -a /dev/sda
BLKPG: Device or resource busy
error adding partition 1
BLKPG: Device or resource busy
error adding partition 2
BLKPG: Device or resource busy
error adding partition 3
[root@Node3 mogilefs]# partx -a /dev/sda
BLKPG: Device or resource busy
error adding partition 1
BLKPG: Device or resource busy
error adding partition 2
BLKPG: Device or resource busy
error adding partition 3
BLKPG: Device or resource busy
error adding partition 4
[root@Node3 mogilefs]# cat /proc/partitions
major minor #blocks name
8 0 41943040 sda
8 1 512000 sda1
8 2 18902016 sda2
8 3 2048000 sda3
8 4 10490245 sda4
[root@Node3 ~]# mke2fs -t ext4 /dev/sda4 #格式化
[root@Node3 ~]# mkdir /mogilefs
[root@Node3 ~]# chown mogilefs. -R /mogilefs
[root@Node3 ~]# vi /etc/fstab #开机自动挂载
[root@Node3 ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Fri Aug 5 17:32:21 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=e0c0bc25-62e5-4896-8749-fce761bc3af7 / ext4 defaults 1 1
UUID=970e070d-b49f-439b-8bbb-11ef49ee95ce /boot ext4 defaults 1 2
UUID=7e6bc97d-be6d-4ac9-bc26-8be3d80e715d swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/sda4 /mogstore ext4 defaults 0 0
[root@Node3 ~]# mount -a
[root@Node3 ~]# mount
/dev/sda2 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/sda4 on /mogstore type ext4 (rw)
[root@Node3 ~]# ls /mogstore
lost+found
[root@Node3 ~]# df -TH
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda2 ext4 20G 2.6G 16G 14% /
tmpfs tmpfs 254M 0 254M 0% /dev/shm
/dev/sda1 ext4 508M 34M 448M 7% /boot
/dev/sda4 ext4 11G 158M 9.9G 2% /mogstore
[root@Node3 ~]# cd /mogstore/
[root@Node3 mogstore]# ls
lost+found
[root@Node3 mogstore]# mkdir dev1 #新建一个设备dev1，
[root@Node3 mogstore]# ls
dev1 lost+found
[root@Node4 ~]# cd /mogstore/
[root@Node4 mogstore]# ls
lost+found
[root@Node4 mogstore]# mkdir dev2
[root@Node4 mogstore]# ls
dev2 lost+found
修改配置文件：/etc/mogilefs/mogstored.conf
[root@Node3 ~]# cat /etc/mogilefs/mogstored.conf maxconns = 10000 httplisten = 0.0.0.0:7500 mgmtlisten = 0.0.0.0:7501 docroot = /mogstore
启动mogstored：
[root@Node3 mogilefs]# service mogstored start Starting mogstored [ OK ]
3）node1上安装客户端工具
[root@Node1 mogilefs]# yum install perl-MogileFS-Client-1.14-1.el6.noarch.rpm #被MogileFS-Utils所依赖
MogileFS-Utils-2.19-1.el6.noarch.rpm
[root@Node1 mogilefs]# rpm -ql MogileFS-Utils
/usr/bin/mogadm #管理工具
/usr/bin/mogdelete
/usr/bin/mogfetch
/usr/bin/mogfiledebug
/usr/bin/mogfileinfo
/usr/bin/moglistfids
/usr/bin/moglistkeys
/usr/bin/mogrename
/usr/bin/mogstats
/usr/bin/mogtool
/usr/bin/mogupload
/usr/share/man/man1/mogadm.1.gz
[root@Node1 mogilefs]# mogadm --trackers=192.168.10.1:7001 check #检查状态，服务器是本机可以省略--trackers
Checking trackers...
192.168.10.1:7001 ... OK
Checking hosts...
No devices found on tracker(s). #没有设备
[root@Node1 mogilefs]# mogadm --trackers=192.168.10.1:7001 stats
mogadm stats is deprecated by new 'mogstats' utility
[root@Node1 mogilefs]# mogadm stats
mogadm stats is deprecated by new 'mogstats' utility
[root@Node1 mogilefs]# mogstats
Fetching statistics... (all)
Can't connect to data source '' because I can't work out what driver to use (it doesn't seem to contain a 'dbi:driver:' prefix and the DBI_DRIVER env var is not set) at /usr/bin/mogstats line 312
[root@Node1 mogilefs]# mogstats --help
Usage:
mogstats --db_dsn="DBI:mysql:mfs（库名）:host=mfshost" --db_user="mfs"
--db_pass="mfs" --verbose --stats="devices,files"
mogstats --stats="all"
mogstats [all options in ~/.mogilefs.conf]
valid stats: all, delete-queue, devices, domains, fids, files, general-queues, replication,
[root@Node1 mogilefs]# mogstats --db_dsn="DBI:mysql:mogdb:host=192.168.10.1" --db_user="moguser" --db_pass="mogpass" --stats="all"
Fetching statistics... (all)
Statistics for devices...
device host files status
---------- ---------------- ------------ ----------
---------- ---------------- ------------ ----------
Statistics for file ids...
Max file id: none
Statistics for files...
domain class files size (m) fullsize (m)
-------------------- ----------- ---------- ----------- -------------
-------------------- ----------- ---------- ----------- -------------
Statistics for replication...
domain class devcount files
-------------------- ----------- ---------- ----------
-------------------- ----------- ---------- ----------
Statistics for replication queue...
status count
-------------------- ------------
-------------------- ------------
Statistics for delete queue...
status count
-------------------- ------------
-------------------- ------------
Statistics for general queues...
queue status count
--------------- -------------------- ------------
--------------- -------------------- ------------
done
[root@Node1 ~]# mogadm host list
[root@Node1 ~]# mogadm host
Help for 'host' command:
(enter any command prefix, leaving off options, for further help)
mogadm host add <hostname> [opts] Add a host to MogileFS.
mogadm host delete <hostname> Delete a host.
mogadm host list List all hosts.
mogadm host mark <hostname> <status> Change the status of a host. (equivalent to 'modify --status')
mogadm host modify <hostname> [opts] Modify a host's properties.
[root@Node1 ~]# mogadm host add
ERROR: Missing argument 'hostname'
Help for 'host-add' command:
mogadm host add <hostname> [opts] Add a host to MogileFS.
<hostname> Hostname of machine
--altip=s Alternate IP that is machine is reachable from
--altmask=s Netmask which, when matches client, uses alt IP
--getport=i Alternate HTTP port serving readonly traffic
--ip=s IP address of machine.
--port=i HTTP port of mogstored
--status=s One of {alive,down}. Default 'down'.
[root@Node1 ~]# mogadm host add 192.168.10.3 --ip=192.168.10.3 #添加主机
[root@Node1 ~]# mogadm host list
192.168.10.3 [1]: down
IP: 192.168.10.3:7500
[root@Node1 ~]# mogadm host mark 192.168.10.3 alive
[root@Node1 ~]# mogadm host list
192.168.10.3 [1]: alive
IP: 192.168.10.3:7500
[root@Node1 ~]# mogadm host add 192.168.10.4 --ip=192.168.10.4 --status=alive
[root@Node1 ~]# mogadm host list
192.168.10.3 [1]: alive
IP: 192.168.10.3:7500
192.168.10.4 [2]: alive
IP: 192.168.10.4:7500
[root@Node1 ~]# mogadm device add 192.168.10.3 1 #添加设备，需要在相应的节点目录下创建目录
[root@Node1 ~]# mogadm device add 192.168.10.3 2
[root@Node1 ~]# mogadm device list
192.168.10.3 [1]: alive
used(G) free(G) total(G) weight(%)
dev1: alive 0.146 9.200 9.347 100
192.168.10.4 [2]: alive
used(G) free(G) total(G) weight(%)
dev2: alive 0.146 9.200 9.347 100
[root@Node1 ~]# mogadm domain add p_w_picpaths #添加域
[root@Node1 ~]# mogadm domain add confiles
[root@Node1 ~]# mogadm domain list
domain class mindevcount replpolicy hashtype
-------------------- -------------------- ------------- ------------ -------
confiles default 2 MultipleHosts() NONE
p_w_picpaths default 2 MultipleHosts() NONE
[root@Node1 ~]# mogadm class add p_w_picpaths nc #添加类
[root@Node1 ~]# mogadm class list
domain class mindevcount replpolicy hashtype
-------------------- -------------------- ------------- ------------ -------
confiles default 2 MultipleHosts() NONE
p_w_picpaths default 2 MultipleHosts() NONE
p_w_picpaths nc 2 MultipleHosts() NONE
mindevcount：最小复制文件的份数
replpolicy ：复制类型
hashtype：采用的hash的类型
MogileFS就安装配置完成了，就可以通过调用其API开始使用了，就是开发的事咯；
我们这里就使用mogupload命令上传文件测试一下：
[root@Node1 ~]# mogupload Usage: /usr/bin/mogupload --trackers=host --domain=foo --key='/hello.jpg' --file='./hello.jpg' [root@Node1 ~]# mogupload --trackers=192.168.10.1 --domain=confiles --key='/1.conf' --file="/etc/fstab" #key上传后的访问路径，file是要上传的文件现在的路径 [root@Node1 ~]# mogfileinfo --trackers=192.168.10.1 --domain=confiles --key="/1.conf" - file: /1.conf #查看文件存储的信息 class: default devcount: 2 domain: confiles fid: 2 key: /1.conf length: 805 - http://192.168.10.3:7500/dev1/0/000/000/0000000002.fid - http://192.168.10.4:7500/dev2/0/000/000/0000000002.fid
验证：
mogfetch 下载文件：
[root@Node1 ~]# mogfetch Usage: /usr/bin/mogfetch --trackers=host --domain=foo --key='/hello.jpg' --file='./output' [root@Node1 ~]# mogfetch --trackers=192.168.10.1 --domain=confiles --key="/1.conf" --file=/root/1.confs
三、使用Nginx实现代理访问MOgileFS
1、编译安装Nginx
[root@node1 mogile]# tar xf nginx-1.8.0.tar.gz -C /usr/src [root@node1 mogile]# tar xf nginx_mogilefs_module-1.0.4.tar.gz -C /usr/src [root@node1 src]# cd nginx-1.8.0/ [root@node1 nginx-1.8.0]# useradd nginx [root@node1 nginx-1.8.0]#./configure \ --prefix=/usr \ --sbin-path=/usr/sbin/nginx \ --conf-path=/etc/nginx/nginx.conf \ --error-log-path=/var/log/nginx/error.log \ --http-log-path=/var/log/nginx/access.log \ --pid-path=/var/run/nginx/nginx.pid \ --lock-path=/var/lock/nginx.lock \ --user=nginx \ --group=nginx \ --with-http_ssl_module \ --with-http_flv_module \ --with-http_stub_status_module \ --with-http_gzip_static_module \ --http-client-body-temp-path=/var/tmp/nginx/client/ \ --http-proxy-temp-path=/var/tmp/nginx/proxy/ \ --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ \ --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \ --http-scgi-temp-path=/var/tmp/nginx/scgi \ --with-pcre \ --with-debug \ --add-module=..//nginx_mogilefs_module-1.0.4/ #添加mogilefs模块 [root@node1 nginx-1.8.0]# make;make install
nginx的mogilefs的第三方模块使用地址：http://www.grid.net.ru/nginx/mogilefs.en.html
2、添加服务脚本
3、编辑nginx配置文件，使其支持mogilefs
[root@node1 ~]# vim /etc/nginx/nginx.conf
location /confiles/ {
mogilefs_tracker 192.168.10.1:7001;
mogilefs_domain confiles;
mogilefs_pass {
proxy_pass $mogilefs_path;
proxy_hide_header Content-Type;
proxy_buffering off;
}
}
4、测试nginx配置语法，并启动nginx
[root@node4 ~]# nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful [root@node4 ~]# /etc/init.d/nginx start
5、打开浏览器访问测试
转载于:https://blog.51cto.com/xiexiaojun/1854697
扫一扫
1450
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
