gearman mysql redis_利用gearman同步mysql数据到redis

一.Gearman

1.Gearman是一个分发任务的程序框架.

2.体系:

a.client:发送一个job

b.server:找到合适的worker,把job交给该worker

c.worker:处理job

二.安装Gearmand服务

1.Gearman

a.官网

http://gearman.org/

b.官网下载:

https://launchpad.net/gearmand

2.安装依赖库

# yum install -y boost boost-devel gperf libevent-devel libuuid-devel

ps:要求boost版本至少大于1.39,系统之前安装的是1.3;所以需要升级

3.升级boost到1.55.0

a.下载boost_1_55_0.tar.bz2

http://downloads.sourceforge.net/project/boost/boost/1.55.0/boost_1_55_0.tar.bz2?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fboost%2Ffiles%2Fboost%2F1.55.0%2F&ts=1385953406&use_mirror=softlayer-ams

b.解压

c. 安装

./bootstrap.sh

./b2

./b2 install

4.安装Gearman

a.解压

b.编译

./configure --prefix=/usr/local/gearmand --with-boost=/usr/local/boost/ --with-boost-libdir=/usr/local/boost/lib

c.make

出现错误:error: cinttypes: No such file or directory

安装:gcc44等

yum install gcc44 gcc44-c++ libstdc++44-devel -y

在/etc/profile中加入

export CC=/usr/bin/gcc44 or export CC=/usr/bin/gcc

export CXX=/usr/bin/g++44

#source /etc/profile

d.再次make

报错:

error: 'SQLITE_OPEN_READWRITE' was not declared in this scope

e.安装新的sqlite

#wget http://www.sqlite.org/2014/sqlite-autoconf-3080704.tar.gz

#tar zxvf sqlite-autoconf-3080704.tar.gz

#cd sqlite-autoconf-3080704

#./configure --prefix=/usr/local/sqlite

#make && make install

f.再次配置编译安装

#./configure --prefix=/usr/local/gearmand --with-boost=/usr/local/boost/ --with-boost-libdir=/usr/local/boost/lib --with-sqlite3=/usr/local/sqlite

#make

出现错误undefined reference to `sqlite3_prepare_v2'

#ln -s /usr/local/sqlite/lib/libsqlite3.so.0.8.6 /usr/lib64

*如果/usr/lib64下存在libsqlite3.so.0.8.6,将其重命名

三.启动Gearman

1.

/usr/local/gearmand/sbin/gearmand -d -u root -L 127.0.0.1 --log-file=/var/log/gearmand.log

报错:

error while loading shared libraries: libboost_program_options.so.1.55.0: cannot open shared object file: No such file or directory

#vi /etc/ld.so.conf.d/gearman.conf

插入以下内容

/usr/local/boost/lib/

/usr/local/gearman/lib/

执行

#ldconfig

2.参数详解

-b，--backlog= 储备的监听连接数量

-d, --daemon 后台运行

-f, --file-descriptors= 文件描述符的数量

-h, --help 帮助

-j, --job-retries= 在ob server移除不可用job之前运行的次数，防止不断运行导致其他可用worker崩溃。默认没有限制

-l, -log-file= 日志文件存放位置(默认记录最简单日志)

-L, --listen= 监听的IP，默认全部接受

-p, --port= 指定监听端口

-P, --pid-file= 指定进程ID写入位置

-r, --protocol= 加载协议模块

-q, --queue-type= 指定持久化队列

-t, --threads= 使用的I/9线程数量。默认为0

-u, --user= 启动后，切换到指定用户

-v, --verbose 增加一级详细程度

-V, --version 显示版本信息

3.监控

#watch -n 1 "(echo status; sleep 0.1) | nc 127.0.0.1 4730"

四.安装php-gearman

1.wget http://pecl.php.NET/get/gearman-1.1.1.tgz

2.解压

3.#/usr/local/php/bin/phpize

4.#./configure --with-gearman=/usr/local/gearmand/ --with-php-config=/usr/local/php/bin/php-config

5.make && make install

6.在php.ini中加入

extension=gearman.so

7.用#php -i | grep gearman或者php -m | grep gearman来验证

五.安装lib_mysqludf_json 和 gearman-mysql-udf

1.安装lib_mysqludf_json

wget https://github.com/mysqludf/lib_mysqludf_json/archive/master.zip

mv master master.zip

unzip master.zip

cd lib_mysqludf_json-master

rm -rf lib_mysqludf_json.so

gcc $(/usr/local/mysql56-server/bin/mysql_config --cflags) -shared -fPIC -o lib_mysqludf_json.so lib_mysqludf_json.c

2.拷贝lib_mysqludf_json.so到mysql的plugin目录下

3.安装gearman-mysql-udf

wget https://launchpad.net/gearman-mysql-udf/trunk/0.6/+download/gearman-mysql-udf-0.6.tar.gz

tar xf gearman-mysql-udf-0.6.tar.gz -C ./

cd gearman-mysql-udf-0.6

./configure --with-mysql=/usr/local/mysql56-server/bin/mysql_config --libdir=/usr/local/mysql56-server/lib/plugin/

make && make install

报错:

configure: error: At least version 0.33 of libgearman is required for gearman-mysql-udf

找不到libgearman,指定路径,加入--with-libgearman-prefix

./configure --with-mysql=/usr/local/mysql56-server/bin/mysql_config --libdir=/usr/local/mysql56-server/lib/plugin/ --with-libgearman-prefix=/usr/local/gearmand

六.创建function

mysql> CREATE FUNCTION json_object RETURNS STRING SONAME 'lib_mysqludf_json.so';

mysql> CREATE FUNCTION gman_do_background RETURNS STRING SONAME 'libgearman_mysql_udf.so';

mysql> CREATE FUNCTION gman_servers_set RETURNS STRING SONAME 'libgearman_mysql_udf.so';

七.在mysql中设置Gearman Server

1.

SELECT gman_servers_set('127.0.0.1:4730');

2.建立一个触发器(更新记录时将数据更新到redis)

CREATE TRIGGER `test_mysql_redis` AFTER UPDATE ON `test` FOR EACH ROW SET @ret=gman_do_background('syncToRedis', json_object(NEW.id as `id`, NEW.name as `name`));

八.查看job及更新数据到redis

1.

watch -n 1 "(echo status; sleep 0.1) | nc 127.0.0.1 4730"

syncToRedis 1 0 0

syncToRedis后面三列的意思是:

1个正在运行的job, 队列为空, 0个可用的woker

2.编写worker, gearman_workder.php

$worker = new GearmanWorker();

$worker->addServer();

$worker->addFunction('syncToRedis', 'syncToRedis');

$redis = new Redis();

$redis->connect('127.0.0.1', '6379');

$redis->select(9);

while($worker->work());

function syncToRedis($job)

{

global $redis;

$workString = $job->workload();

$work = json_decode($workString);

if(!isset($work->id)){

return false;

}

$redis->set($work->id, $workString);

}

然后运行gearman_workder.php,去redis中验证,看数据是否有同步.