BottledWater-PG安装部署

一 前言

Bottled Water是Confluent公司开发的一款可以将postgresql数据库转换为kafka events的工具。主要用于监视PG的数据增删改的变化，将变化的数据实时推送到kafka消息队列，其他关心数据变化的平台只要监听kafka的消息，就能得到改变的数据。具体参考德哥的博客《实时数据交换平台 - BottledWater-pg with confluent》，本文具体的安装步骤也参考了《Bottled Water: 实时集成postgresql与kafka》。

二 安装准备

安装完成PostgreSQL，参考《Centos7安装PostgreSQL9.6》；
安装完成Kafka，参考《Kafka部署》；
此外还依赖avro-c，Jansson，libcurl，librdkafka

三 安装步骤bottledwater-pg

3.1 安装libjansson

wget http://www.digip.org/jansson/releases/jansson-2.9.tar.bz2
tar -jxvf jansson-2.9.tar.bz2
cd jansson-2.9
./configure --prefix=/home/free/jansson
make
make install

export PKG_CONFIG_PATH=/home/free/jansson/lib/pkgconfig:$PKG_CONFIG_PATH

pkg-config --cflags --libs jansson
-I/home/free/jansson/include  -L/home/free/jansson/lib -ljansson

3.1 安装avro

wget http://mirrors.hust.edu.cn/apache/avro/avro-1.8.1/avro-src-1.8.1.tar.gz

tar -zxvf avro-src-1.8.1.tar.gz

cd avro-src-1.8.1/lang/c
mkdir build
cd build

cmake .. -DCMAKE_INSTALL_PREFIX=/home/free/avro -DCMAKE_BUILD_TYPE=Release -DTHREADSAFE=true
make
make test
make install
export PKG_CONFIG_PATH=/home/free/avro/lib/pkgconfig:$PKG_CONFIG_PATH

在make这一步骤报错如下：

opt/avro-src-1.8.1/lang/c/src/schema.c:31:21: fatal error: jansson.h: No such file or directory
 #include "jansson.h"
                     ^
compilation terminated.
make[2]: *** [src/CMakeFiles/avro-shared.dir/schema.o] Error 1
make[1]: *** [src/CMakeFiles/avro-shared.dir/all] Error 2
make: *** [all] Error 2

这一步骤应该是编译缺少了头文件，先加上去，然后重新编译。

export C_INCLUDE_PATH=/home/free/jansson/include:$C_INCLUDE_PATH

3.3 安装libcurl

wget https://curl.haxx.se/download/curl-7.54.0.tar.gz
tar -zxvf curl-7.54.0.tar.gz
cd curl-7.54.0
 ./configure --prefix=/home/free/curl
make 
make install
export PKG_CONFIG_PATH=/home/free/curl/lib/pkgconfig:$PKG_CONFIG_PATH

3.4 安装librdkafka

git clone https://github.com/edenhill/librdkafka
cd librdkafka
./configure --prefix=/home/free/librdkafka
#假如报错如下：
#[root@bogon librdkafka]# ./configure --prefix=/home/freegis/librdkafka
#-bash: ./configure: Permission denied
#赋予执行权限
#[root@bogon librdkafka]# chmod 777 configure

make -j 32
make install

export PKG_CONFIG_PATH=/home/free/librdkafka/lib/pkgconfig:$PKG_CONFIG_PATH

在make的时候，64位的linux会报错如下：

/bin/ld:librdkafka.lds:1: syntax errer in VERSION script

只需要在Makefile.config里面的WITH_LDS=y这一行注释掉重新make就可以了。

3.5 安装bottledwater-pg

在环境变量中新增如下依赖：

vi /etc/profile
#编辑内容如下
export CURLHOME=/home/freegis/curl
PATH=$PATH:$HOME/.local/bin:$HOME/bin:$GCCHOME/bin:$CURLHOME/bin
# wq!保存
source /etc/profile

下载安装文件并编译部署：

git clone https://github.com/confluentinc/bottledwater-pg

cd bottledwater-pg
#启用postgres的环境变量
source /home/postgres/.bashrc
make
make install

make可能报错如下：

ake[1]: Entering directory `/opt/bottledwater-pg-master/kafka'
gcc bottledwater.c -c -std=c99 -D_POSIX_C_SOURCE=200809L -I../client -I../ext -I/home/postgres/include -I/home/postgres/include/server  -I/home/freegis/avro/include   -DAVRO_1_8    -Wall -Wmissing-prototypes -Wpointer-arith -Wendif-labels -Wmissing-format-attribute -Wformat-security -o bottledwater.o
In file included from table_mapper.h:4:0,
                 from json.h:5,
                 from bottledwater.c:2:
registry.h:4:32: fatal error: librdkafka/rdkafka.h: No such file or directory
 #include <librdkafka/rdkafka.h>
                                ^
compilation terminated.

这种情况仍然是头文件找不到，为了麻烦，把用到的头文件都指定。

export C_INCLUDE_PATH=/home/free/jansson/include:$C_INCLUDE_PATH
export C_INCLUDE_PATH=/home/free/avro/include:$C_INCLUDE_PATH
export C_INCLUDE_PATH=/home/free/librdkafka/include:$C_INCLUDE_PATH
export C_INCLUDE_PATH=/home/free/curl/include:$C_INCLUDE_PATH

继续编译后报错：

/usr/bin/ld: cannot find -lrdkafka

详细如下：

缺失依赖.png

建立依赖软连接：

ln -s /home/free/librdkafka/lib/librdkafka.so /home/postgres/lib/librdkafka.so

检查安装情况

su - postgres
psql
create extension bottledwater;

如果创建扩展报错如下：

mcsas=# create extension bottledwater;
ERROR:  could not load library "/home/postgres/lib/bottledwater.so": libavro.so.23.0.0: cannot open shared object file: No such file or directory

查看下bottledwater.so的依赖：

[postgres@bogon lib]$ ldd bottledwater.so 
    linux-vdso.so.1 =>  (0x00007fffc41fe000)
    libavro.so.23.0.0 => /home/freegis/avro/lib/libavro.so.23.0.0 (0x00007f1f26769000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f1f263a8000)
    libjansson.so.4 => /lib64/libjansson.so.4 (0x00007f1f2619c000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1f25f7f000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f1f26bba000)

发现所有的依赖都正确。
后经大神提示，重启数据库，然后执行扩展。

pg_ctl restart
psql
\c mcsas
create extension bottledwater;
CREATE EXTENSION