GreenPlum 4节点集群模式安装 （Master*1，Standby*1，Segment*2）

一、环境说明

主机名	IP	操作系统版本	GreenPlum角色
hadoop02	192.168.56.202	CentOS release 6.4	Master DW
hadoop03	192.168.56.203	CentOS release 6.4	Standby DW
hadoop04	192.168.56.204	CentOS release 6.4	Segment DW
hadoop07	192.168.56.207	CentOS release 6.4	Segment DW

GreenPlum版本：greenplum-db-4.3.8.1-build-1-RHEL5-x86_64.zip

 

二、操作系统配置

1．关闭SELinux, IPtables

[root@hadoop02 ~]# service iptables stop
[root@hadoop02 ~]# chkconfig iptables off
[root@hadoop02 ~]# service iptables status
iptables: Firewall is not running.

setenforce 0命令临时关闭SELINUX。要永久关闭SELINUX，需修改/etc/selinux/config配置文件，修改配置为SELINUX=disabled。

[root@hadoop02 ~]# sestatus
SELinux status:                 disabled

2．系统参数配置

将/etc/sysctl.conf文件的内容修改为如下内容，重启生效(或执行sysctl -p生效)

 

kernel.shmmax = 500000000

kernel.shmmni = 4096

kernel.shmall = 4000000000

kernel.sem = 250 512000 100 2048

kernel.sysrq = 1

kernel.core_uses_pid = 1

kernel.msgmnb = 65536

kernel.msgmax = 65536

kernel.msgmni = 2048

net.ipv4.tcp_syncookies = 1

net.ipv4.ip_forward = 0

net.ipv4.conf.default.accept_source_route = 0

net.ipv4.tcp_tw_recycle = 1

net.ipv4.tcp_max_syn_backlog = 4096

net.ipv4.conf.all.arp_filter = 1

net.ipv4.ip_local_port_range = 1025 65535

net.core.netdev_max_backlog = 10000

net.core.rmem_max = 2097152

net.core.wmem_max = 2097152

vm.overcommit_memory = 2

 

在/etc/security/limits.conf配置文件末尾处增加如下内容

* soft nofile 65536

* hard nofile 65536

* soft nproc 131072

* hard nproc 131072

* soft core unlimited

 

注意：对于RedHat / CentOS 6.x系统，还需要将/etc/security/limits.d/90-nproc.conf文件中

的1024修改为131072。

 

3．禁用OOM

echo 0 > /proc/sys/vm/panic_on_oom

 

三、存储配置

1．XFS文件系统

GP建议使用下面的挂载参数：

rw,noatime,inode64,allocsize=16m

 

创建/data目录

 

比如，挂载XFS格式的设备/dev/sdb到目录/data，/etc/fstab中的配置如下：

/dev/sdb /data xfs rw,noatime,inode64,allocsize=16m 1 1

 

使用XFS文件系统，需安装相应的rpm软件包，并对磁盘设备进行格式化：

# rpm -ivh xfsprogs-2.9.4-4.el5.x86_64.rpm

# mkfs.xfs -f /dev/sdb

 

2．Linux磁盘I/O调度器配置

Linux磁盘I/O调度器对磁盘的访问支持不同的策略，默认的为CFQ，GP建议设置为deadline

[root@hadoop02 data]# cat /sys/block/sdd/queue/scheduler

noop anticipatory deadline [cfq]

 

如上显示调度策略为CFQ，接下来我们需要将其修改为deadline

修改完成后如下显示：

[root@hadoop02 data]# cat /boot/grub/menu.lst

# grub.conf generated by anaconda

#

# Note that you do not have to rerun grub after making changes to this file

# NOTICE:  You have a /boot partition.  This means that

#          all kernel and initrd paths are relative to /boot/, eg.

#          root (hd0,0)

#          kernel /vmlinuz-version ro root=/dev/sda3

#          initrd /initrd-[generic-]version.img

#boot=/dev/sda

default=0

timeout=5

splashimage=(hd0,0)/grub/splash.xpm.gz

hiddenmenu

title CentOS (2.6.32-358.el6.x86_64)

root (hd0,0)

kernel /vmlinuz-2.6.32-358.el6.x86_64 ro root=UUID=d6909812-6e1c-466f-81fc-24c5bf924b52 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quietelevator=deadline

initrd /initramfs-2.6.32-358.el6.x86_64.img

 

红色为添加的内容。

reboot服务器使参数生效。

验证修改：

[root@hadoop02 ~]# cat /sys/block/sdd/queue/scheduler

noop anticipatory [deadline] cfq

修改成功。

 

3．配置read-ahead

查看：

[root@hadoop03 ~]# /sbin/blockdev --getra /dev/sdb

256

修改：

[root@hadoop03 ~]# /sbin/blockdev --setra 16384 /dev/sdb

验证：

[root@hadoop03 ~]# /sbin/blockdev --getra /dev/sdb

16384

 

永久生效需要将/sbin/blockdev --setra 16384 /dev/sdb追加到/etc/rc.d/rc.local文件末尾。

 

4．关闭Disable Transparent Huge Pages (THP)

修改/boot/grub/grub.conf为如下内容

# grub.conf generated by anaconda

#

# Note that you do not have to rerun grub after making changes to this file

# NOTICE:  You have a /boot partition.  This means that

#          all kernel and initrd paths are relative to /boot/, eg.

#          root (hd0,0)

#          kernel /vmlinuz-version ro root=/dev/sda3

#          initrd /initrd-[generic-]version.img

#boot=/dev/sda

default=0

timeout=5

splashimage=(hd0,0)/grub/splash.xpm.gz

hiddenmenu

title CentOS (2.6.32-358.el6.x86_64)

root (hd0,0)

kernel /vmlinuz-2.6.32-358.el6.x86_64 ro root=UUID=d6909812-6e1c-466f-81fc-24c5bf924b52 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet elevator=deadlinetransparent_hugepage=never

initrd /initramfs-2.6.32-358.el6.x86_64.img

 

红色为添加的内容。

 

四、GreenPlum安装

1．上传安装文件到master节点

[root@hadoop02 ~]# ls greenplum-db-4.3.8.1-build-1-RHEL5-x86_64.bin

greenplum-db-4.3.8.1-build-1-RHEL5-x86_64.bin

2．运行.bin文件

根据命令行提示安装GP

 

I HAVE READ AND AGREE TO THE TERMS OF THE ABOVE PIVOTAL SOFTWARE

LICENSE AGREEMENT.

 

 

*******************************************************************************

Do you accept the Pivotal Database license agreement? [yes|no]

*******************************************************************************

 

yes           

 

*******************************************************************************

Provide the installation path for Greenplum Database or press ENTER to

accept the default installation path: /usr/local/greenplum-db-4.3.8.1

*******************************************************************************

 

 

 

*******************************************************************************

Install Greenplum Database into </usr/local/greenplum-db-4.3.8.1>? [yes|no]

*******************************************************************************

 

yes

 

*******************************************************************************

/usr/local/greenplum-db-4.3.8.1 does not exist.

Create /usr/local/greenplum-db-4.3.8.1 ? [yes|no]

(Selecting no will exit the installer)

*******************************************************************************

 

yes

 

Extracting product to /usr/local/greenplum-db-4.3.8.1

 

*******************************************************************************

Installation complete.

Greenplum Database is installed in /usr/local/greenplum-db-4.3.8.1

 

Pivotal Greenplum documentation is available

for download at http://docs.gopivotal.com/gpdb

*******************************************************************************

 

 

3．各个节点创建GreenPlum gpadmin用户

创建hostfile_exkeys文件，其中包含所有主机名。

[root@hadoop02 ~]# cat hostfile_exkeys

hadoop02

hadoop03

hadoop04

hadoop07

 

确定host文件中对这些主机名进行了IP映射

[root@hadoop02 ~]# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.56.201 hadoop01

192.168.56.202 hadoop02

192.168.56.203 hadoop03

192.168.56.204 hadoop04

192.168.56.205 hadoop05

192.168.56.206 hadoop06

192.168.56.207 hadoop07

 

在所有节点创建gpadmin用户

gpseginstall -f hostfile_exkeys -u gpadmin -p gpadmin

 

输出如下：

[root@hadoop02 greenplum-db]# gpseginstall -f hostfile_exkeys -u gpadmin -p gpadmin

20160905:13:51:33:002733 gpseginstall:hadoop02:root-[INFO]:-Installation Info:

link_name greenplum-db

binary_path /usr/local/greenplum-db-4.3.8.1

binary_dir_location /usr/local

binary_dir_name greenplum-db-4.3.8.1

20160905:13:51:33:002733 gpseginstall:hadoop02:root-[INFO]:-check cluster password access

20160905:13:51:33:002733 gpseginstall:hadoop02:root-[INFO]:-de-duplicate hostnames

20160905:13:51:33:002733 gpseginstall:hadoop02:root-[INFO]:-master hostname: hadoop02

20160905:13:51:34:002733 gpseginstall:hadoop02:root-[INFO]:-check for user gpadmin on cluster

20160905:13:51:34:002733 gpseginstall:hadoop02:root-[INFO]:-add user gpadmin on master

20160905:13:51:34:002733 gpseginstall:hadoop02:root-[INFO]:-add user gpadmin on cluster

20160905:13:51:34:002733 gpseginstall:hadoop02:root-[INFO]:-chown -R gpadmin:gpadmin /usr/local/greenplum-db

20160905:13:51:34:002733 gpseginstall:hadoop02:root-[INFO]:-chown -R gpadmin:gpadmin /usr/local/greenplum-db-4.3.8.1

20160905:13:51:34:002733 gpseginstall:hadoop02:root-[INFO]:-rm -f /usr/local/greenplum-db-4.3.8.1.tar; rm -f /usr/local/greenplum-db-4.3.8.1.tar.gz

20160905:13:51:34:002733 gpseginstall:hadoop02:root-[INFO]:-cd /usr/local; tar cf greenplum-db-4.3.8.1.tar greenplum-db-4.3.8.1

20160905:13:51:35:002733 gpseginstall:hadoop02:root-[INFO]:-gzip /usr/local/greenplum-db-4.3.8.1.tar

20160905:13:51:58:002733 gpseginstall:hadoop02:root-[INFO]:-remote command: mkdir -p /usr/local

20160905:13:51:58:002733 gpseginstall:hadoop02:root-[INFO]:-remote command: rm -rf /usr/local/greenplum-db-4.3.8.1

20160905:13:51:59:002733 gpseginstall:hadoop02:root-[INFO]:-scp software to remote location

20160905:13:52:06:002733 gpseginstall:hadoop02:root-[INFO]:-remote command: gzip -f -d /usr/local/greenplum-db-4.3.8.1.tar.gz

20160905:13:52:30:002733 gpseginstall:hadoop02:root-[INFO]:-md5 check on remote location

20160905:13:52:31:002733 gpseginstall:hadoop02:root-[INFO]:-remote command: cd /usr/local; tar xf greenplum-db-4.3.8.1.tar

20160905:13:53:13:002733 gpseginstall:hadoop02:root-[INFO]:-remote command: rm -f /usr/local/greenplum-db-4.3.8.1.tar

20160905:13:53:16:002733 gpseginstall:hadoop02:root-[INFO]:-remote command: cd /usr/local; rm -f greenplum-db; ln -fs greenplum-db-4.3.8.1 greenplum-db

20160905:13:53:18:002733 gpseginstall:hadoop02:root-[INFO]:-remote command: chown -R gpadmin:gpadmin /usr/local/greenplum-db

20160905:13:53:19:002733 gpseginstall:hadoop02:root-[INFO]:-remote command: chown -R gpadmin:gpadmin /usr/local/greenplum-db-4.3.8.1

20160905:13:53:23:002733 gpseginstall:hadoop02:root-[INFO]:-rm -f /usr/local/greenplum-db-4.3.8.1.tar.gz

20160905:13:53:23:002733 gpseginstall:hadoop02:root-[INFO]:-Changing system passwords ...

20160905:13:53:24:002733 gpseginstall:hadoop02:root-[INFO]:-exchange ssh keys for user root

20160905:13:53:25:002733 gpseginstall:hadoop02:root-[INFO]:-exchange ssh keys for user gpadmin

20160905:13:53:27:002733 gpseginstall:hadoop02:root-[INFO]:-/usr/local/greenplum-db/./sbin/gpfixuserlimts -f /etc/security/limits.conf -u gpadmin

20160905:13:53:27:002733 gpseginstall:hadoop02:root-[INFO]:-remote command: . /usr/local/greenplum-db/./greenplum_path.sh; /usr/local/greenplum-db/./sbin/gpfixuserlimts -f /etc/security/limits.conf -u gpadmin

20160905:13:53:27:002733 gpseginstall:hadoop02:root-[INFO]:-version string on master: gpssh version 4.3.8.1 build 1

20160905:13:53:27:002733 gpseginstall:hadoop02:root-[INFO]:-remote command: . /usr/local/greenplum-db/./greenplum_path.sh; /usr/local/greenplum-db/./bin/gpssh --version

20160905:13:53:28:002733 gpseginstall:hadoop02:root-[INFO]:-remote command: . /usr/local/greenplum-db-4.3.8.1/greenplum_path.sh; /usr/local/greenplum-db-4.3.8.1/bin/gpssh --version

20160905:13:53:33:002733 gpseginstall:hadoop02:root-[INFO]:-SUCCESS -- Requested commands completed

 

4．创建数据存储区域

master节点：

[root@hadoop02 greenplum-db]# mkdir /data/master

[root@hadoop02 greenplum-db]# chown gpadmin /data/master

 

[root@hadoop02 greenplum-db]# source /usr/local/greenplum-db/greenplum_path.sh

 

[root@hadoop02 greenplum-db]# gpssh -h hadoop03 -e 'mkdir /data/master'

[hadoop03] mkdir /data/master

[root@hadoop02 greenplum-db]# gpssh -h hadoop03 -e 'chown gpadmin /data/master'

[hadoop03] chown gpadmin /data/master

 

segment节点：

创建hostfile_gpssh_segonly文件

[root@hadoop02 greenplum-db]# cat hostfile_gpssh_segonly

hadoop04

hadoop07

 

[root@hadoop02 greenplum-db]# source greenplum_path.sh

[root@hadoop02 greenplum-db]# gpssh -f hostfile_gpssh_segonly -e 'mkdir /data/primary'

[hadoop07] mkdir /data/primary

[hadoop04] mkdir /data/primary

[root@hadoop02 greenplum-db]# gpssh -f hostfile_gpssh_segonly -e 'mkdir /data/mirror'

[hadoop07] mkdir /data/mirror

[hadoop04] mkdir /data/mirror

[root@hadoop02 greenplum-db]# gpssh -f hostfile_gpssh_segonly -e 'chown gpadmin /data/primary'

[hadoop07] chown gpadmin /data/primary

[hadoop04] chown gpadmin /data/primary

[root@hadoop02 greenplum-db]# gpssh -f hostfile_gpssh_segonly -e 'chown gpadmin /data/mirror'

[hadoop07] chown gpadmin /data/mirror

[hadoop04] chown gpadmin /data/mirror

 

 

5．验证安装正确性

[gpadmin@hadoop02 ~]$ gpssh -f hostfile_exkeys -e ls -l $GPHOME

[hadoop02] ls -l /usr/local/greenplum-db/.

[hadoop02] total 280

[hadoop02] drwxr-xr-x 4 gpadmin gpadmin   4096 Apr 20 23:26 bin

[hadoop02] drwxr-xr-x 2 gpadmin gpadmin   4096 Apr 20 23:13 demo

[hadoop02] drwxr-xr-x 5 gpadmin gpadmin   4096 Apr 20 23:16 docs

[hadoop02] drwxr-xr-x 2 gpadmin gpadmin   4096 Apr 20 23:16 etc

[hadoop02] drwxr-xr-x 3 gpadmin gpadmin   4096 Apr 20 23:16 ext

[hadoop02] -rw-r--r-- 1 gpadmin gpadmin  43025 Apr 20 23:26 GPDB-LICENSE.txt

[hadoop02] -rwxrwxrwx 1 gpadmin gpadmin    735 Sep  5 13:20 greenplum_path.sh

[hadoop02] -rwxrwxrwx 1 gpadmin gpadmin     36 Sep  5 13:51 hostfile_exkeys

[hadoop02] drwxr-xr-x 6 gpadmin gpadmin   4096 Apr 20 23:16 include

[hadoop02] drwxr-xr-x 9 gpadmin gpadmin   4096 Apr 20 23:22 lib

[hadoop02] -rw-r--r-- 1 gpadmin gpadmin 192912 Apr 20 23:26 LICENSE.thirdparty

[hadoop02] drwxr-xr-x 2 gpadmin gpadmin   4096 Apr 20 23:26 sbin

[hadoop02] drwxr-xr-x 4 gpadmin gpadmin   4096 Apr 20 23:14 share

[hadoop07] ls -l /usr/local/greenplum-db/.

[hadoop07] total 280

[hadoop07] drwxr-xr-x 4 gpadmin gpadmin   4096 Apr 20 23:26 bin

[hadoop07] drwxr-xr-x 2 gpadmin gpadmin   4096 Apr 20 23:13 demo

[hadoop07] drwxr-xr-x 5 gpadmin gpadmin   4096 Apr 20 23:16 docs

[hadoop07] drwxr-xr-x 2 gpadmin gpadmin   4096 Apr 20 23:16 etc

[hadoop07] drwxr-xr-x 3 gpadmin gpadmin   4096 Apr 20 23:16 ext

[hadoop07] -rw-r--r-- 1 gpadmin gpadmin  43025 Apr 20 23:26 GPDB-LICENSE.txt

[hadoop07] -rwxrwxrwx 1 gpadmin gpadmin    735 Sep  5 13:20 greenplum_path.sh

[hadoop07] -rwxrwxrwx 1 gpadmin gpadmin     36 Sep  5 13:51 hostfile_exkeys

[hadoop07] drwxr-xr-x 6 gpadmin gpadmin   4096 Apr 20 23:16 include

[hadoop07] drwxr-xr-x 9 gpadmin gpadmin   4096 Apr 20 23:22 lib

[hadoop07] -rw-r--r-- 1 gpadmin gpadmin 192912 Apr 20 23:26 LICENSE.thirdparty

[hadoop07] drwxr-xr-x 2 gpadmin gpadmin   4096 Apr 20 23:26 sbin

[hadoop07] drwxr-xr-x 4 gpadmin gpadmin   4096 Apr 20 23:14 share

[hadoop03] ls -l /usr/local/greenplum-db/.

[hadoop03] total 280

[hadoop03] drwxr-xr-x 4 gpadmin gpadmin   4096 Apr 20 23:26 bin

[hadoop03] drwxr-xr-x 2 gpadmin gpadmin   4096 Apr 20 23:13 demo

[hadoop03] drwxr-xr-x 5 gpadmin gpadmin   4096 Apr 20 23:16 docs

[hadoop03] drwxr-xr-x 2 gpadmin gpadmin   4096 Apr 20 23:16 etc

[hadoop03] drwxr-xr-x 3 gpadmin gpadmin   4096 Apr 20 23:16 ext

[hadoop03] -rw-r--r-- 1 gpadmin gpadmin  43025 Apr 20 23:26 GPDB-LICENSE.txt

[hadoop03] -rwxrwxrwx 1 gpadmin gpadmin    735 Sep  5 13:20 greenplum_path.sh

[hadoop03] -rwxrwxrwx 1 gpadmin gpadmin     36 Sep  5 13:51 hostfile_exkeys

[hadoop03] drwxr-xr-x 6 gpadmin gpadmin   4096 Apr 20 23:16 include

[hadoop03] drwxr-xr-x 9 gpadmin gpadmin   4096 Apr 20 23:22 lib

[hadoop03] -rw-r--r-- 1 gpadmin gpadmin 192912 Apr 20 23:26 LICENSE.thirdparty

[hadoop03] drwxr-xr-x 2 gpadmin gpadmin   4096 Apr 20 23:26 sbin

[hadoop03] drwxr-xr-x 4 gpadmin gpadmin   4096 Apr 20 23:14 share

[hadoop04] ls -l /usr/local/greenplum-db/.

[hadoop04] total 280

[hadoop04] drwxr-xr-x 4 gpadmin gpadmin   4096 Apr 20 23:26 bin

[hadoop04] drwxr-xr-x 2 gpadmin gpadmin   4096 Apr 20 23:13 demo

[hadoop04] drwxr-xr-x 5 gpadmin gpadmin   4096 Apr 20 23:16 docs

[hadoop04] drwxr-xr-x 2 gpadmin gpadmin   4096 Apr 20 23:16 etc

[hadoop04] drwxr-xr-x 3 gpadmin gpadmin   4096 Apr 20 23:16 ext

[hadoop04] -rw-r--r-- 1 gpadmin gpadmin  43025 Apr 20 23:26 GPDB-LICENSE.txt

[hadoop04] -rwxrwxrwx 1 gpadmin gpadmin    735 Sep  5 13:20 greenplum_path.sh

[hadoop04] -rwxrwxrwx 1 gpadmin gpadmin     36 Sep  5 13:51 hostfile_exkeys

[hadoop04] drwxr-xr-x 6 gpadmin gpadmin   4096 Apr 20 23:16 include

[hadoop04] drwxr-xr-x 9 gpadmin gpadmin   4096 Apr 20 23:22 lib

[hadoop04] -rw-r--r-- 1 gpadmin gpadmin 192912 Apr 20 23:26 LICENSE.thirdparty

[hadoop04] drwxr-xr-x 2 gpadmin gpadmin   4096 Apr 20 23:26 sbin

[hadoop04] drwxr-xr-x 4 gpadmin gpadmin   4096 Apr 20 23:14 share

 

 

[gpadmin@hadoop02 ~]$ gpcheck -f hostfile_exkeys -m hadoop02 -s hadoop03

20160905:14:03:40:003265 gpcheck:hadoop02:gpadmin-[INFO]:-dedupe hostnames

20160905:14:03:40:003265 gpcheck:hadoop02:gpadmin-[INFO]:-Detected platform: Generic Linux Cluster

20160905:14:03:40:003265 gpcheck:hadoop02:gpadmin-[INFO]:-generate data on servers

20160905:14:03:40:003265 gpcheck:hadoop02:gpadmin-[INFO]:-copy data files from servers

20160905:14:03:40:003265 gpcheck:hadoop02:gpadmin-[INFO]:-delete remote tmp files

20160905:14:03:41:003265 gpcheck:hadoop02:gpadmin-[INFO]:-Using gpcheck config file: /usr/local/greenplum-db/./etc/gpcheck.cnf

20160905:14:03:41:003265 gpcheck:hadoop02:gpadmin-[ERROR]:-GPCHECK_ERROR host(None): utility will not check all settings when run as non-root user

20160905:14:03:41:003265 gpcheck:hadoop02:gpadmin-[ERROR]:-GPCHECK_ERROR host(hadoop03): /etc/sysctl.conf value for key 'kernel.shmmax' has value '5000000000' and expects '500000000'

20160905:14:03:41:003265 gpcheck:hadoop02:gpadmin-[ERROR]:-GPCHECK_ERROR host(hadoop03): /etc/sysctl.conf value for key 'kernel.sem' has value '250 5120000 100 20480' and expects '250 512000 100 2048'

20160905:14:03:41:003265 gpcheck:hadoop02:gpadmin-[ERROR]:-GPCHECK_ERROR host(hadoop03): /etc/sysctl.conf value for key 'kernel.shmall' has value '40000000000' and expects '4000000000'

20160905:14:03:41:003265 gpcheck:hadoop02:gpadmin-[ERROR]:-GPCHECK_ERROR host(hadoop03): ntpd not detected on machine

20160905:14:03:41:003265 gpcheck:hadoop02:gpadmin-[ERROR]:-GPCHECK_ERROR host(hadoop02): ntpd not detected on machine

20160905:14:03:41:003265 gpcheck:hadoop02:gpadmin-[ERROR]:-GPCHECK_ERROR host(hadoop07): /etc/sysctl.conf value for key 'kernel.shmmax' has value '5000000000' and expects '500000000'

20160905:14:03:41:003265 gpcheck:hadoop02:gpadmin-[ERROR]:-GPCHECK_ERROR host(hadoop07): /etc/sysctl.conf value for key 'kernel.sem' has value '250 5120000 100 20480' and expects '250 512000 100 2048'

20160905:14:03:41:003265 gpcheck:hadoop02:gpadmin-[ERROR]:-GPCHECK_ERROR host(hadoop07): /etc/sysctl.conf value for key 'kernel.shmall' has value '40000000000' and expects '4000000000'

20160905:14:03:41:003265 gpcheck:hadoop02:gpadmin-[ERROR]:-GPCHECK_ERROR host(hadoop07): ntpd not detected on machine

20160905:14:03:41:003265 gpcheck:hadoop02:gpadmin-[ERROR]:-GPCHECK_ERROR host(hadoop04): /etc/sysctl.conf value for key 'kernel.shmmax' has value '5000000000' and expects '500000000'

20160905:14:03:41:003265 gpcheck:hadoop02:gpadmin-[ERROR]:-GPCHECK_ERROR host(hadoop04): /etc/sysctl.conf value for key 'kernel.sem' has value '250 5120000 100 20480' and expects '250 512000 100 2048'

20160905:14:03:41:003265 gpcheck:hadoop02:gpadmin-[ERROR]:-GPCHECK_ERROR host(hadoop04): /etc/sysctl.conf value for key 'kernel.shmall' has value '40000000000' and expects '4000000000'

20160905:14:03:41:003265 gpcheck:hadoop02:gpadmin-[ERROR]:-GPCHECK_ERROR host(hadoop04): ntpd not detected on machine

20160905:14:03:41:003265 gpcheck:hadoop02:gpadmin-[INFO]:-gpcheck completing...

[gpadmin@hadoop02 ~]$ logout

[root@hadoop02 greenplum-db]# gpcheck -f hostfile_exkeys -m hadoop02 -s hadoop03

20160905:14:05:05:003392 gpcheck:hadoop02:root-[INFO]:-dedupe hostnames

20160905:14:05:05:003392 gpcheck:hadoop02:root-[INFO]:-Detected platform: Generic Linux Cluster

20160905:14:05:05:003392 gpcheck:hadoop02:root-[INFO]:-generate data on servers

20160905:14:05:05:003392 gpcheck:hadoop02:root-[INFO]:-copy data files from servers

20160905:14:05:05:003392 gpcheck:hadoop02:root-[INFO]:-delete remote tmp files

20160905:14:05:05:003392 gpcheck:hadoop02:root-[INFO]:-Using gpcheck config file: /usr/local/greenplum-db/./etc/gpcheck.cnf

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop03): on device (/dev/sda3) blockdev readahead value '256' does not match expected value '16384'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop03): on device (/dev/sda1) blockdev readahead value '256' does not match expected value '16384'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop03): on device (/dev/sda2) blockdev readahead value '256' does not match expected value '16384'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop03): on device (/dev/sda) blockdev readahead value '256' does not match expected value '16384'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop03): /etc/sysctl.conf value for key 'kernel.shmmax' has value '5000000000' and expects '500000000'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop03): /etc/sysctl.conf value for key 'kernel.sem' has value '250 5120000 100 20480' and expects '250 512000 100 2048'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop03): /etc/sysctl.conf value for key 'kernel.shmall' has value '40000000000' and expects '4000000000'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop03): ntpd not detected on machine

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop02): on device (/dev/sda3) blockdev readahead value '256' does not match expected value '16384'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop02): on device (/dev/sda1) blockdev readahead value '256' does not match expected value '16384'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop02): on device (/dev/sda2) blockdev readahead value '256' does not match expected value '16384'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop02): on device (/dev/sda) blockdev readahead value '256' does not match expected value '16384'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop02): ntpd not detected on machine

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop07): on device (/dev/sda3) blockdev readahead value '256' does not match expected value '16384'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop07): on device (/dev/sda1) blockdev readahead value '256' does not match expected value '16384'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop07): on device (/dev/sda2) blockdev readahead value '256' does not match expected value '16384'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop07): on device (/dev/sda) blockdev readahead value '256' does not match expected value '16384'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop07): /etc/sysctl.conf value for key 'kernel.shmmax' has value '5000000000' and expects '500000000'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop07): /etc/sysctl.conf value for key 'kernel.sem' has value '250 5120000 100 20480' and expects '250 512000 100 2048'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop07): /etc/sysctl.conf value for key 'kernel.shmall' has value '40000000000' and expects '4000000000'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop07): ntpd not detected on machine

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop04): on device (/dev/sda3) blockdev readahead value '256' does not match expected value '16384'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop04): on device (/dev/sda1) blockdev readahead value '256' does not match expected value '16384'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop04): on device (/dev/sda2) blockdev readahead value '256' does not match expected value '16384'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop04): on device (/dev/sda) blockdev readahead value '256' does not match expected value '16384'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop04): /etc/sysctl.conf value for key 'kernel.shmmax' has value '5000000000' and expects '500000000'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop04): /etc/sysctl.conf value for key 'kernel.sem' has value '250 5120000 100 20480' and expects '250 512000 100 2048'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop04): /etc/sysctl.conf value for key 'kernel.shmall' has value '40000000000' and expects '4000000000'

20160905:14:05:05:003392 gpcheck:hadoop02:root-[ERROR]:-GPCHECK_ERROR host(hadoop04): ntpd not detected on machine

20160905:14:05:05:003392 gpcheck:hadoop02:root-[INFO]:-gpcheck completing...

 

五、初始化GreenPlum

1．创建文件hostfile_gpinitsystem

里面包含所有segment节点的主机名

[gpadmin@hadoop02 ~]$ cat hostfile_gpinitsystem

hadoop04

hadoop07

 

2．创建GP配置文件

[gpadmin@hadoop02 ~]$ cp /usr/local/greenplum-db/docs/cli_help/gpconfigs/gpinitsystem_config .

 

关键信息如下：

ARRAY_NAME="EMC Greenplum DW"

SEG_PREFIX=gpseg

PORT_BASE=40000

declare -a DATA_DIRECTORY=(/data/primary /data/primary)

MASTER_HOSTNAME=hadoop02

MASTER_DIRECTORY=/data/master

MASTER_PORT=5432

TRUSTED_SHELL=ssh

 

CHECK_POINT_SEGMENTS=8

ENCODING=UNICODE

 

红色部分根据具体环境信息配置。

 

3．初始化GP

[gpadmin@hadoop02 ~]$ gpinitsystem -c gpinitsystem_config -h hostfile_gpinitsystem -s hadoop03 -S

20160905:14:33:51:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Checking configuration parameters, please wait...

20160905:14:33:51:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Reading Greenplum configuration file gpinitsystem_config

20160905:14:33:51:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Locale has not been set in gpinitsystem_config, will set to default value

20160905:14:33:51:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Locale set to en_US.utf8

20160905:14:33:51:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-No DATABASE_NAME set, will exit following template1 updates

20160905:14:33:52:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-MASTER_MAX_CONNECT not set, will set to default value 250

20160905:14:33:52:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Checking configuration parameters, Completed

20160905:14:33:52:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Commencing multi-home checks, please wait...

..

20160905:14:33:53:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Configuring build for standard array

20160905:14:33:53:008343 gpinitsystem:hadoop02:gpadmin-[WARN]:-Option -S supplied, but no mirrors have been defined, ignoring -S option

20160905:14:33:53:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Commencing multi-home checks, Completed

20160905:14:33:53:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Building primary segment instance array, please wait...

....

20160905:14:33:54:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Checking Master host

20160905:14:33:54:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Checking new segment hosts, please wait...

....

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Checking new segment hosts, Completed

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Greenplum Database Creation Parameters

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:---------------------------------------

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master Configuration

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:---------------------------------------

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master instance name       = EMC Greenplum DW

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master hostname            = hadoop02

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master port                = 5432

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master instance dir        = /data/master/gpseg-1

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master LOCALE              = en_US.utf8

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Greenplum segment prefix   = gpseg

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master Database            =

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master connections         = 250

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master buffers             = 128000kB

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Segment connections        = 750

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Segment buffers            = 128000kB

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Checkpoint segments        = 8

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Encoding                   = UNICODE

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Postgres param file        = Off

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Initdb to be used          = /usr/local/greenplum-db/./bin/initdb

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-GP_LIBRARY_PATH is         = /usr/local/greenplum-db/./lib

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Ulimit check               = Passed

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Array host connect type    = Single hostname per node

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master IP address [1]      = ::1

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master IP address [2]      = 10.0.0.201

20160905:14:33:57:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master IP address [3]      = 192.168.56.202

20160905:14:33:58:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master IP address [4]      = fe80::a00:27ff:fe3a:8bc7

20160905:14:33:58:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master IP address [5]      = fe80::a00:27ff:fe5a:31ad

20160905:14:33:58:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Standby Master             = hadoop03

20160905:14:33:58:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Primary segment #          = 2

20160905:14:33:58:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Standby IP address         = ::1

20160905:14:33:58:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Standby IP address         = 192.168.56.203

20160905:14:33:58:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Standby IP address         = fe80::a00:27ff:fe77:a2ab

20160905:14:33:58:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Total Database segments    = 4

20160905:14:33:58:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Trusted shell              = ssh

20160905:14:33:58:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Number segment hosts       = 2

20160905:14:33:58:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Mirroring config           = OFF

20160905:14:33:58:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:----------------------------------------

20160905:14:33:58:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Greenplum Primary Segment Configuration

20160905:14:33:58:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:----------------------------------------

20160905:14:33:58:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-hadoop04 /data/primary/gpseg0 40000 2 0

20160905:14:33:58:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-hadoop04 /data/primary/gpseg1 40001 3 1

20160905:14:33:58:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-hadoop07 /data/primary/gpseg2 40000 4 2

20160905:14:33:58:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-hadoop07 /data/primary/gpseg3 40001 5 3

Continue with Greenplum creation Yy/Nn>

y

20160905:14:34:11:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Building the Master instance database, please wait...

20160905:14:34:19:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Starting the Master in admin mode

20160905:14:34:25:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Commencing parallel build of primary segment instances

20160905:14:34:25:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Spawning parallel processes    batch [1], please wait...

....

20160905:14:34:25:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Waiting for parallel processes batch [1], please wait...

...................

20160905:14:34:44:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:------------------------------------------------

20160905:14:34:44:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Parallel process exit status

20160905:14:34:44:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:------------------------------------------------

20160905:14:34:44:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Total processes marked as completed           = 0

20160905:14:34:45:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Total processes marked as killed              = 0

20160905:14:34:45:008343 gpinitsystem:hadoop02:gpadmin-[WARN]:-Total processes marked as failed              = 4 <<<<<

20160905:14:34:45:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:------------------------------------------------

20160905:14:34:45:008343 gpinitsystem:hadoop02:gpadmin-[FATAL]:-Errors generated from parallel processes

20160905:14:34:45:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Dumped contents of status file to the log file

20160905:14:34:45:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Building composite backout file

20160905:14:34:45:gpinitsystem:hadoop02:gpadmin-[FATAL]:-Failures detected, see log file /home/gpadmin/gpAdminLogs/gpinitsystem_20160905.log for more detail Script Exiting!

20160905:14:34:45:008343 gpinitsystem:hadoop02:gpadmin-[WARN]:-Script has left Greenplum Database in an incomplete state

20160905:14:34:45:008343 gpinitsystem:hadoop02:gpadmin-[WARN]:-Run command /bin/bash /home/gpadmin/gpAdminLogs/backout_gpinitsystem_gpadmin_20160905_143351 to remove these changes

20160905:14:34:45:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-Start Function BACKOUT_COMMAND

20160905:14:34:45:008343 gpinitsystem:hadoop02:gpadmin-[INFO]:-End Function BACKOUT_COMMAND

 

初始化失败，运行回退脚本。

回退脚本默认生成在~/gpAdminLogs/目录下

文件名格式为backout_gpinitsystem_< user >_< timestamp >

 

[gpadmin@hadoop02 gpAdminLogs]$ bash backout_gpinitsystem_gpadmin_20160905_143351

Stopping Master instance

waiting for server to shut down.... done

server stopped

Removing Master log file

Removing Master lock files

Removing Master data directory files

removing directory /data/primary/gpseg0 on hadoop04

removing directory /data/primary/gpseg2 on hadoop07

removing directory /data/primary/gpseg1 on hadoop04

removing directory /data/primary/gpseg3 on hadoop07

 

回退成功

 

定位问题原因，日志中报错如下：

20160905:14:38:48:030960 gpcreateseg.sh:hadoop02:gpadmin-[INFO][2]:-Configuring segment postgresql.conf

20160905:14:38:48:030865 gpcreateseg.sh:hadoop02:gpadmin-[WARN]:-Failed to insert port=40000 in /data/primary/gpseg2/postgresql.conf on hadoop07

20160905:14:38:48:030787 gpcreateseg.sh:hadoop02:gpadmin-[INFO]:-Start Function BACKOUT_COMMAND

20160905:14:38:48:030960 gpcreateseg.sh:hadoop02:gpadmin-[INFO][2]:-Completed Update /data/primary/gpseg1/postgresql.conf file

20160905:14:38:48:030787 gpcreateseg.sh:hadoop02:gpadmin-[INFO]:-End Function BACKOUT_COMMAND

20160905:14:38:48:030865 gpcreateseg.sh:hadoop02:gpadmin-[INFO]:-End Function ED_PG_CONF

20160905:14:38:48:030787 gpcreateseg.sh:hadoop02:gpadmin-[INFO]:-Start Function BACKOUT_COMMAND

20160905:14:38:48:030865 gpcreateseg.sh:hadoop02:gpadmin-[FATAL][1]:-Failed Update port number to 40000

20160905:14:38:48:030787 gpcreateseg.sh:hadoop02:gpadmin-[INFO]:-End Function BACKOUT_COMMAND

20160905:14:38:48:030787 gpcreateseg.sh:hadoop02:gpadmin-[INFO][0]:-Completed to start segment instance database hadoop04 /data/primary/gpseg0

 

找到资料：https://support.pivotal.io/hc/communities/public/questions/200372738-HAWQ-Initialization

解决方法：

所有节点安装ed

[root@hadoop02 ~]# yum install -y ed

 

重新初始化

[gpadmin@hadoop02 ~]$ source /usr/local/greenplum-db/greenplum_path.sh

[gpadmin@hadoop02 ~]$ gpinitsystem -c gpinitsystem_config -h hostfile_gpinitsystem -s hadoop03 -S

20160905:14:53:15:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Checking configuration parameters, please wait...

20160905:14:53:15:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Reading Greenplum configuration file gpinitsystem_config

20160905:14:53:15:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Locale has not been set in gpinitsystem_config, will set to default value

20160905:14:53:15:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Locale set to en_US.utf8

20160905:14:53:16:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-No DATABASE_NAME set, will exit following template1 updates

20160905:14:53:16:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-MASTER_MAX_CONNECT not set, will set to default value 250

20160905:14:53:16:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Checking configuration parameters, Completed

20160905:14:53:16:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Commencing multi-home checks, please wait...

..

20160905:14:53:16:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Configuring build for standard array

20160905:14:53:16:001726 gpinitsystem:hadoop02:gpadmin-[WARN]:-Option -S supplied, but no mirrors have been defined, ignoring -S option

20160905:14:53:16:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Commencing multi-home checks, Completed

20160905:14:53:16:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Building primary segment instance array, please wait...

....

20160905:14:53:17:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Checking Master host

20160905:14:53:18:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Checking new segment hosts, please wait...

....

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Checking new segment hosts, Completed

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Greenplum Database Creation Parameters

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:---------------------------------------

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master Configuration

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:---------------------------------------

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master instance name       = EMC Greenplum DW

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master hostname            = hadoop02

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master port                = 5432

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master instance dir        = /data/master/gpseg-1

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master LOCALE              = en_US.utf8

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Greenplum segment prefix   = gpseg

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master Database            =

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master connections         = 250

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master buffers             = 128000kB

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Segment connections        = 750

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Segment buffers            = 128000kB

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Checkpoint segments        = 8

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Encoding                   = UNICODE

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Postgres param file        = Off

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Initdb to be used          = /usr/local/greenplum-db/./bin/initdb

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-GP_LIBRARY_PATH is         = /usr/local/greenplum-db/./lib

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Ulimit check               = Passed

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Array host connect type    = Single hostname per node

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master IP address [1]      = ::1

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master IP address [2]      = 10.0.0.201

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master IP address [3]      = 192.168.56.202

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master IP address [4]      = fe80::a00:27ff:fe3a:8bc7

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Master IP address [5]      = fe80::a00:27ff:fe5a:31ad

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Standby Master             = hadoop03

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Primary segment #          = 2

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Standby IP address         = ::1

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Standby IP address         = 192.168.56.203

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Standby IP address         = fe80::a00:27ff:fe77:a2ab

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Total Database segments    = 4

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Trusted shell              = ssh

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Number segment hosts       = 2

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Mirroring config           = OFF

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:----------------------------------------

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Greenplum Primary Segment Configuration

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:----------------------------------------

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-hadoop04 /data/primary/gpseg0 40000 2 0

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-hadoop04 /data/primary/gpseg1 40001 3 1

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-hadoop07 /data/primary/gpseg2 40000 4 2

20160905:14:53:21:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-hadoop07 /data/primary/gpseg3 40001 5 3

Continue with Greenplum creation Yy/Nn>

y

20160905:14:53:25:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Building the Master instance database, please wait...

20160905:14:53:34:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Starting the Master in admin mode

20160905:14:53:39:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Commencing parallel build of primary segment instances

20160905:14:53:39:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Spawning parallel processes    batch [1], please wait...

....

20160905:14:53:39:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Waiting for parallel processes batch [1], please wait...

........................

20160905:14:54:03:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:------------------------------------------------

20160905:14:54:03:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Parallel process exit status

20160905:14:54:03:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:------------------------------------------------

20160905:14:54:03:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Total processes marked as completed           = 4

20160905:14:54:04:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Total processes marked as killed              = 0

20160905:14:54:04:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Total processes marked as failed              = 0

20160905:14:54:04:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:------------------------------------------------

20160905:14:54:04:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Deleting distributed backout files

20160905:14:54:04:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Removing back out file

20160905:14:54:04:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-No errors generated from parallel processes

20160905:14:54:04:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Starting initialization of standby master hadoop03

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Validating environment and parameters for standby initialization...

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Checking for filespace directory /data/master/gpseg-1 on hadoop03

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:------------------------------------------------------

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Greenplum standby master initialization parameters

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:------------------------------------------------------

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Greenplum master hostname               = hadoop02

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Greenplum master data directory         = /data/master/gpseg-1

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Greenplum master port                   = 5432

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Greenplum standby master hostname       = hadoop03

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Greenplum standby master port           = 5432

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Greenplum standby master data directory = /data/master/gpseg-1

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Greenplum update system catalog         = On

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:------------------------------------------------------

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:- Filespace locations

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:------------------------------------------------------

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-pg_system -> /data/master/gpseg-1

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Syncing Greenplum Database extensions to standby

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-The packages on hadoop03 are consistent.

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Adding standby master to catalog...

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Database catalog updated successfully.

20160905:14:54:04:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Updating pg_hba.conf file...

20160905:14:54:10:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-pg_hba.conf files updated successfully.

20160905:14:54:12:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Updating filespace flat files...

20160905:14:54:12:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Filespace flat file updated successfully.

20160905:14:54:12:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Starting standby master

20160905:14:54:12:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Checking if standby master is running on host: hadoop03  in directory: /data/master/gpseg-1

20160905:14:54:12:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Cleaning up pg_hba.conf backup files...

20160905:14:54:18:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Backup files of pg_hba.conf cleaned up successfully.

20160905:14:54:18:017486 gpinitstandby:hadoop02:gpadmin-[INFO]:-Successfully created standby master on hadoop03

20160905:14:54:18:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Successfully completed standby master initialization

20160905:14:54:18:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Restarting the Greenplum instance in production mode

20160905:14:54:18:017876 gpstop:hadoop02:gpadmin-[INFO]:-Starting gpstop with args: -a -i -m -d /data/master/gpseg-1

20160905:14:54:18:017876 gpstop:hadoop02:gpadmin-[INFO]:-Gathering information and validating the environment...

20160905:14:54:18:017876 gpstop:hadoop02:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information

20160905:14:54:18:017876 gpstop:hadoop02:gpadmin-[INFO]:-Obtaining Segment details from master...

20160905:14:54:18:017876 gpstop:hadoop02:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 4.3.8.1 build 1'

20160905:14:54:18:017876 gpstop:hadoop02:gpadmin-[INFO]:-There are 0 connections to the database

20160905:14:54:18:017876 gpstop:hadoop02:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='immediate'

20160905:14:54:18:017876 gpstop:hadoop02:gpadmin-[INFO]:-Master host=hadoop02

20160905:14:54:18:017876 gpstop:hadoop02:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=immediate

20160905:14:54:18:017876 gpstop:hadoop02:gpadmin-[INFO]:-Master segment instance directory=/data/master/gpseg-1

20160905:14:54:19:017876 gpstop:hadoop02:gpadmin-[INFO]:-Attempting forceful termination of any leftover master process

20160905:14:54:19:017876 gpstop:hadoop02:gpadmin-[INFO]:-Terminating processes for segment /data/master/gpseg-1

20160905:14:54:20:017963 gpstart:hadoop02:gpadmin-[INFO]:-Starting gpstart with args: -a -d /data/master/gpseg-1

20160905:14:54:20:017963 gpstart:hadoop02:gpadmin-[INFO]:-Gathering information and validating the environment...

20160905:14:54:20:017963 gpstart:hadoop02:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 4.3.8.1 build 1'

20160905:14:54:20:017963 gpstart:hadoop02:gpadmin-[INFO]:-Greenplum Catalog Version: '201310150'

20160905:14:54:20:017963 gpstart:hadoop02:gpadmin-[INFO]:-Starting Master instance in admin mode

20160905:14:54:21:017963 gpstart:hadoop02:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information

20160905:14:54:21:017963 gpstart:hadoop02:gpadmin-[INFO]:-Obtaining Segment details from master...

20160905:14:54:21:017963 gpstart:hadoop02:gpadmin-[INFO]:-Setting new master era

20160905:14:54:21:017963 gpstart:hadoop02:gpadmin-[INFO]:-Master Started...

20160905:14:54:21:017963 gpstart:hadoop02:gpadmin-[INFO]:-Shutting down master

20160905:14:54:22:017963 gpstart:hadoop02:gpadmin-[INFO]:-Commencing parallel segment instance startup, please wait...

..

20160905:14:54:24:017963 gpstart:hadoop02:gpadmin-[INFO]:-Process results...

20160905:14:54:24:017963 gpstart:hadoop02:gpadmin-[INFO]:-----------------------------------------------------

20160905:14:54:24:017963 gpstart:hadoop02:gpadmin-[INFO]:-   Successful segment starts                                            = 4

20160905:14:54:24:017963 gpstart:hadoop02:gpadmin-[INFO]:-   Failed segment starts                                                = 0

20160905:14:54:24:017963 gpstart:hadoop02:gpadmin-[INFO]:-   Skipped segment starts (segments are marked down in configuration)   = 0

20160905:14:54:24:017963 gpstart:hadoop02:gpadmin-[INFO]:-----------------------------------------------------

20160905:14:54:24:017963 gpstart:hadoop02:gpadmin-[INFO]:-

20160905:14:54:24:017963 gpstart:hadoop02:gpadmin-[INFO]:-Successfully started 4 of 4 segment instances

20160905:14:54:24:017963 gpstart:hadoop02:gpadmin-[INFO]:-----------------------------------------------------

20160905:14:54:24:017963 gpstart:hadoop02:gpadmin-[INFO]:-Starting Master instance hadoop02 directory /data/master/gpseg-1

20160905:14:54:25:017963 gpstart:hadoop02:gpadmin-[INFO]:-Command pg_ctl reports Master hadoop02 instance active

20160905:14:54:25:017963 gpstart:hadoop02:gpadmin-[INFO]:-Starting standby master

20160905:14:54:25:017963 gpstart:hadoop02:gpadmin-[INFO]:-Checking if standby master is running on host: hadoop03  in directory: /data/master/gpseg-1

20160905:14:54:27:017963 gpstart:hadoop02:gpadmin-[INFO]:-Database successfully started

20160905:14:54:27:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Completed restart of Greenplum instance in production mode

20160905:14:54:27:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Loading gp_toolkit...

20160905:14:54:28:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Scanning utility log file for any warning messages

20160905:14:54:28:001726 gpinitsystem:hadoop02:gpadmin-[WARN]:-*******************************************************

20160905:14:54:28:001726 gpinitsystem:hadoop02:gpadmin-[WARN]:-Scan of log file indicates that some warnings or errors

20160905:14:54:28:001726 gpinitsystem:hadoop02:gpadmin-[WARN]:-were generated during the array creation

20160905:14:54:28:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Please review contents of log file

20160905:14:54:28:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-/home/gpadmin/gpAdminLogs/gpinitsystem_20160905.log

20160905:14:54:28:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-To determine level of criticality

20160905:14:54:28:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-These messages could be from a previous run of the utility

20160905:14:54:28:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-that was called today!

20160905:14:54:28:001726 gpinitsystem:hadoop02:gpadmin-[WARN]:-*******************************************************

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Greenplum Database instance successfully created

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-------------------------------------------------------

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-To complete the environment configuration, please

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-update gpadmin .bashrc file with the following

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-1. Ensure that the greenplum_path.sh file is sourced

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-2. Add "export MASTER_DATA_DIRECTORY=/data/master/gpseg-1"

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-   to access the Greenplum scripts for this instance:

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-   or, use -d /data/master/gpseg-1 option for the Greenplum scripts

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-   Example gpstate -d /data/master/gpseg-1

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Script log file = /home/gpadmin/gpAdminLogs/gpinitsystem_20160905.log

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-To remove instance, run gpdeletesystem utility

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Standby Master hadoop03 has been configured

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-To activate the Standby Master Segment in the event of Master

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-failure review options for gpactivatestandby

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-------------------------------------------------------

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-The Master /data/master/gpseg-1/pg_hba.conf post gpinitsystem

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-has been configured to allow all hosts within this new

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-array to intercommunicate. Any hosts external to this

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-new array must be explicitly added to this file

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-Refer to the Greenplum Admin support guide which is

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-located in the /usr/local/greenplum-db/./docs directory

20160905:14:54:29:001726 gpinitsystem:hadoop02:gpadmin-[INFO]:-------------------------------------------------------

 

初始化成功。

 

六、配置环境变量

master节点：

[gpadmin@hadoop02 ~]$ cat .bashrc

# .bashrc

 

# Source global definitions

if [ -f /etc/bashrc ]; then

. /etc/bashrc

fi

 

# User specific aliases and functions

source /usr/local/greenplum-db/greenplum_path.sh

export MASTER_DATA_DIRECTORY=/data/master/gpseg-1

 

如果有standby master节点需要将拷贝你的环境变量到standby节点

[gpadmin@hadoop02 ~]$ scp .bashrc hadoop03:~/

.bashrc    100%  223     0.2KB/s   00:00

 

七、创建测试数据库

[gpadmin@hadoop02 ~]$ createdb testDB -E utf-8

[gpadmin@hadoop02 ~]$ psql -d testDB

psql (8.2.15)

Type "help" for help.

 

testDB=# select version();

                                                                       version                                                       

                 

-------------------------------------------------------------------------------------------------------------------------------------

-----------------

 PostgreSQL 8.2.15 (Greenplum Database 4.3.8.1 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Apr

20 2016 08:08:56

(1 row)

 

至此，4节点GreenPlum集群安装完毕。