greenplum的扩容（3）-增加数据节点

按计划将原有的数据节点从2组主备扩充到3组主备

1，配置文件input_file

因为按常规要求，需要增加至少两个主机，并用gpexpand -f new_hosts_file命令来生成扩展配置文件，其中new_hosts_file记录了新增主机名，详情见官方文档。

而这里只增加1个主机和1 组主备节点，所以需按要求编写扩展配置文件input_file：

建议先通过select * from gp_segment_configuration;查看当前节点配置信息再按相关信息缩写。

按hostname:address:port:fselocation:dbid:content:preferred_role:replication_port格式，则原节点的信息如下：

sdw1:sdw1:40000:/data1/primary/gpseg0:2:0:p:41000

sdw2:sdw2:40000:/data1/primary/gpseg1:3:1:p:41000

sdw3:sdw3:50000:/data1/mirror/gpseg0:4:0:m:51000

sdw1:sdw1:50000:/data1/mirror/gpseg1:5:1:m:51000

 

--input_file新增节点配置为：

sdw3:sdw3:40000:/data1/primary/gpseg2:6:2:p:41000

sdw2:sdw2:50000:/data1/mirror/gpseg2:7:2:m:51000

 

--并在sdw2和sdw3上分别创建数据目录，非常重要，没有建好则无法装入文件系统

[gpadmin@sdw2~]$ mkdir /data1/mirror/gpseg2

[gpadmin@sdw3~]$ mkdir /data1/primary/gpseg2

2，配置文件input_file.fs

--当有新增的文件系统时，应设置此步，否则可跳过此步。

3，运行gpexpand

[gpadmin@mdw~]$ gpexpand -i input_file

--可以不用指定数据库，这里指定了则后面也要指定，如果这里出现重分布，而不是新建segment,说明有以前的gpexpand schema没有清理，运行gpexpand -c来清除。

--最后的错误可能是虚拟机分配的内存不足，而且sdw2上的m-seg2也未创建成功，直接全量恢复就OK。

20150505:23:41:50:015560gpexpand:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database)4.3.4.1 build 2'

20150505:23:41:50:015560gpexpand:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.2.15(Greenplum Database 4.3.4.1 build 2) on x86_64-unknown-linux-gnu, compiled byGCC gcc (GCC) 4.4.2 compiled on Feb 10 2015 14:15:10'

20150505:23:41:50:015560gpexpand:mdw:gpadmin-[INFO]:-Querying gpexpand schema for current expansionstate

20150505:23:41:50:015560gpexpand:mdw:gpadmin-[INFO]:-Readying Greenplum Database for a new expansion

20150505:23:42:19:015560gpexpand:mdw:gpadmin-[INFO]:-Checking database postgres for unalterabletables...

20150505:23:42:19:015560gpexpand:mdw:gpadmin-[INFO]:-Checking database tutorial for unalterabletables...

20150505:23:42:19:015560gpexpand:mdw:gpadmin-[INFO]:-Checking database template1 for unalterabletables...

20150505:23:42:19:015560gpexpand:mdw:gpadmin-[INFO]:-Checking database testDB for unalterable tables...

20150505:23:42:19:015560gpexpand:mdw:gpadmin-[INFO]:-Checking database default_login_database_name forunalterable tables...

20150505:23:42:19:015560gpexpand:mdw:gpadmin-[INFO]:-Checking database gpperfmon for unalterabletables...

20150505:23:42:19:015560gpexpand:mdw:gpadmin-[INFO]:-Checking database postgres for tables with uniqueindexes...

20150505:23:42:19:015560gpexpand:mdw:gpadmin-[INFO]:-Checking database tutorial for tables with uniqueindexes...

20150505:23:42:19:015560gpexpand:mdw:gpadmin-[INFO]:-Checking database template1 for tables with uniqueindexes...

20150505:23:42:19:015560gpexpand:mdw:gpadmin-[INFO]:-Checking database testDB for tables with uniqueindexes...

20150505:23:42:19:015560gpexpand:mdw:gpadmin-[INFO]:-Checking database default_login_database_name fortables with unique indexes...

20150505:23:42:19:015560gpexpand:mdw:gpadmin-[INFO]:-Checking database gpperfmon for tables with uniqueindexes...

20150505:23:42:19:015560gpexpand:mdw:gpadmin-[INFO]:-Syncing Greenplum Database extensions

20150505:23:42:19:015560gpexpand:mdw:gpadmin-[INFO]:-The packages on sdw2 are consistent.

20150505:23:42:19:015560gpexpand:mdw:gpadmin-[INFO]:-The packages on sdw3 are consistent.

20150505:23:42:19:015560gpexpand:mdw:gpadmin-[INFO]:-Creating segment template

20150505:23:42:19:015560gpexpand:mdw:gpadmin-[INFO]:-VACUUM FULL on the catalog tables

20150505:23:42:22:015560gpexpand:mdw:gpadmin-[INFO]:-Starting copy of segment dbid 1 to location/data/master/gpexpand_05052015_15560

20150505:23:42:29:015560gpexpand:mdw:gpadmin-[INFO]:-Copying postgresql.conf from existing segment intotemplate

20150505:23:42:29:015560gpexpand:mdw:gpadmin-[INFO]:-Copying pg_hba.conf from existing segment intotemplate

20150505:23:42:30:015560gpexpand:mdw:gpadmin-[INFO]:-Adding new segments into template pg_hba.conf

20150505:23:42:30:015560gpexpand:mdw:gpadmin-[INFO]:-Creating schema tar file

20150505:23:42:37:015560gpexpand:mdw:gpadmin-[INFO]:-Distributing template tar file to new hosts

20150505:23:42:51:015560gpexpand:mdw:gpadmin-[INFO]:-Configuring new segments (primary)

20150505:23:43:00:015560gpexpand:mdw:gpadmin-[INFO]:-Configuring new segments (mirror)

20150505:23:43:00:015560gpexpand:mdw:gpadmin-[INFO]:-Backing up pg_hba.conf file on original segments

20150505:23:43:01:015560gpexpand:mdw:gpadmin-[INFO]:-Copying new pg_hba.conf file to original segments

20150505:23:43:02:015560gpexpand:mdw:gpadmin-[INFO]:-Configuring original segments

20150505:23:43:02:015560gpexpand:mdw:gpadmin-[INFO]:-Cleaning up temporary template files

20150505:23:43:02:015560gpexpand:mdw:gpadmin-[INFO]:-Starting Greenplum Database in restricted mode

20150505:23:43:14:015560gpexpand:mdw:gpadmin-[INFO]:-Stopping database

20150505:23:43:40:015560gpexpand:mdw:gpadmin-[INFO]:-Checking if Transaction filespace was moved

20150505:23:43:40:015560gpexpand:mdw:gpadmin-[INFO]:-Checking if Temporary filespace was moved

20150505:23:43:40:015560gpexpand:mdw:gpadmin-[INFO]:-Configuring new segment filespaces

20150505:23:43:40:015560gpexpand:mdw:gpadmin-[INFO]:-Cleaning up databases in new segments.

20150505:23:43:40:015560gpexpand:mdw:gpadmin-[INFO]:-Starting master in utility mode

20150505:23:43:42:015560gpexpand:mdw:gpadmin-[INFO]:-Stopping master in utility mode

20150505:23:43:46:015560gpexpand:mdw:gpadmin-[INFO]:-Starting Greenplum Database in restricted mode

20150505:23:43:58:015560gpexpand:mdw:gpadmin-[INFO]:-Creating expansion schema

20150505:23:44:02:015560gpexpand:mdw:gpadmin-[INFO]:-Populating gpexpand.status_detail with data fromdatabase postgres

20150505:23:44:03:015560gpexpand:mdw:gpadmin-[INFO]:-Populating gpexpand.status_detail with data fromdatabase tutorial

20150505:23:44:04:015560gpexpand:mdw:gpadmin-[INFO]:-Populating gpexpand.status_detail with data fromdatabase template1

20150505:23:44:04:015560gpexpand:mdw:gpadmin-[INFO]:-Populating gpexpand.status_detail with data fromdatabase testDB

20150505:23:44:06:015560gpexpand:mdw:gpadmin-[INFO]:-Populating gpexpand.status_detail with data fromdatabase default_login_database_name

20150505:23:44:06:015560gpexpand:mdw:gpadmin-[INFO]:-Populating gpexpand.status_detail with data fromdatabase gpperfmon

20150505:23:44:06:015560gpexpand:mdw:gpadmin-[INFO]:-Stopping Greenplum Database

20150505:23:44:34:015560gpexpand:mdw:gpadmin-[ERROR]:-gpexpand failed: ExecutionError: 'non-zero rc: 2'occured.  Details:'GPSTART_INTERNAL_MASTER_ONLY=1 $GPHOME/bin/gpstop -a -f'  cmd had rc=2 completed=True halted=False

  stdout='20150505:23:44:06:016888gpstop:mdw:gpadmin-[INFO]:-Starting gpstop with args: -a -f

20150505:23:44:06:016888gpstop:mdw:gpadmin-[INFO]:-Gathering information and validating theenvironment...

20150505:23:44:06:016888gpstop:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information

20150505:23:44:06:016888gpstop:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...

20150505:23:44:07:016888gpstop:mdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database)4.3.4.1 build 2'

20150505:23:44:07:016888gpstop:mdw:gpadmin-[INFO]:-There are 1 connections to the database

20150505:23:44:07:016888gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='fast'

20150505:23:44:07:016888gpstop:mdw:gpadmin-[INFO]:-Master host=mdw

20150505:23:44:07:016888gpstop:mdw:gpadmin-[INFO]:-Detected 1 connections to database

20150505:23:44:07:016888gpstop:mdw:gpadmin-[INFO]:-Switching to WAIT mode

20150505:23:44:07:016888gpstop:mdw:gpadmin-[INFO]:-Will wait for shutdown to complete, this may takesome time if

20150505:23:44:07:016888gpstop:mdw:gpadmin-[INFO]:-there are a large number of active complextransactions, please wait...

20150505:23:44:07:016888gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=fast

20150505:23:44:07:016888gpstop:mdw:gpadmin-[INFO]:-Master segment instancedirectory=/data/master/gpseg-1

20150505:23:44:08:016888gpstop:mdw:gpadmin-[INFO]:-Attempting forceful termination of any leftovermaster process

20150505:23:44:08:016888gpstop:mdw:gpadmin-[INFO]:-Terminating processes for segment/data/master/gpseg-1

20150505:23:44:08:016888gpstop:mdw:gpadmin-[INFO]:-No standby master host configured

20150505:23:44:08:016888gpstop:mdw:gpadmin-[INFO]:-Commencing parallel primary segment instanceshutdown, please wait...

20150505:23:44:08:016888gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed

20150505:23:44:18:016888gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed

20150505:23:44:18:016888gpstop:mdw:gpadmin-[INFO]:-Commencing parallel mirror segment instanceshutdown, please wait...

20150505:23:44:18:016888gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed

20150505:23:44:28:016888gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed

20150505:23:44:28:016888gpstop:mdw:gpadmin-[INFO]:-------------------------------------------------

20150505:23:44:28:016888gpstop:mdw:gpadmin-[INFO]:-Failed Segment Stop Information

20150505:23:44:28:016888gpstop:mdw:gpadmin-[INFO]:-------------------------------------------------

20150505:23:44:28:016888gpstop:mdw:gpadmin-[INFO]:-DBID:6 FAILED  host:'sdw3'datadir:'/data1/primary/gpseg2' with reason:'Shutdown failed! [Errno 12] Cannotallocate memory'

20150505:23:44:28:016888gpstop:mdw:gpadmin-[INFO]:-DBID:7 FAILED  host:'sdw2'datadir:'/data1/mirror/gpseg2' with reason:'Shutdown failed'

20150505:23:44:28:016888gpstop:mdw:gpadmin-[INFO]:-DBID:4 FAILED  host:'sdw3'datadir:'/data1/mirror/gpseg0' with reason:'Shutdown failed! [Errno 12] Cannotallocate memory'

20150505:23:44:28:016888gpstop:mdw:gpadmin-[INFO]:-----------------------------------------------------

20150505:23:44:28:016888gpstop:mdw:gpadmin-[INFO]:-   Segmentsstopped successfully                              = 3

20150505:23:44:28:016888gpstop:mdw:gpadmin-[WARNING]:-Segments with errors during stop                           = 2   <<<<<<<<

20150505:23:44:28:016888gpstop:mdw:gpadmin-[INFO]:-  

20150505:23:44:28:016888gpstop:mdw:gpadmin-[WARNING]:-Segments that are currently marked down in configuration   = 1  <<<<<<<<

20150505:23:44:28:016888gpstop:mdw:gpadmin-[INFO]:-           (stop was still attempted on these segments)

20150505:23:44:28:016888gpstop:mdw:gpadmin-[INFO]:-----------------------------------------------------

20150505:23:44:28:016888gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown 3 of 6 segment instances<<<<<<<<

20150505:23:44:28:016888gpstop:mdw:gpadmin-[WARNING]:-------------------------------------------------

20150505:23:44:28:016888gpstop:mdw:gpadmin-[WARNING]:-Segment instance shutdown failures reported

20150505:23:44:28:016888gpstop:mdw:gpadmin-[WARNING]:-Failed to shutdown 2 of 6 segment instances<<<<<

20150505:23:44:28:016888gpstop:mdw:gpadmin-[WARNING]:-A total of 2 errors were encountered

20150505:23:44:28:016888gpstop:mdw:gpadmin-[WARNING]:-Review logfile/home/gpadmin/gpAdminLogs/gpstop_20150505.log

20150505:23:44:28:016888gpstop:mdw:gpadmin-[WARNING]:-For more details on segment shutdown failure(s)

20150505:23:44:28:016888gpstop:mdw:gpadmin-[WARNING]:-------------------------------------------------

20150505:23:44:28:016888gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpmmon process

20150505:23:44:28:016888gpstop:mdw:gpadmin-[WARNING]:-No leftover gpmmon process found

20150505:23:44:28:016888gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpsmon processes

20150505:23:44:28:016888gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover shared memory

20150505:23:44:34:016888gpstop:mdw:gpadmin-[ERROR]:-'NoneType' object has no attribute 'rc'

Traceback(most recent call last):

  File"/usr/local/greenplum-db/lib/python/gppylib/commands/base.py", line211, in run

    self.cmd.run()

  File"/usr/local/greenplum-db/lib/python/gppylib/operations/__init__.py",line 49, in run

    self.ret = self.execute()

  File "/usr/local/greenplum-db/lib/python/gppylib/operations/utils.py",line 52, in execute

    raise ret

AttributeError:'NoneType' object has no attribute 'rc'

20150505:23:44:34:016888gpstop:mdw:gpadmin-[WARNING]:-Unable to clean shared memory ('NoneType' objecthas no attribute 'rc')

'

  stderr=''

 

Exiting...

20150505:23:44:34:015560gpexpand:mdw:gpadmin-[ERROR]:-gpexpand is past the point of rollback. Anyremaining issues must be addressed outside of gpexpand.

20150505:23:44:34:015560gpexpand:mdw:gpadmin-[INFO]:-Shutting down gpexpand...

[gpadmin@mdw ~]$gpexpand -d 60:00:00

--都是重分布信息

 

--随机查看一个表的分布情况,由下可见重分布成功了。

testDB=# selectgp_segment_id,count(*) from a group by 1;

 gp_segment_id | count

---------------+-------

             2 |   337

             1 |   342

             0 |   321

 

(3 rows) [gpadmin@mdw ~]$ gpexpand -c     --移除gpexpand schema

20150505:23:09:00:011327gpexpand:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (GreenplumDatabase) 4.3.4.1 build 2'

20150505:23:09:01:011327 gpexpand:mdw:gpadmin-[INFO]:-masterGreenplum Version: 'PostgreSQL 8.2.15 (Greenplum Database 4.3.4.1 build 2) onx86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on Feb 102015 14:15:10'

20150505:23:09:01:011327gpexpand:mdw:gpadmin-[INFO]:-Querying gpexpand schema for current expansionstate

 

Do you want to dump thegpexpand.status_detail table to file? Yy|Nn (default=Y):

> y

20150505:23:09:19:011327gpexpand:mdw:gpadmin-[INFO]:-Dumping gpexpand.status_detail to/data/master/gpseg-1/gpexpand.status_detail

20150505:23:09:21:011327gpexpand:mdw:gpadmin-[INFO]:-Removing gpexpand schema

20150505:23:09:31:011327gpexpand:mdw:gpadmin-[INFO]:-Cleanup Finished. exiting...