greenplum如何删除segment节点

121 阅读 0 评论 80 点赞

我是靠谱客的博主专注乌冬面，最近开发中收集的这篇文章主要介绍greenplum如何删除segment节点，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

环境信息：

Red Hat Enterprise Linux Server release 7.5 (Maipo)

Greenplum Database 5.16.0

[gpadmin@rhmdw gpadata]$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.15.201 rhmdw
192.168.15.202 rhsdw1
192.168.15.203 rhsdw2
192.168.15.205 rhsdw03
192.168.15.206 rhsdw04

gp官网也未指出如何删除节点，网上有多个删除方式，针对于删除方式进行验证，在进行删除时需要注意以下两点内容：
1、删除节点的content必须是从大到小删除，如果存在单个content集群将会启动失败。
2、删除节点后必须将问题节点的内容恢复到其他节点。

测试过程删除rhsdw03和rhsdw04节点
集群状态查看：
[gpadmin@rhmdw ~]$ gpstate -m
20190418:17:58:44:025396 gpstate:rhmdw:gpadmin-[INFO]:-Starting gpstate with args: -m
20190418:17:58:44:025396 gpstate:rhmdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44'
20190418:17:58:44:025396 gpstate:rhmdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Jan 16 2019 02:32:15'
20190418:17:58:44:025396 gpstate:rhmdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20190418:17:58:44:025396 gpstate:rhmdw:gpadmin-[INFO]:--------------------------------------------------------------
20190418:17:58:44:025396 gpstate:rhmdw:gpadmin-[INFO]:--Current GPDB mirror list and status
20190418:17:58:44:025396 gpstate:rhmdw:gpadmin-[INFO]:--Type = Spread
20190418:17:58:44:025396 gpstate:rhmdw:gpadmin-[INFO]:--------------------------------------------------------------
20190418:17:58:44:025396 gpstate:rhmdw:gpadmin-[INFO]:-   Mirror    Datadir                    Port   Status    Data Status
20190418:17:58:44:025396 gpstate:rhmdw:gpadmin-[INFO]:-   rhsdw2    /gp/gpdata/mirror/gpseg0   7000   Passive   Synchronized
20190418:17:58:44:025396 gpstate:rhmdw:gpadmin-[INFO]:-   rhsdw1    /gp/gpdata/mirror/gpseg1   7000   Passive   Synchronized
20190418:17:58:44:025396 gpstate:rhmdw:gpadmin-[INFO]:-   rhsdw03   /gp/gpdata/mirror/gpseg2   7000   Passive   Synchronized
20190418:17:58:44:025396 gpstate:rhmdw:gpadmin-[INFO]:-   rhsdw04   /gp/gpdata/mirror/gpseg3   7000   Passive   Synchronized
20190418:17:58:44:025396 gpstate:rhmdw:gpadmin-[INFO]:--------------------------------------------------------------
[gpadmin@rhmdw ~]$ psql -d postgres
psql (8.3.23)
Type "help" for help.

[gpadmin@rhmdw ~]$ gp_dump --gp-d=/gp/gpdata gaoyc
20190418:18:08:09|gp_dump-[INFO]:-Read params: <empty>
20190418:18:08:09|gp_dump-[INFO]:-Command line options analyzed.
20190418:18:08:09|gp_dump-[INFO]:-Connecting to master database on host localhost port 5432 database gaoyc.
20190418:18:08:09|gp_dump-[INFO]:-Reading Greenplum Database configuration info from master database.
20190418:18:08:09|gp_dump-[INFO]:-Preparing to dump the following segments:
20190418:18:08:09|gp_dump-[INFO]:-Segment 3 (dbid 8)
20190418:18:08:09|gp_dump-[INFO]:-Segment 2 (dbid 7)
20190418:18:08:09|gp_dump-[INFO]:-Segment 1 (dbid 3)
20190418:18:08:09|gp_dump-[INFO]:-Segment 0 (dbid 2)
20190418:18:08:09|gp_dump-[INFO]:-Master (dbid 1)
20190418:18:08:09|gp_dump-[INFO]:-Starting a transaction on master database gaoyc.
20190418:18:08:09|gp_dump-[INFO]:-Getting a lock on pg_class in database gaoyc.
20190418:18:08:09|GetTimestampKey-[INFO]:-Timestamp key is generated as it is not provided by the user.
20190418:18:08:09|gp_dump-[INFO]:-About to spin off 5 threads with timestamp key 20190418180809
20190418:18:08:09|gp_dump-[INFO]:-Creating thread to backup dbid 8: host rhsdw04 port 6000 database gaoyc
20190418:18:08:09|gp_dump-[INFO]:-Creating thread to backup dbid 7: host rhsdw03 port 6000 database gaoyc
20190418:18:08:09|gp_dump-[INFO]:-Creating thread to backup dbid 3: host rhsdw2 port 6000 database gaoyc
20190418:18:08:09|gp_dump-[INFO]:-Creating thread to backup dbid 2: host rhsdw1 port 6000 database gaoyc
20190418:18:08:09|gp_dump-[INFO]:-Creating thread to backup dbid 1: host rhmdw port 5432 database gaoyc
20190418:18:08:09|gp_dump-[INFO]:-Waiting for remote gp_dump_agent processes to start transactions in serializable isolation level
20190418:18:08:09|gp_dump-[INFO]:-Listening for messages from server on dbid 8 connection
20190418:18:08:09|gp_dump-[INFO]:-Successfully launched Greenplum Database backup on dbid 8 server
20190418:18:08:09|gp_dump-[INFO]:-Listening for messages from server on dbid 7 connection
20190418:18:08:09|gp_dump-[INFO]:-Successfully launched Greenplum Database backup on dbid 7 server
20190418:18:08:09|gp_dump-[INFO]:-Listening for messages from server on dbid 2 connection
20190418:18:08:09|gp_dump-[INFO]:-Successfully launched Greenplum Database backup on dbid 2 server
20190418:18:08:09|gp_dump-[INFO]:-Listening for messages from server on dbid 3 connection
20190418:18:08:09|gp_dump-[INFO]:-Successfully launched Greenplum Database backup on dbid 3 server
20190418:18:08:09|gp_dump-[INFO]:-Listening for messages from server on dbid 1 connection
20190418:18:08:12|gp_dump-[INFO]:-backup succeeded for dbid 7 on host rhsdw03
20190418:18:08:12|gp_dump-[INFO]:-Successfully launched Greenplum Database backup on dbid 1 server
20190418:18:08:12|gp_dump-[INFO]:-backup succeeded for dbid 1 on host rhmdw
20190418:18:08:12|gp_dump-[INFO]:-backup succeeded for dbid 2 on host rhsdw1
20190418:18:08:12|gp_dump-[INFO]:-backup succeeded for dbid 8 on host rhsdw04
20190418:18:08:12|gp_dump-[INFO]:-All remote gp_dump_agent processes have began transactions in serializable isolation level
20190418:18:08:12|gp_dump-[INFO]:-Waiting for remote gp_dump_agent processes to obtain local locks on dumpable objects
20190418:18:08:12|gp_dump-[INFO]:-All remote gp_dump_agent processes have obtains the necessary locks
20190418:18:08:12|gp_dump-[INFO]:-Committing transaction on the master database, thereby releasing locks.
20190418:18:08:12|gp_dump-[INFO]:-Waiting for all remote gp_dump_agent programs to finish.
20190418:18:08:12|gp_dump-[INFO]:-backup succeeded for dbid 3 on host rhsdw2
20190418:18:08:12|gp_dump-[INFO]:-All remote gp_dump_agent programs are finished.
20190418:18:08:12|gp_dump-[INFO]:-Report results also written to /gp/gpdata/master/gpseg-1/gp_dump_20190418180809.rpt.

Greenplum Database Backup Report
Timestamp Key: 20190418180809
gp_dump Command Line: --gp-d=/gp/gpdata gaoyc
Pass through Command Line Options: None
Compression Program: None
Backup Type: Full

Individual Results
        segment 3 (dbid 8) Host rhsdw04 Port 6000 Database gaoyc BackupFile /gp/gpdata/gp_dump_3_8_20190418180809: Succeeded
        segment 2 (dbid 7) Host rhsdw03 Port 6000 Database gaoyc BackupFile /gp/gpdata/gp_dump_2_7_20190418180809: Succeeded
        segment 1 (dbid 3) Host rhsdw2 Port 6000 Database gaoyc BackupFile /gp/gpdata/gp_dump_1_3_20190418180809: Succeeded
        segment 0 (dbid 2) Host rhsdw1 Port 6000 Database gaoyc BackupFile /gp/gpdata/gp_dump_0_2_20190418180809: Succeeded
        Master (dbid 1) Host rhmdw Port 5432 Database gaoyc BackupFile /gp/gpdata/gp_dump_-1_1_20190418180809: Succeeded
        Master (dbid 1) Host rhmdw Port 5432 Database gaoyc BackupFile /gp/gpdata/gp_dump_-1_1_20190418180809_post_data: Succeeded

gp_dump utility finished successfully.

所有数据库备份完成。
关闭数据库：
[gpadmin@rhmdw ~]$ gpstop -a
20190418:18:09:57:026073 gpstop:rhmdw:gpadmin-[INFO]:-Starting gpstop with args: -a
20190418:18:09:57:026073 gpstop:rhmdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20190418:18:09:57:026073 gpstop:rhmdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20190418:18:09:57:026073 gpstop:rhmdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20190418:18:09:58:026073 gpstop:rhmdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44'
20190418:18:09:58:026073 gpstop:rhmdw:gpadmin-[INFO]:-There are 0 connections to the database
20190418:18:09:58:026073 gpstop:rhmdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='smart'
20190418:18:09:58:026073 gpstop:rhmdw:gpadmin-[INFO]:-Master host=rhmdw
20190418:18:09:58:026073 gpstop:rhmdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=smart
20190418:18:09:58:026073 gpstop:rhmdw:gpadmin-[INFO]:-Master segment instance directory=/gp/gpdata/master/gpseg-1
20190418:18:09:59:026073 gpstop:rhmdw:gpadmin-[INFO]:-Attempting forceful termination of any leftover master process
20190418:18:09:59:026073 gpstop:rhmdw:gpadmin-[INFO]:-Terminating processes for segment /gp/gpdata/master/gpseg-1
20190418:18:09:59:026073 gpstop:rhmdw:gpadmin-[ERROR]:-Failed to kill processes for segment /gp/gpdata/master/gpseg-1: ([Errno 3] No such process)
20190418:18:09:59:026073 gpstop:rhmdw:gpadmin-[INFO]:-Stopping master standby host rhsdw1 mode=fast
20190418:18:10:01:026073 gpstop:rhmdw:gpadmin-[INFO]:-Successfully shutdown standby process on rhsdw1
20190418:18:10:01:026073 gpstop:rhmdw:gpadmin-[INFO]:-Targeting dbid [2, 4, 3, 5, 7, 9, 8, 10] for shutdown
20190418:18:10:01:026073 gpstop:rhmdw:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait...
20190418:18:10:01:026073 gpstop:rhmdw:gpadmin-[INFO]:-0.00% of jobs completed
20190418:18:10:06:026073 gpstop:rhmdw:gpadmin-[INFO]:-100.00% of jobs completed
20190418:18:10:06:026073 gpstop:rhmdw:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait...
20190418:18:10:06:026073 gpstop:rhmdw:gpadmin-[INFO]:-0.00% of jobs completed
20190418:18:10:09:026073 gpstop:rhmdw:gpadmin-[INFO]:-100.00% of jobs completed
20190418:18:10:09:026073 gpstop:rhmdw:gpadmin-[INFO]:-----------------------------------------------------
20190418:18:10:09:026073 gpstop:rhmdw:gpadmin-[INFO]:- Segments stopped successfully = 8
20190418:18:10:09:026073 gpstop:rhmdw:gpadmin-[INFO]:- Segments with errors during stop = 0
20190418:18:10:09:026073 gpstop:rhmdw:gpadmin-[INFO]:-----------------------------------------------------
20190418:18:10:09:026073 gpstop:rhmdw:gpadmin-[INFO]:-Successfully shutdown 8 of 8 segment instances
20190418:18:10:09:026073 gpstop:rhmdw:gpadmin-[INFO]:-Database successfully shutdown with no errors reported
20190418:18:10:09:026073 gpstop:rhmdw:gpadmin-[INFO]:-Cleaning up leftover gpmmon process
20190418:18:10:09:026073 gpstop:rhmdw:gpadmin-[INFO]:-No leftover gpmmon process found
20190418:18:10:09:026073 gpstop:rhmdw:gpadmin-[INFO]:-Cleaning up leftover gpsmon processes
20190418:18:10:10:026073 gpstop:rhmdw:gpadmin-[INFO]:-No leftover gpsmon processes on some hosts. not attempting forceful termination on these hosts
20190418:18:10:10:026073 gpstop:rhmdw:gpadmin-[INFO]:-Cleaning up leftover shared memory
启动数据库到master模式，进入管理模式手动删除节点信息：
[gpadmin@rhmdw ~]$ gpstart -m
20190418:18:13:49:026195 gpstart:rhmdw:gpadmin-[INFO]:-Starting gpstart with args: -m
20190418:18:13:49:026195 gpstart:rhmdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20190418:18:13:49:026195 gpstart:rhmdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44'
20190418:18:13:49:026195 gpstart:rhmdw:gpadmin-[INFO]:-Greenplum Catalog Version: '301705051'
20190418:18:13:49:026195 gpstart:rhmdw:gpadmin-[WARNING]:-****************************************************************************
20190418:18:13:49:026195 gpstart:rhmdw:gpadmin-[WARNING]:-Master-only start requested in a configuration with a standby master.
20190418:18:13:49:026195 gpstart:rhmdw:gpadmin-[WARNING]:-This is advisable only under the direct supervision of Greenplum support.
20190418:18:13:49:026195 gpstart:rhmdw:gpadmin-[WARNING]:-This mode of operation is not supported in a production environment and
20190418:18:13:49:026195 gpstart:rhmdw:gpadmin-[WARNING]:-may lead to a split-brain condition and possible unrecoverable data loss.
20190418:18:13:49:026195 gpstart:rhmdw:gpadmin-[WARNING]:-****************************************************************************

Continue with master-only startup Yy|Nn (default=N):
> y
20190418:18:13:52:026195 gpstart:rhmdw:gpadmin-[INFO]:-Starting Master instance in admin mode
20190418:18:13:53:026195 gpstart:rhmdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20190418:18:13:53:026195 gpstart:rhmdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20190418:18:13:53:026195 gpstart:rhmdw:gpadmin-[INFO]:-Setting new master era
20190418:18:13:53:026195 gpstart:rhmdw:gpadmin-[INFO]:-Master Started...
[gpadmin@rhmdw ~]$ PGOPTIONS="-c gp_session_role=utility" psql -d postgres

psql (8.3.23)
Type "help" for help.

postgres=# select * from gp_segment_configuration order by dbid;
dbid | content | role | preferred_role | mode | status | port | hostname | address | replication_port
------+---------+------+----------------+------+--------+------+----------+---------+------------------
    1 |      -1 | p    | p              | s    | u      | 5432 | rhmdw    | rhmdw   |
    2 |       0 | p    | p              | s    | u      | 6000 | rhsdw1   | rhsdw1 |             9000
    3 |       1 | p    | p              | s    | u      | 6000 | rhsdw2   | rhsdw2 |             9000
    4 |       0 | m    | m              | s    | u      | 7000 | rhsdw2   | rhsdw2 |             8000
    5 |       1 | m    | m              | s    | u      | 7000 | rhsdw1   | rhsdw1 |             8000
    6 |      -1 | m    | m              | s    | u      | 5432 | rhsdw1   | rhsdw1 |
    7 |       2 | p    | p              | s    | u      | 6000 | rhsdw03 | rhsdw03 |             9000
    8 |       3 | p    | p              | s    | u      | 6000 | rhsdw04 | rhsdw04 |             9000
    9 |       2 | m    | m              | s    | u      | 7000 | rhsdw03 | rhsdw03 |             8000
   10 |       3 | m    | m              | s    | u      | 7000 | rhsdw04 | rhsdw04 |             8000
(10 rows)
发现删除rhsdw03和rhsdw04后content内容为2和3的丢失，表示我们在删除后需要进行手动恢复数据。
postgres=# set allow_system_table_mods='dml';
SET
postgres=# delete from gp_segment_configuration where dbid in (10,9,8,7);
DELETE 4
postgres=# select * from gp_segment_configuration order by dbid;
dbid | content | role | preferred_role | mode | status | port | hostname | address | replication_port
------+---------+------+----------------+------+--------+------+----------+---------+------------------
    1 |      -1 | p    | p              | s    | u      | 5432 | rhmdw    | rhmdw   |
    2 |       0 | p    | p              | s    | u      | 6000 | rhsdw1   | rhsdw1 |             9000
    3 |       1 | p    | p              | s    | u      | 6000 | rhsdw2   | rhsdw2 |             9000
    4 |       0 | m    | m              | s    | u      | 7000 | rhsdw2   | rhsdw2 |             8000
    5 |       1 | m    | m              | s    | u      | 7000 | rhsdw1   | rhsdw1 |             8000
    6 |      -1 | m    | m              | s    | u      | 5432 | rhsdw1   | rhsdw1 |
(6 rows)

postgres=# select * from pg_filespace_entry;
fsefsoid | fsedbid |        fselocation
----------+---------+---------------------------
     3052 |       1 | /gp/gpdata/master/gpseg-1
     3052 |       2 | /gp/gpdata/primary/gpseg0
     3052 |       3 | /gp/gpdata/primary/gpseg1
     3052 |       4 | /gp/gpdata/mirror/gpseg0
     3052 |       5 | /gp/gpdata/mirror/gpseg1
     3052 |       7 | /gp/gpdata/primary/gpseg2
     3052 |       6 | /gp/gpdata/master/gpseg-1
     3052 |       8 | /gp/gpdata/primary/gpseg3
     3052 |       9 | /gp/gpdata/mirror/gpseg2
     3052 |      10 | /gp/gpdata/mirror/gpseg3
(10 rows)

经过多次删除节点测试，如果数据库中现存的节点content存在单个，集群启动失败，提示一下报错：
[gpadmin@rhmdw ~]$ gpstart -a
20190418:17:16:14:018220 gpstart:rhmdw:gpadmin-[INFO]:-Starting gpstart with args: -a
20190418:17:16:14:018220 gpstart:rhmdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20190418:17:16:14:018220 gpstart:rhmdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44'
20190418:17:16:14:018220 gpstart:rhmdw:gpadmin-[INFO]:-Greenplum Catalog Version: '301705051'
20190418:17:16:14:018220 gpstart:rhmdw:gpadmin-[INFO]:-Starting Master instance in admin mode
20190418:17:16:16:018220 gpstart:rhmdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20190418:17:16:16:018220 gpstart:rhmdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20190418:17:16:16:018220 gpstart:rhmdw:gpadmin-[INFO]:-Setting new master era
20190418:17:16:16:018220 gpstart:rhmdw:gpadmin-[INFO]:-Master Started...
20190418:17:16:16:018220 gpstart:rhmdw:gpadmin-[CRITICAL]:-gpstart failed. (Reason=''NoneType' object has no attribute 'hostname'') exiting...
处理方式删除单个content内容即可。

删除完成，重新启动数据库到集群模式
[gpadmin@rhmdw ~]$ gpstop
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:-Starting gpstop with args:
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44'
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:---------------------------------------------
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:-Master instance parameters
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:---------------------------------------------
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:-   Master Greenplum instance process active PID   = 26207
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:-   Database                                       = template1
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:-   Master port                                    = 5432
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:-   Master directory                               = /gp/gpdata/master/gpseg-1
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:-   Shutdown mode                                  = smart
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:-   Timeout                                        = 120
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:-   Shutdown Master standby host                   = On
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:---------------------------------------------
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:-Segment instances that will be shutdown:
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:---------------------------------------------
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:-   Host     Datadir                     Port   Status
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:-   rhsdw1   /gp/gpdata/primary/gpseg0   6000   u
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:-   rhsdw2   /gp/gpdata/mirror/gpseg0    7000   u
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:-   rhsdw2   /gp/gpdata/primary/gpseg1   6000   u
20190418:18:21:12:026228 gpstop:rhmdw:gpadmin-[INFO]:-   rhsdw1   /gp/gpdata/mirror/gpseg1    7000   u

Continue with Greenplum instance shutdown Yy|Nn (default=N):
> y
20190418:18:21:16:026228 gpstop:rhmdw:gpadmin-[INFO]:-There are 0 connections to the database
20190418:18:21:16:026228 gpstop:rhmdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='smart'
20190418:18:21:16:026228 gpstop:rhmdw:gpadmin-[INFO]:-Master host=rhmdw
20190418:18:21:16:026228 gpstop:rhmdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=smart
20190418:18:21:16:026228 gpstop:rhmdw:gpadmin-[INFO]:-Master segment instance directory=/gp/gpdata/master/gpseg-1
20190418:18:21:17:026228 gpstop:rhmdw:gpadmin-[INFO]:-Attempting forceful termination of any leftover master process
20190418:18:21:17:026228 gpstop:rhmdw:gpadmin-[INFO]:-Terminating processes for segment /gp/gpdata/master/gpseg-1
20190418:18:21:17:026228 gpstop:rhmdw:gpadmin-[ERROR]:-Failed to kill processes for segment /gp/gpdata/master/gpseg-1: ([Errno 3] No such process)
20190418:18:21:17:026228 gpstop:rhmdw:gpadmin-[INFO]:-Stopping master standby host rhsdw1 mode=fast
20190418:18:21:17:026228 gpstop:rhmdw:gpadmin-[WARNING]:-Error occured while stopping the standby master: ExecutionError: 'non-zero rc: 1' occured. Details: 'ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=60 rhsdw1 ". /gp/greenplum-db/./greenplum_path.sh; $GPHOME/bin/pg_ctl -D /gp/gpdata/master/gpseg-1 -m fast -w -t 120 stop"' cmd had rc=1 completed=True halted=False
stdout=''
stderr='pg_ctl: PID file "/gp/gpdata/master/gpseg-1/postmaster.pid" does not exist
Is server running?
'
20190418:18:21:18:026228 gpstop:rhmdw:gpadmin-[INFO]:-Successfully shutdown standby process on rhsdw1
20190418:18:21:18:026228 gpstop:rhmdw:gpadmin-[INFO]:-Targeting dbid [2, 4, 3, 5] for shutdown
20190418:18:21:18:026228 gpstop:rhmdw:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait...
20190418:18:21:18:026228 gpstop:rhmdw:gpadmin-[INFO]:-0.00% of jobs completed
20190418:18:21:19:026228 gpstop:rhmdw:gpadmin-[INFO]:-100.00% of jobs completed
20190418:18:21:19:026228 gpstop:rhmdw:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait...
20190418:18:21:19:026228 gpstop:rhmdw:gpadmin-[INFO]:-0.00% of jobs completed
20190418:18:21:20:026228 gpstop:rhmdw:gpadmin-[INFO]:-100.00% of jobs completed
20190418:18:21:20:026228 gpstop:rhmdw:gpadmin-[INFO]:-----------------------------------------------------
20190418:18:21:20:026228 gpstop:rhmdw:gpadmin-[INFO]:- Segments stopped successfully = 4
20190418:18:21:20:026228 gpstop:rhmdw:gpadmin-[INFO]:- Segments with errors during stop = 0
20190418:18:21:20:026228 gpstop:rhmdw:gpadmin-[INFO]:-----------------------------------------------------
20190418:18:21:20:026228 gpstop:rhmdw:gpadmin-[INFO]:-Successfully shutdown 4 of 4 segment instances
20190418:18:21:20:026228 gpstop:rhmdw:gpadmin-[INFO]:-Database successfully shutdown with no errors reported
20190418:18:21:20:026228 gpstop:rhmdw:gpadmin-[INFO]:-Cleaning up leftover gpmmon process
20190418:18:21:20:026228 gpstop:rhmdw:gpadmin-[INFO]:-No leftover gpmmon process found
20190418:18:21:20:026228 gpstop:rhmdw:gpadmin-[INFO]:-Cleaning up leftover gpsmon processes
20190418:18:21:21:026228 gpstop:rhmdw:gpadmin-[INFO]:-No leftover gpsmon processes on some hosts. not attempting forceful termination on these hosts
20190418:18:21:21:026228 gpstop:rhmdw:gpadmin-[INFO]:-Cleaning up leftover shared memory
报错提示gpstop删除整个集群，其他节点未在运行提示失败，可以使用gpstop -m 停止master即可。
[gpadmin@rhmdw config]$ gpstop -m
20190418:19:00:40:030875 gpstop:rhmdw:gpadmin-[INFO]:-Starting gpstop with args: -m
20190418:19:00:40:030875 gpstop:rhmdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20190418:19:00:40:030875 gpstop:rhmdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20190418:19:00:40:030875 gpstop:rhmdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20190418:19:00:40:030875 gpstop:rhmdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44'

Continue with master-only shutdown Yy|Nn (default=N):
> y
20190418:19:00:41:030875 gpstop:rhmdw:gpadmin-[INFO]:-There are 0 connections to the database
20190418:19:00:41:030875 gpstop:rhmdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='smart'
20190418:19:00:41:030875 gpstop:rhmdw:gpadmin-[INFO]:-Master host=rhmdw
20190418:19:00:41:030875 gpstop:rhmdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=smart
20190418:19:00:41:030875 gpstop:rhmdw:gpadmin-[INFO]:-Master segment instance directory=/gp/gpdata/master/gpseg-1
20190418:19:00:43:030875 gpstop:rhmdw:gpadmin-[INFO]:-Attempting forceful termination of any leftover master process
20190418:19:00:43:030875 gpstop:rhmdw:gpadmin-[INFO]:-Terminating processes for segment /gp/gpdata/master/gpseg-1
20190418:19:00:43:030875 gpstop:rhmdw:gpadmin-[ERROR]:-Failed to kill processes for segment /gp/gpdata/master/gpseg-1: ([Errno 3] No such process)
[gpadmin@rhmdw ~]$ gpstart -a
20190418:18:21:30:026332 gpstart:rhmdw:gpadmin-[INFO]:-Starting gpstart with args: -a
20190418:18:21:30:026332 gpstart:rhmdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20190418:18:21:30:026332 gpstart:rhmdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44'
20190418:18:21:30:026332 gpstart:rhmdw:gpadmin-[INFO]:-Greenplum Catalog Version: '301705051'
20190418:18:21:30:026332 gpstart:rhmdw:gpadmin-[INFO]:-Starting Master instance in admin mode
20190418:18:21:31:026332 gpstart:rhmdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20190418:18:21:31:026332 gpstart:rhmdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20190418:18:21:31:026332 gpstart:rhmdw:gpadmin-[INFO]:-Setting new master era
20190418:18:21:31:026332 gpstart:rhmdw:gpadmin-[INFO]:-Master Started...
20190418:18:21:31:026332 gpstart:rhmdw:gpadmin-[INFO]:-Shutting down master
20190418:18:21:33:026332 gpstart:rhmdw:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait...
....
20190418:18:21:37:026332 gpstart:rhmdw:gpadmin-[INFO]:-Process results...
20190418:18:21:37:026332 gpstart:rhmdw:gpadmin-[INFO]:-----------------------------------------------------
20190418:18:21:37:026332 gpstart:rhmdw:gpadmin-[INFO]:-   Successful segment starts                                            = 4
20190418:18:21:37:026332 gpstart:rhmdw:gpadmin-[INFO]:-   Failed segment starts                                                = 0
20190418:18:21:37:026332 gpstart:rhmdw:gpadmin-[INFO]:-   Skipped segment starts (segments are marked down in configuration)   = 0
20190418:18:21:37:026332 gpstart:rhmdw:gpadmin-[INFO]:-----------------------------------------------------
20190418:18:21:37:026332 gpstart:rhmdw:gpadmin-[INFO]:-Successfully started 4 of 4 segment instances
20190418:18:21:37:026332 gpstart:rhmdw:gpadmin-[INFO]:-----------------------------------------------------
20190418:18:21:37:026332 gpstart:rhmdw:gpadmin-[INFO]:-Starting Master instance rhmdw directory /gp/gpdata/master/gpseg-1
20190418:18:21:38:026332 gpstart:rhmdw:gpadmin-[INFO]:-Command pg_ctl reports Master rhmdw instance active
20190418:18:21:38:026332 gpstart:rhmdw:gpadmin-[INFO]:-Starting standby master
20190418:18:21:38:026332 gpstart:rhmdw:gpadmin-[INFO]:-Checking if standby master is running on host: rhsdw1 in directory: /gp/gpdata/master/gpseg-1
20190418:18:21:42:026332 gpstart:rhmdw:gpadmin-[INFO]:-Database successfully started

查看集群状态：
[gpadmin@rhmdw ~]$ gpstate -m
20190418:18:22:48:026594 gpstate:rhmdw:gpadmin-[INFO]:-Starting gpstate with args: -m
20190418:18:22:48:026594 gpstate:rhmdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44'
20190418:18:22:48:026594 gpstate:rhmdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Jan 16 2019 02:32:15'
20190418:18:22:49:026594 gpstate:rhmdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20190418:18:22:49:026594 gpstate:rhmdw:gpadmin-[INFO]:--------------------------------------------------------------
20190418:18:22:49:026594 gpstate:rhmdw:gpadmin-[INFO]:--Current GPDB mirror list and status
20190418:18:22:49:026594 gpstate:rhmdw:gpadmin-[INFO]:--Type = Spread
20190418:18:22:49:026594 gpstate:rhmdw:gpadmin-[INFO]:--------------------------------------------------------------
20190418:18:22:49:026594 gpstate:rhmdw:gpadmin-[INFO]:-   Mirror   Datadir                    Port   Status    Data Status
20190418:18:22:49:026594 gpstate:rhmdw:gpadmin-[INFO]:-   rhsdw2   /gp/gpdata/mirror/gpseg0   7000   Passive   Synchronized
20190418:18:22:49:026594 gpstate:rhmdw:gpadmin-[INFO]:-   rhsdw1   /gp/gpdata/mirror/gpseg1   7000   Passive   Synchronized
20190418:18:22:49:026594 gpstate:rhmdw:gpadmin-[INFO]:--------------------------------------------------------------

[gpadmin@rhmdw ~]$ gpstate -b
20190418:18:21:47:026494 gpstate:rhmdw:gpadmin-[INFO]:-Starting gpstate with args: -b
20190418:18:21:47:026494 gpstate:rhmdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44'
20190418:18:21:47:026494 gpstate:rhmdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.16.0 build commit:23cec7df0406d69d6552a4bbb77035dba4d7dd44) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Jan 16 2019 02:32:15'
20190418:18:21:47:026494 gpstate:rhmdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20190418:18:21:47:026494 gpstate:rhmdw:gpadmin-[INFO]:-Gathering data from segments...
..
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-Greenplum instance status summary
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-----------------------------------------------------
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Master instance                                           = Active
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Master standby                                            = rhsdw1
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Standby master state                                      = Standby host passive
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total segment instance count from metadata                = 4
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-----------------------------------------------------
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Primary Segment Status
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-----------------------------------------------------
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total primary segments                                    = 2
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total primary segment valid (at master)                   = 2
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total primary segment failures (at master)                = 0
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total number of postmaster.pid files missing              = 0
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total number of postmaster.pid files found                = 2
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total number of postmaster.pid PIDs missing               = 0
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total number of postmaster.pid PIDs found                 = 2
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total number of /tmp lock files missing                   = 0
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total number of /tmp lock files found                     = 2
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total number postmaster processes missing                 = 0
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total number postmaster processes found                   = 2
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-----------------------------------------------------
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Mirror Segment Status
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-----------------------------------------------------
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total mirror segments                                     = 2
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total mirror segment valid (at master)                    = 2
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total mirror segment failures (at master)                 = 0
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total number of postmaster.pid files missing              = 0
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total number of postmaster.pid files found                = 2
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total number of postmaster.pid PIDs missing               = 0
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total number of postmaster.pid PIDs found                 = 2
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total number of /tmp lock files missing                   = 0
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total number of /tmp lock files found                     = 2
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total number postmaster processes missing                 = 0
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total number postmaster processes found                   = 2
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total number mirror segments acting as primary segments   = 0
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-   Total number mirror segments acting as mirror segments    = 2
20190418:18:21:49:026494 gpstate:rhmdw:gpadmin-[INFO]:-----------------------------------------------------

恢复删除数据
[gpadmin@rhsdw03 gpdata]$ scp gp_dump_* rhmdw:/gp/gpdata/rhsdw03/
gp_dump_2_7_20190418180809                                                                                                 100% 794   377.8KB/s   00:00
gp_dump_2_7_20190418180823                                                                                                 100% 448   221.1KB/s   00:00
gp_dump_2_7_20190418180835                                                                                                 100%   41KB   8.3MB/s   00:00
gp_dump_status_2_7_20190418180809                                                                                          100% 1291   604.0KB/s   00:00
gp_dump_status_2_7_20190418180823                                                                                          100% 1294    82.8KB/s   00:00
gp_dump_status_2_7_20190418180835                                                                                          100% 1295   683.6KB/s   00:00
[gpadmin@rhsdw03 gpdata]$
[gpadmin@rhmdw rhsdw03]$ psql postgres -f gp_dump_2_7_20190418180823
SET
SET
SET
SET
SET
SET
SET
[gpadmin@rhmdw rhsdw03]$ psql gpperfmon -f gp_dump_2_7_20190418180835
SET
SET
SET
SET
SET
SET
SET
setval
--------
      1
(1 row)