I have scenario, in which I need to add one new node in pcs cluster and delete on previous node which need to ship in some other Datacenter for some other stuff.Addition and deleting of Nodes in PCS Cluster RHEL6

So we have scenario in which We are using RHEL6 machine update as of now June 2018, with below configuration.

SetUP

[root@cluster1 ~]# pcs --version
0.9.155

[root@cluster1 ~]# corosync -v
Corosync Cluster Engine, version '1.4.7'
Copyright (c) 2006-2009 Red Hat, Inc.

[root@cluster1 ~]# pacemakerd --version
Pacemaker 1.1.15-5.el6
Written by Andrew Beekhof

[root@cluster1 ~]# cat /etc/redhat-release 
CentOS release 6.9 (Final)

[root@cluster1 ~]# uname  -r
2.6.32-696.30.1.el6.x86_64

So we are using two machine cluster1 and cluster2 with below cluster configuration. We are using two service to host through cluster Apache (httpd) and FTP (vsftpd)

[root@cluster1 ~]# pcs status
Cluster name: Apache
Stack: cman
Current DC: cluster2 (version 1.1.15-5.el6-e174ec8) - partition with quorum
Last updated: Sat Jun 23 09:33:08 2018		Last change: Sat Jun 23 09:02:27 2018 by root via crmd on cluster2

2 nodes and 4 resources configured

Online: [ cluster1 cluster2 ]

Full list of resources:

 Resource Group: Group_vsftpd
     vsftpd_vip	(ocf::heartbeat:IPaddr2):	Started cluster2
     vsftpd	(lsb:vsftpd):	Started cluster2
 Resource Group: Group_apache
     apache_vip	(ocf::heartbeat:IPaddr2):	Started cluster1
 apache	(ocf::heartbeat:apache):	Started cluster1

Daemon Status:
  cman: active/disabled
  corosync: active/disabled
  pacemaker: active/enabled
  pcsd: active/enabled

Now about details configuration of PCS Cluster is like below

[root@cluster1 ~]# pcs config show
Cluster Name: Apache
Corosync Nodes:
 cluster2 cluster1
Pacemaker Nodes:
 cluster1 cluster2

Resources:
 Group: Group_vsftpd
  Resource: vsftpd_vip (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: ip=192.168.123.220 cidr_netmask=24
   Operations: start interval=0s timeout=20s (vsftpd_vip-start-interval-0s)
               stop interval=0s timeout=20s (vsftpd_vip-stop-interval-0s)
               monitor interval=10s timeout=20s (vsftpd_vip-monitor-interval-10s)
  Resource: vsftpd (class=lsb type=vsftpd)
   Operations: start interval=0s timeout=15 (vsftpd-start-interval-0s)
               stop interval=0s timeout=15 (vsftpd-stop-interval-0s)
               monitor interval=15 timeout=15 (vsftpd-monitor-interval-15)
 Group: Group_apache
  Resource: apache_vip (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: ip=192.168.123.200 cidr_netmask=24
   Operations: start interval=0s timeout=20s (apache_vip-start-interval-0s)
               stop interval=0s timeout=20s (apache_vip-stop-interval-0s)
               monitor interval=10s timeout=20s (apache_vip-monitor-interval-10s)
 Resource: apache (class=ocf provider=heartbeat type=apache)
  Operations: start interval=0s timeout=40s (apache-start-interval-0s)
              stop interval=0s timeout=60s (apache-stop-interval-0s)
              monitor interval=10 timeout=20s (apache-monitor-interval-10)

Stonith Devices:
Fencing Levels:

Location Constraints:
  Resource: Group_apache
    Enabled on: cluster1 (score:INFINITY) (role: Started) (id:cli-prefer-Group_apache)
  Resource: Group_vsftpd
    Enabled on: cluster2 (score:INFINITY) (role: Started) (id:cli-prefer-Group_vsftpd)
  Resource: apache
    Enabled on: cluster1 (score:INFINITY) (role: Started) (id:cli-prefer-apache)
Ordering Constraints:
  start vsftpd_vip then start vsftpd (kind:Mandatory) (id:order-vsftpd_vip-vsftpd-mandatory)
  start apache_vip then start apache (kind:Mandatory) (id:order-apache_vip-apache-mandatory)
Colocation Constraints:
Ticket Constraints:

Alerts:
 No alerts defined

Resources Defaults:
 migration-threshold: 3
Operations Defaults:
 No defaults set

Cluster Properties:
 cluster-infrastructure: cman
 dc-version: 1.1.15-5.el6-e174ec8
 enable-acl: false
 have-watchdog: false
 last-lrm-refresh: 1529606868
 stonith-enabled: false

Now as per requirement, we need to add third node in cluster and delete one previous node from it. I hope you guys aware how to create Custer, I have earlier wrote one page for same on RHEL7, which has same setup and commands as of RHEL6 expect how to run service through init.d/systemd.

So first we need to understand setup.

OLD Setup
Cluster1  ---  Cluster2

NEW Setup
Cluster2  ---  Cluster3

So first we need to prepare New Cluster node before adding it cluster. So we have cluster3 node, which has same setup as we had with earlier two nodes.

Need to Install pcs packages

yum install pcs pacemaker-y

Install resource packages

Need to install required service packages on New cluster node.I request you to please care on this part. It could cause failcount infinity, which will not allow resource to run on particular node.

yum install httpd vsftpd -y

Assign password – hacluster User

Assign password to hacluster user, i suggest to have same password on all nodes.

[root@cluster3 ~]# passwd hacluster
Changing password for user hacluster.
New password: 
BAD PASSWORD: it is based on a dictionary word
Retype new password: 
passwd: all authentication tokens updated successfully.

Identify with FQDN

I suggest you that machine should able to resolve FQDN/Hostname of each other and able to ping with same. Either you have DNS or better to mentioned IP Address and Hostname in /etc/hosts file like below.

192.168.123.78	cluster1.geekpills.com	cluster1
192.168.123.88 	cluster2.geekpills.com	cluster2
192.168.123.98	cluster3.geekpills.com  cluster3

Now we could start Adding new node in cluster.

Start pcs service on new node

We need to start pcs service, so we could communicate with other while they try to authenticate this machine in below step

[root@cluster3 ~]# service  pcsd start
Starting pcsd: [  OK  ]
[root@cluster3 ~]# chkconfig pcsd on

Authenticate New node in cluster

Now we need authenticate New node with existing node , so that further during cluster operations they could authenticate to work on same.

[root@cluster2 ~]# pcs cluster auth cluster3
Username: hacluster
Password: 
cluster3: Authorized

Adding New Cluster node

Finally we would Add New node in existing cluster with below commands.

[root@cluster2 ~]# pcs cluster node add cluster3
Disabling SBD service...
cluster3: sbd disabled
cluster2: Corosync updated
cluster1: Corosync updated
Setting up corosync...
cluster3: Updated cluster.conf...
cluster3: Starting Cluster...
Synchronizing pcsd certificates on nodes cluster3...
cluster3: Success

Restarting pcsd on the nodes in order to reload the certificates...
cluster3: Success

Check cluster status

Now we should check cluster status, which should show New Node in cluster.

[root@cluster1 ~]# pcs status
Cluster name: Apache
Stack: cman
Current DC: cluster2 (version 1.1.15-5.el6-e174ec8) - partition with quorum
Last updated: Sat Jun 23 10:26:53 2018		Last change: Sat Jun 23 10:21:41 2018 by root via crmd on cluster2

3 nodes and 4 resources configured

Online: [ cluster1 cluster2 cluster3 ]

Full list of resources:

 Resource Group: Group_vsftpd
     vsftpd_vip	(ocf::heartbeat:IPaddr2):	Started cluster2
     vsftpd	(lsb:vsftpd):	Started cluster2
 Resource Group: Group_apache
     apache_vip	(ocf::heartbeat:IPaddr2):	Started cluster1
 apache	(ocf::heartbeat:apache):	Started cluster1

Daemon Status:
  cman: active/disabled
  corosync: active/disabled
  pacemaker: active/enabled
  pcsd: active/enabled

We also enable cluster on this node otherwise, it will cause issues during reboot of this node. We could enable specific node or nodes or all nodes like below

[root@cluster2 ~]# pcs cluster enable cluster2 cluster3
cluster2: Cluster Enabled
cluster3: Cluster Enabled

	or 

[root@cluster2 ~]# pcs cluster enable cluster3
cluster3: Cluster Enabled

	or

[root@cluster2 ~]# pcs cluster enable --all
cluster2: Cluster Enabled
cluster1: Cluster Enabled
cluster3: Cluster Enabled

Now we can delete previous node from cluster to move in some other project.

Removal on existing node

Now we need to remove cluster node like below.

[root@cluster2 ~]#  colpcs cluster disable cluster1
cluster1: Cluster Disabled

[root@cluster2 ~]# pcs cluster node delete cluster1
cluster1: Stopping Cluster (pacemaker)...
dlm: closing connection to node 1
cluster1: Successfully destroyed cluster
cluster2: Corosync updated
cluster3: Corosync updated

This way we have Add New node and Remove on exiting node from Cluster. Now cluster should show as per configure above only cluster2 and cluster3 machines in pcs status

[root@cluster2 ~]# pcs status
Cluster name: Apache
Stack: cman
Current DC: cluster2 (version 1.1.15-5.el6-e174ec8) - partition with quorum
Last updated: Sat Jun 23 10:42:53 2018		Last change: Sat Jun 23 10:38:24 2018 by root via crm_node on cluster2

2 nodes and 4 resources configured

Online: [ cluster2 cluster3 ]

Full list of resources:

 Resource Group: Group_vsftpd
     vsftpd_vip	(ocf::heartbeat:IPaddr2):	Started cluster2
     vsftpd	(lsb:vsftpd):	Started cluster2
 Resource Group: Group_apache
     apache_vip	(ocf::heartbeat:IPaddr2):	Started cluster3
 apache	(ocf::heartbeat:apache):	Started cluster3

Daemon Status:
  cman: active/disabled
  corosync: active/disabled
  pacemaker: active/enabled
  pcsd: active/enabled