In our infrastructure services we used Linux pacemaker cluster a lot to provide load balancing and fault tolerance. This provide great features to setup our production application so that we could make them available in any case.

So it’s always good for any Linux administrator to have knowledge and understanding about pacemaker cluster. In this post we check How we can configure pacemaker Cluster in CentOS8. Earlier we wrote some post for previous version as well.

Configure pacemake Cluster in centOS

SetUP

In this setup we will how to configure Pacemaker cluster to provide fault tolerance for Apache Service.
We will have two Apache Servers (One active and one passive) and Active Apache server would have virtual IP which has IP based hosting configured on Apache configuration, So whenever active machine went down , IP and service will automatically move to passive machine.

Using CentOS8

[root@srv18 ~]# cat /etc/redhat-release  
CentOS Linux release 8.1.1911 (Core)     
[root@srv18 ~]# uname -r                 
4.18.0-147.8.1.el8_1.x86_64              

=======================================

[root@srv18 ~]# pacemakerd --version             
Pacemaker 2.0.2-3.el8_1.2                        
Written by Andrew Beekhof                        
[root@srv18 ~]# corosync -v                      
Corosync Cluster Engine, version '3.0.2'         
Copyright (c) 2006-2018 Red Hat, Inc.            
[root@srv18 ~]# pcs --version                    
0.10.2                                 

=======================================

Server version: Apache/2.4.37 (centos)
Server built:   Dec 23 2019 20:45:34  

We are using above packages version. For this setup we are using two Linux machines

srv18.geekpills.com   192.168.0.18

srv16.geekpills.com   192.168.0.16

#Virtual Apache IP

testapache.geekpills.com  192.168.0.50

# Shared Storage
# SAN or ISCSI targets

Install of Cluster packages

To start configure pacemaker cluster, first we need to install Cluster packages on CentOS8 Machine. In CentOS8 Machines, we need enabled HA repositories first. So Question raise, how to get know which all repositories we have and what need to install for pacemaker cluster.

To know what all repositories installed in your machine, you need to run below command.

# yum repolist all

So , if we grep for HighAvailability. It will show you below repository.

[root@srv16 ~]# yum repolist all | grep -i HighAvailability                        
Last metadata expiration check: 0:14:21 ago on Tue 09 Jun 2020 12:21:57 AM BST.    
HighAvailability               CentOS-8 - HA                      disabled         

So right now , it is disable. In case we need to install we need to enable this repository. Before enable it, let’s see how can we list all packages available on this repository.

dnf repository-packages  HighAvailability list

With above command we can see available packages in this repository. Let’s see how we can enable this repository to install these packages on both machines.

Enable Cluster repository

[root@srv16 ~]# yum install pcs pacemaker fence-agents-common                            
Last metadata expiration check: 0:23:00 ago on Tue 09 Jun 2020 12:21:57 AM BST.          
No match for argument: pcs                                                               
No match for argument: pacemaker                                                         
Error: Unable to find a match: pcs pacemaker                                             

[root@srv16 ~]#  dnf config-manager --set-enabled HighAvailability   

[root@srv16 ~]# yum repolist all | grep -i HighAvailability                        
Last metadata expiration check: 0:00:58 ago on Tue 09 Jun 2020 12:45:15 AM BST.    
HighAvailability               CentOS-8 - HA                      enabled:    99   

In above command, first we try to install without enable repo, but it said “no match for argument” Now we can install pacemaker packages.

# yum install pcs pacemaker fence-agents-common
.......................
...Output-truncated....
.......................

Transaction Summary            
===============================
Install  80 Packages           
                               
Total download size: 39 M      
Installed size: 112 M          
Is this ok [y/N]: y            

Configuration of pacemaker

Now we can start configuration of pacemaker. Let’s start working on same.

Enable Cluster Network

Before start configuration between nodes, please get exception through firewalld.

# firewall-cmd --permanent --add-service=high-availability   

# firewall-cmd --add-service=high-availability               

# firewall-cmd --reload      

Please execute above command, so that cluster network will communicate between machines.

Authorize Cluster user

With Cluster packages, there is one user created hacluster, which would further used in cluster configuration, we need to set password for this user on both machines.

[root@srv18 ~]# passwd hacluster     
Changing password for user hacluster.                               
New password:                                                       
BAD PASSWORD: The password contains the user name in some form      
Retype new password:                                                
passwd: all authentication tokens updated successfully.             

I suggest, please use same password on both machine for this user.

Start and enable pcs service on both machine like below.

[root@srv18 ~]# systemctl start pcsd

[root@srv18 ~]# systemctl enable pcsd  
Created symlink /etc/systemd/system/multi-user.target.wants/pcsd.service → /usr/lib/systemd/system/pcsd.service.     

We need to authorised hacluster user from both machine, so further we can create cluster and these machine can connect each other without any difficulties.

[root@srv16 ~]# pcs host auth srv16 srv18  
Username: hacluster                        
Password:                                  
srv16: Authorized                          
srv18: Authorized                          

Both nodes are communicated and authorised each other for further configuration of cluster between two nodes.

Creation of Cluster

Let’s create cluster, like setup cluster with one name and included nodes in that, like below.

[root@srv16 ~]# pcs cluster setup --start Apache_cluster srv16 srv18  
No addresses specified for host 'srv16', using 'srv16'                
No addresses specified for host 'srv18', using 'srv18'                
Destroying cluster on hosts: 'srv16', 'srv18'...                      
srv16: Successfully destroyed cluster                                 
srv18: Successfully destroyed cluster                                 
Requesting remove 'pcsd settings' from 'srv16', 'srv18'               
srv16: successful removal of the file 'pcsd settings'                 
srv18: successful removal of the file 'pcsd settings'                 
Sending 'corosync authkey', 'pacemaker authkey' to 'srv16', 'srv18'   
srv16: successful distribution of the file 'corosync authkey'         
srv16: successful distribution of the file 'pacemaker authkey'        
srv18: successful distribution of the file 'corosync authkey'         
srv18: successful distribution of the file 'pacemaker authkey'        
Sending 'corosync.conf' to 'srv16', 'srv18'                           
srv16: successful distribution of the file 'corosync.conf'            
srv18: successful distribution of the file 'corosync.conf'            
Cluster has been successfully set up.                                 
Starting cluster on hosts: 'srv16', 'srv18'...                        

Let’s enable it on both nodes.

[root@srv16 ~]# pcs cluster enable --all   
srv16: Cluster Enabled                     
srv18: Cluster Enabled                     

Cluster is created , now we can check cluster status, with below command.

[root@srv16 ~]# pcs status                                                         
Cluster name: Apache_cluster                                                       
                                                                                   
WARNINGS:                                                                          
No stonith devices and stonith-enabled is not false                                
                                                                                   
Stack: corosync                                                                    
Current DC: srv18 (version 2.0.2-3.el8_1.2-744a30d655) - partition with quorum     
Last updated: Tue Jun  9 02:17:39 2020                                             
Last change: Tue Jun  9 02:16:24 2020 by hacluster via crmd on srv18               
                                                                                   
2 nodes configured                                                                 
0 resources configured                                                             
                                                                                   
Online: [ srv16 srv18 ]                                                            
                                                                                   
No resources                                                                       
                                                                                                                                                                
Daemon Status:                     
  corosync: active/disabled                                                        
  pacemaker: active/disabled                                                       
  pcsd: active/enabled                                                             

Disabling Stonith

We are not going to use stonith in this cluster as we are using KVM Guest machine for building Cluster, which don’t support any hardware stonith device. But sure will cover some soft stonith features in another post like sbd (Stonith block device) devices
In above, we can see there is one Warning related to “stonith-enabled is no false” , let’s make it false.

[root@srv16 ~]# pcs property set stonith-enabled=false 

Once it’s disable, this warning message will disappear from Cluster status command.

[root@srv16 ~]# pcs status                                                    
Cluster name: Apache_cluster                                                  
Stack: corosync                                                               
Current DC: srv18 (version 2.0.2-3.el8_1.2-744a30d655) - partition with quorum
Last updated: Tue Jun  9 02:23:33 2020                                        
Last change: Tue Jun  9 02:23:31 2020 by root via cibadmin on srv16           
                                                                              
2 nodes configured                                                            
0 resources configured                                                        
                                                                              
Online: [ srv16 srv18 ]                                                       
                                                                              
No resources                                                                  
                                                                                                                                                        
Daemon Status:                                                                
  corosync: active/enabled                                                    
  pacemaker: active/enabled                                                   
  pcsd: active/enabled                                                        

Creation of Cluster resource

Now we can create some resource on this cluster.
Let’s first create one virtual IP resource on this cluster.
So we planning to create a virtual IP which is grouped with Apache service, So Apache service will always move with same and So user will able to connect that IP to access Apache Web Page.

 pcs resource create Apache_vip ocf:heartbeat:IPaddr2 ip=192.168.0.50 cidr_netmask=24 --group Apache_Grp

Once we add this resource, we can see this resource on both host Cluster configuration.

root@srv16 ~]# pcs resource
Resource Group: Apache_Grp
    Apache_vip (ocf::heartbeat:IPaddr2):       Started srv16

Virtual floating IP has been configured on this machine through Cluster.

[root@srv16 ~]# ip -4 a  show dev ens2
2: ens2:  mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 192.168.0.16/24 brd 192.168.0.255 scope global dynamic noprefixroute ens2
       valid_lft 82564sec preferred_lft 82564sec
    inet 192.168.0.50/24 brd 192.168.0.255 scope global secondary ens2
       valid_lft forever preferred_lft forever

Create Apache Service

We also need to add Apache service on Cluster which also grouped in same Cluster group and then we can move both resource on single node.

This way Apache service with always be available for end-users.

# pcs resource create apache_ser service:httpd --group Apache_Grp

After this we can see both resource on pcs status configuration.

[root@srv16 ~]# pcs resource
 Resource Group: Apache_Grp
     Apache_vip (ocf::heartbeat:IPaddr2):       Started srv16
     apache_ser (service:httpd):        Started srv16

We can also see complete configuration like below way.

# pcs config show

Also need to open Apache/SSL ports on both nodes, only then we can able to access Apache service.

# firewall-cmd --permanent --add-port=80/tcp
# firewall-cmd --permanent --add-port=443/tcp
# firewall-cmd --reload

With above command, we will able to access web-page which is hosted on 192.168.0.50.

Now we have cluster configured with Floating IP and Apache service which could float on healthy machine and locate there till find any issues in that machine.

Move resource between nodes

We can move resource between hosts like below in case we need safe movement nodes before any activity.

[root@srv16 ~]# pcs resource
 Resource Group: Apache_Grp
     Apache_vip (ocf::heartbeat:IPaddr2):       Started srv16
     apache_ser (service:httpd):        Started srv16


[root@srv16 ~]# pcs resource move Apache_Grp  srv18 


[root@srv16 ~]# pcs resource
 Resource Group: Apache_Grp
     Apache_vip (ocf::heartbeat:IPaddr2):       Started srv18
     apache_ser (service:httpd):        Started srv18