Pacemake High Availability Cluster on CentOS / RHEL 8
In our infrastructure services we used Linux pacemaker cluster a lot to provide load balancing and fault tolerance. This provide great features to setup our production application so that we could make them available in any case.
So it’s always good for any Linux administrator to have knowledge and understanding about pacemaker cluster. In this post we check How we can configure pacemaker Cluster in CentOS8. Earlier we wrote some post for previous version as well.
In this setup we will how to configure Pacemaker cluster to provide fault tolerance for Apache Service.
We will have two Apache Servers (One active and one passive) and Active Apache server would have virtual IP which has IP based hosting configured on Apache configuration, So whenever active machine went down , IP and service will automatically move to passive machine.
[root@srv18 ~]# cat /etc/redhat-release CentOS Linux release 8.1.1911 (Core) [root@srv18 ~]# uname -r 4.18.0-147.8.1.el8_1.x86_64 ======================================= [root@srv18 ~]# pacemakerd --version Pacemaker 2.0.2-3.el8_1.2 Written by Andrew Beekhof [root@srv18 ~]# corosync -v Corosync Cluster Engine, version '3.0.2' Copyright (c) 2006-2018 Red Hat, Inc. [root@srv18 ~]# pcs --version 0.10.2 ======================================= Server version: Apache/2.4.37 (centos) Server built: Dec 23 2019 20:45:34
We are using above packages version. For this setup we are using two Linux machines
srv18.geekpills.com 192.168.0.18 srv16.geekpills.com 192.168.0.16 #Virtual Apache IP testapache.geekpills.com 192.168.0.50 # Shared Storage # SAN or ISCSI targets
Install of Cluster packages
To start configure pacemaker cluster, first we need to install Cluster packages on CentOS8 Machine. In CentOS8 Machines, we need enabled HA repositories first. So Question raise, how to get know which all repositories we have and what need to install for pacemaker cluster.
To know what all repositories installed in your machine, you need to run below command.
# yum repolist all
So , if we grep for HighAvailability. It will show you below repository.
[root@srv16 ~]# yum repolist all | grep -i HighAvailability Last metadata expiration check: 0:14:21 ago on Tue 09 Jun 2020 12:21:57 AM BST. HighAvailability CentOS-8 - HA disabled
So right now , it is disable. In case we need to install we need to enable this repository. Before enable it, let’s see how can we list all packages available on this repository.
dnf repository-packages HighAvailability list
With above command we can see available packages in this repository. Let’s see how we can enable this repository to install these packages on both machines.
Enable Cluster repository
[root@srv16 ~]# yum install pcs pacemaker fence-agents-common Last metadata expiration check: 0:23:00 ago on Tue 09 Jun 2020 12:21:57 AM BST. No match for argument: pcs No match for argument: pacemaker Error: Unable to find a match: pcs pacemaker [root@srv16 ~]# dnf config-manager --set-enabled HighAvailability [root@srv16 ~]# yum repolist all | grep -i HighAvailability Last metadata expiration check: 0:00:58 ago on Tue 09 Jun 2020 12:45:15 AM BST. HighAvailability CentOS-8 - HA enabled: 99
In above command, first we try to install without enable repo, but it said “no match for argument” Now we can install pacemaker packages.
# yum install pcs pacemaker fence-agents-common ....................... ...Output-truncated.... ....................... Transaction Summary =============================== Install 80 Packages Total download size: 39 M Installed size: 112 M Is this ok [y/N]: y
Configuration of pacemaker
Now we can start configuration of pacemaker. Let’s start working on same.
Enable Cluster Network
Before start configuration between nodes, please get exception through firewalld.
# firewall-cmd --permanent --add-service=high-availability # firewall-cmd --add-service=high-availability # firewall-cmd --reload
Please execute above command, so that cluster network will communicate between machines.
Authorize Cluster user
With Cluster packages, there is one user created hacluster, which would further used in cluster configuration, we need to set password for this user on both machines.
[root@srv18 ~]# passwd hacluster Changing password for user hacluster. New password: BAD PASSWORD: The password contains the user name in some form Retype new password: passwd: all authentication tokens updated successfully.
I suggest, please use same password on both machine for this user.
Start and enable pcs service on both machine like below.
[root@srv18 ~]# systemctl start pcsd [root@srv18 ~]# systemctl enable pcsd Created symlink /etc/systemd/system/multi-user.target.wants/pcsd.service → /usr/lib/systemd/system/pcsd.service.
We need to authorised hacluster user from both machine, so further we can create cluster and these machine can connect each other without any difficulties.
[root@srv16 ~]# pcs host auth srv16 srv18 Username: hacluster Password: srv16: Authorized srv18: Authorized
Both nodes are communicated and authorised each other for further configuration of cluster between two nodes.
Creation of Cluster
Let’s create cluster, like setup cluster with one name and included nodes in that, like below.
[root@srv16 ~]# pcs cluster setup --start Apache_cluster srv16 srv18 No addresses specified for host 'srv16', using 'srv16' No addresses specified for host 'srv18', using 'srv18' Destroying cluster on hosts: 'srv16', 'srv18'... srv16: Successfully destroyed cluster srv18: Successfully destroyed cluster Requesting remove 'pcsd settings' from 'srv16', 'srv18' srv16: successful removal of the file 'pcsd settings' srv18: successful removal of the file 'pcsd settings' Sending 'corosync authkey', 'pacemaker authkey' to 'srv16', 'srv18' srv16: successful distribution of the file 'corosync authkey' srv16: successful distribution of the file 'pacemaker authkey' srv18: successful distribution of the file 'corosync authkey' srv18: successful distribution of the file 'pacemaker authkey' Sending 'corosync.conf' to 'srv16', 'srv18' srv16: successful distribution of the file 'corosync.conf' srv18: successful distribution of the file 'corosync.conf' Cluster has been successfully set up. Starting cluster on hosts: 'srv16', 'srv18'...
Let’s enable it on both nodes.
[root@srv16 ~]# pcs cluster enable --all srv16: Cluster Enabled srv18: Cluster Enabled
Cluster is created , now we can check cluster status, with below command.
[root@srv16 ~]# pcs status Cluster name: Apache_cluster WARNINGS: No stonith devices and stonith-enabled is not false Stack: corosync Current DC: srv18 (version 2.0.2-3.el8_1.2-744a30d655) - partition with quorum Last updated: Tue Jun 9 02:17:39 2020 Last change: Tue Jun 9 02:16:24 2020 by hacluster via crmd on srv18 2 nodes configured 0 resources configured Online: [ srv16 srv18 ] No resources Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled
We are not going to use stonith in this cluster as we are using KVM Guest machine for building Cluster, which don’t support any hardware stonith device. But sure will cover some soft stonith features in another post like sbd (Stonith block device) devices
In above, we can see there is one Warning related to “stonith-enabled is no false” , let’s make it false.
[root@srv16 ~]# pcs property set stonith-enabled=false
Once it’s disable, this warning message will disappear from Cluster status command.
[root@srv16 ~]# pcs status Cluster name: Apache_cluster Stack: corosync Current DC: srv18 (version 2.0.2-3.el8_1.2-744a30d655) - partition with quorum Last updated: Tue Jun 9 02:23:33 2020 Last change: Tue Jun 9 02:23:31 2020 by root via cibadmin on srv16 2 nodes configured 0 resources configured Online: [ srv16 srv18 ] No resources Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled
Creation of Cluster resource
Now we can create some resource on this cluster.
Let’s first create one virtual IP resource on this cluster.
So we planning to create a virtual IP which is grouped with Apache service, So Apache service will always move with same and So user will able to connect that IP to access Apache Web Page.
pcs resource create Apache_vip ocf:heartbeat:IPaddr2 ip=192.168.0.50 cidr_netmask=24 --group Apache_Grp
Once we add this resource, we can see this resource on both host Cluster configuration.
root@srv16 ~]# pcs resource Resource Group: Apache_Grp Apache_vip (ocf::heartbeat:IPaddr2): Started srv16
Virtual floating IP has been configured on this machine through Cluster.
[root@srv16 ~]# ip -4 a show dev ens2 2: ens2:
mtu 1500 qdisc fq_codel state UP group default qlen 1000 inet 192.168.0.16/24 brd 192.168.0.255 scope global dynamic noprefixroute ens2 valid_lft 82564sec preferred_lft 82564sec inet 192.168.0.50/24 brd 192.168.0.255 scope global secondary ens2 valid_lft forever preferred_lft forever
Create Apache Service
We also need to add Apache service on Cluster which also grouped in same Cluster group and then we can move both resource on single node.
This way Apache service with always be available for end-users.
# pcs resource create apache_ser service:httpd --group Apache_Grp
After this we can see both resource on pcs status configuration.
[root@srv16 ~]# pcs resource Resource Group: Apache_Grp Apache_vip (ocf::heartbeat:IPaddr2): Started srv16 apache_ser (service:httpd): Started srv16
We can also see complete configuration like below way.
# pcs config show
Also need to open Apache/SSL ports on both nodes, only then we can able to access Apache service.
# firewall-cmd --permanent --add-port=80/tcp # firewall-cmd --permanent --add-port=443/tcp # firewall-cmd --reload
With above command, we will able to access web-page which is hosted on 192.168.0.50.
Now we have cluster configured with Floating IP and Apache service which could float on healthy machine and locate there till find any issues in that machine.
Move resource between nodes
We can move resource between hosts like below in case we need safe movement nodes before any activity.
[root@srv16 ~]# pcs resource Resource Group: Apache_Grp Apache_vip (ocf::heartbeat:IPaddr2): Started srv16 apache_ser (service:httpd): Started srv16 [root@srv16 ~]# pcs resource move Apache_Grp srv18 [root@srv16 ~]# pcs resource Resource Group: Apache_Grp Apache_vip (ocf::heartbeat:IPaddr2): Started srv18 apache_ser (service:httpd): Started srv18