Build Two-node cluster with Heartbeat

Here is the link for two-node cluster with heartbeat on Linux Journal

I was asked to build two-node cluster with heartbeat on two remote servers.

I don't event know what is cluster,heartbeat before I install it on remote servers. I am very new to Linux.

Here's the problems I encountered when I try to install, this article is first to solve those 4 problems, and secondly to show all steps to setup two-node cluster with heartbeat[Second part is not finished yet]

1. What is cluster and heartbeat, why we use it?

2.  How to setup IP Aliasing(virtual IP)?

3.  How to install EPEL Repo on CentOS5.X or CentOS6.X?

Now, I will solve those problems one by one.

1. What is cluster and heartbeat, why we use it?

This problems is quit easy. 

A computer cluster consists of a set of loosely connected computers that work together so that in many respects they can be viewed as a single system.

The components of a cluster are usually connected to each other through fast local area networks, each node (computer used as a server) running its own instance of an operating system. Computer clusters emerged as a result of convergence of a number of computing trends including the availability of low cost microprocessors, high speed networks, and software for high performance distributed computing.

Clusters are usually deployed to improve performance and availability over that of a single computer, while typically being much more cost-effective than single computers of comparable speed or availability.[resources from Wikipedia]

In computer clusters, heartbeat network is a private network which is shared only by the cluster nodes, and is not accessible from outside the cluster. It is used by cluster nodes in order to monitor each node's status and communicate with each other.[resources from Wikipedia]

For more information, check here.

2.  How to setup IP Aliasing(virtual IP)?

IP aliasing is associating more than one IP address to a network interface. With this, one node on a network can have multiple connections to a network, each serving a different purpose.[resources from Wikipedia]

My system is CentOS, now we start to setup ip aliasing:

# cd /etc/sysconfig/network-scripts/

now create a file named "ifcfg-eth0:0". The reason why you create this file is that in order to create an alias for the interface, you need to create a file in the format ifcfg-ethX:Y - where X is the interface number and Y represents the alias number. The first alias would be numbered 0, the next alias 1, and so forth. For the first alias on the first interface, you would create a file called ifcfg-eth0:0.

Add the following to ifcfg-eth0:0

DEVICE=eth0:0
BOOTPROTO=dhcp
ONBOOT=yes
IPADDR="192.168.111.111"
NETMASK="255.255.255.0"

IPADDR is your virtual ip address, for example: 192.168.111.111

In order to apply the new ip address, you can restart the full network or use ifup to bring up the ip alias interface:

# service network restart   

OR

#ifup eth0:0

To verify everything is okay, use the following command

# ifconfig

to list all active interfaces, it will look like this:

eth0      Link encap:Ethernet  HWaddr XX:XX:XX:XX:XX:XX
          inet addr:XXX.XXX.XXX.XX  Bcast:XXX.XXX.XXX.255  Mask:255.255.255.0
          inet6 addr: XXX::XXX:XXXX:XXXX:XXXX/XX Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1672376185 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1830099250 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1022777289 (975.3 MiB)  TX bytes:111054671 (105.9 MiB)

eth0:0    Link encap:Ethernet  HWaddr XX:XX:XX:XX:XX:XX
          inet addr:192.168.111.111  Bcast:XXX.XXX.XXX.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

You can also use "ping" to check the new ip adderss works or not.

Lets go to question 3.

3.  How to install EPEL Repo on CentOS5.X or CentOS6.X?

This took me several hours to install heartbeat on CentOS6.X, I tried "yum install heartbeat", it failed and return Error, Nothing to do. This is because I did not download EPEL Repo. Once EPEL Repo was downloaded, all problems would be solved.Here is the instructions:

Centos 5.x
wget http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
wget http://rpms.famillecollet.com/enterprise/remi-release-5.rpm
sudo rpm -Uvh remi-release-5*.rpm epel-release-5*.rpm
Centos 6.x
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-7.noarch.rpm
wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
sudo rpm -Uvh remi-release-6*.rpm epel-release-6*.rpm
Once installed you should see some additional repo definitions under the /etc/yum.repos.d directory.
$ ls -1 /etc/yum.repos.d/epel* /etc/yum.repos.d/remi.repo
/etc/yum.repos.d/epel.repo
/etc/yum.repos.d/epel-testing.repo
/etc/yum.repos.d/remi.repo
Enable the remi repository
The remi repository provides a variety of up-to-date packages that are useful or are a requirement for many popular web-based services.  That means it generally is not a bad idea to enable the remi repositories by default.
First, open the /etc/yum.repos.d/remi.repo repository file using a text editor of your choice:
#sudo vim /etc/yum.repos.d/remi.repo
Edit the [remi] portion of the file so that the enabled option is set to 1.  This will enable the remi repository.
name=Les RPM de remi pour Enterprise Linux $releasever - $basearch
#baseurl=http://rpms.famillecollet.com/enterprise/$releasever/remi/$basearch/
mirrorlist=http://rpms.famillecollet.com/enterprise/$releasever/remi/mirror
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-remi
failovermethod=priority

Okay, so far question 3 is solved.

For more information, check here .

---------------------------------------------------------------------------------------------------------------------------------

Configuration of Two-Node Cluster with HeartBeat on CentOS

Pre-configuration requirements:

     1. hostnames for your two nodes.

      setup hostname  in the file "hosts", it is under /etc/hosts

      for example, I have two remote servers cat and dog. configure the hosts file for both servers.

       then use command "uname -n" for each of them.

       if server1 returns "cat",  use cat, if it returns "mycat", use mycat; the same for dog.

     2. ip aliasing

       we only need to setup ip aliasing for your primary node(master), don't do anything about ip aliasing on your secondary node(slave). [we've talked about how to setup IP aliasing at the beginning of this tutorial].

Configuration:
     1. Download and install heartbeat packages .In my case I use CentOS so install heartbeat with yum:
      # yum install heartbeat
      [if  an ERROR occurred and said Nothing to do, see How to install EPEL Repo at the beginning of this tutorial]
     2. Before configure heartbeat files , we need to move them( ha.cf, authkeys and haresources) to /etc/ha.d

     #cp /usr/share/doc/heartbeat-3.0.4/ha.cf /etc/ha.d/

     #cp /usr/share/doc/heartbeat-3.0.4/authkeys /etc/ha.d/
     #cp /usr/share/doc/heartbeat-3.0.4/haresources /etc/ha.d/

     3. Now, lets start to configure them one by one:

        (1)Lets configure ha.cf first:

            #sudo vim /etc/ha.d/ha.cf

            Add the following lines to ha.cf

            logfile /var/log/ha-log
            logfacility local0
            keepalive 2
            deadtime 10
            initdead 120
            bcast eth0
            udpport 694
            auto_failback on
            node name_of_node1
            node name_of_node2

        (2) configure authkeys:

            #sudo vim /etc/ha.d/authkeys

            Add the following lines to it:
             auth 2
             2 sha1 test-ha

           then, change the permission for authkeys:

           # sudo chmod 600 /etc/ha.d/authkeys

        (3) configure haresources:

           #sudo vim /etc/ha.d/haresources

           Add the following lines to it:

           cat 192.168.111.111/24/eth0 mysqld

           cat is my primary server name(name of node 1), 192.168.111.111 is my IP Aliasing(virtual ip).

           24 is the port number we use, mysqld is the service we want to run.

Configuration is done so far, make sure the content of authkeys and haresources files are the same on both servers.

Testing  whether two-node cluster with heartbeat works properly or not.

      1.  Turn off your firewall on both servers of just modify iptables and let it allow 649 go through[Usually the former way is better].

      2.  Monitor the log files of heartbeat using the following command, we setup the log files in our ha.cf file.

         # tail -f /var/log/ha-log

      3.  Start primary server heartbeat and then start secondary server heartbeat using

         # sudo service heartbeat start

      Then, it may show something looks like this :Local status now set to: 'up'

       Now wait several seconds [at most 3 minutes], it will show the service is started. 

     4. Stop heartbeat on primary server using following command

        # sudo service heartbeat stop

         Now the service on secondary server will be activated.

    5 . Check the service/process is starting  using:

        # ps -ef | grep mysqld

         mysqld is the service/process name

I get more problems about heartbeat:

1. Message hist queue is filling up.

2. Both machines own our resources.

Those two problems occurred due to the firewall in most cases, so make sure firewall is closed on both

servers using:

# sudo service iptables stop