Manhal.Mohamed: /* 1. DNS Update Script (/usr/local/bin/update_dns.sh) */

2025-07-22T09:38:02Z

‎1. DNS Update Script (/usr/local/bin/update_dns.sh)

Manhal.Mohamed: /* 1. DNS Update Script (/usr/local/bin/update_dns.sh) */

2025-07-22T09:36:13Z

‎1. DNS Update Script (/usr/local/bin/update_dns.sh)

Manhal.Mohamed at 09:34, 22 July 2025

2025-07-22T09:34:56Z

Show changes

Manhal.Mohamed at 09:31, 22 July 2025

2025-07-22T09:31:29Z

Manhal.Mohamed: /* Sample DNS Key File (/etc/bind/cluster.key) */

2025-07-22T09:30:20Z

‎Sample DNS Key File (/etc/bind/cluster.key)

Manhal.Mohamed: Created page with "= High Availability Cluster with Pacemaker, Chronyd, and HAProxy = '''A Comprehensive Guide for Debian Systems''' == Introduction == This guide documents the setup of a 4-no..."

2025-07-22T09:29:39Z

Created page with "= High Availability Cluster with Pacemaker, Chronyd, and HAProxy = '''A Comprehensive Guide for Debian Systems''' == Introduction == This guide documents the setup of a 4-no..."

New page

= High Availability Cluster with Pacemaker, Chronyd, and HAProxy =
'''A Comprehensive Guide for Debian Systems'''

== Introduction ==
This guide documents the setup of a 4-node high availability cluster using:
* '''Pacemaker/Corosync''' for cluster management
* '''HAProxy''' for load balancing
* '''Chronyd''' for time synchronization
* '''Automated DNS updates''' during failover scenarios

== Prerequisites ==
=== Hardware Requirements ===
* 4 identical Debian servers (≥ 2GB RAM, ≥ 2 CPU cores recommended)
* Network interfaces:
** Primary: 1Gbps (for client traffic)
** Secondary: 100Mbps minimum (for heartbeat)

=== Software Requirements ===
* Debian 10/11 (tested on both)
* Root access to all nodes
* DNS server supporting dynamic updates

=== Network Requirements ===
* Static IP assignments for all nodes
* Fully qualified domain names for each node
* Unrestricted communication on ports:
** 2224 (pcsd)
** 5404-5405 (corosync)
** 80/443 (HAProxy)

== Initial Node Setup ==
=== 1. System Preparation ===
On all nodes (node1-node4):

<syntaxhighlight lang="bash">
# Update system
sudo apt update && sudo apt upgrade -y

# Install base packages
sudo apt install -y \
pacemaker \
pcs \
corosync \
crmsh \
haproxy \
chrony \
bind9utils \
mailutils \
vim
</syntaxhighlight>

=== 2. Hosts File Configuration ===
Add to '''/etc/hosts''' on every node:
<pre>
192.168.1.101 node1.cluster.local node1
192.168.1.102 node2.cluster.local node2
192.168.1.103 node3.cluster.local node3
192.168.1.104 node4.cluster.local node4
</pre>

=== 3. Firewall Configuration ===
<syntaxhighlight lang="bash">
sudo apt install -y ufw
sudo ufw allow from 192.168.1.0/24
sudo ufw enable
</syntaxhighlight>

== Cluster Configuration ==
=== 1. Initialize Cluster Services ===
<syntaxhighlight lang="bash">
sudo systemctl enable --now pcsd corosync pacemaker
</syntaxhighlight>

=== 2. Set hacluster Password ===
<syntaxhighlight lang="bash">
echo "hacluster:SecureP@ssw0rd123" | sudo chpasswd
</syntaxhighlight>

=== 3. Authenticate Nodes (from node1) ===
<syntaxhighlight lang="bash">
sudo pcs cluster auth \
node1.cluster.local \
node2.cluster.local \
node3.cluster.local \
node4.cluster.local \
-u hacluster \
-p SecureP@ssw0rd123 \
--force
</syntaxhighlight>

=== 4. Create Cluster ===
<syntaxhighlight lang="bash">
sudo pcs cluster setup \
--name haproxy_cluster \
node1.cluster.local \
node2.cluster.local \
node3.cluster.local \
node4.cluster.local \
--force
</syntaxhighlight>

=== 5. Start Cluster ===
<syntaxhighlight lang="bash">
sudo pcs cluster start --all
sudo pcs cluster enable --all
</syntaxhighlight>

=== 6. Verify Status ===
<syntaxhighlight lang="bash">
sudo pcs status
</syntaxhighlight>
Expected output:
<pre>
Cluster name: haproxy_cluster
Stack: corosync
Current DC: node1.cluster.local (version x.x.x-x)
4 nodes configured
0 resources configured
</pre>

== HAProxy Setup ==
=== 1. Install HAProxy on All Nodes ===
<syntaxhighlight lang="bash">
sudo apt install -y haproxy
</syntaxhighlight>

=== 2. Configuration Template (/etc/haproxy/haproxy.cfg) ===
<syntaxhighlight lang="conf">
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
maxconn 4000
tune.ssl.default-dh-param 2048

defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5s
timeout client 50s
timeout server 50s
option forwardfor
option http-server-close

frontend http_front
bind *:80
bind *:443 ssl crt /etc/ssl/private/example.com.pem
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
default_backend http_back

backend http_back
balance roundrobin
cookie SERVERID insert indirect nocache
server webserver1 192.168.1.201:80 check cookie s1
server webserver2 192.168.1.202:80 check cookie s2

listen stats
bind *:1936
stats enable
stats uri /
stats hide-version
stats auth admin:SecureStatsP@ss
</syntaxhighlight>

=== 3. Add as Cluster Resource ===
<syntaxhighlight lang="bash">
sudo pcs resource create haproxy systemd:haproxy \
op monitor interval=10s timeout=20s \
op start interval=0s timeout=30s \
op stop interval=0s timeout=30s \
--group haproxy_group
</syntaxhighlight>

== Time Synchronization ==
=== 1. Configure Chronyd (/etc/chrony/chrony.conf) ===
<syntaxhighlight lang="conf">
pool 0.debian.pool.ntp.org iburst
pool 1.debian.pool.ntp.org iburst
pool 2.debian.pool.ntp.org iburst
pool 3.debian.pool.ntp.org iburst

allow 192.168.1.0/24
leapsectz right/UTC
makestep 1.0 3
</syntaxhighlight>

=== 2. Verify Time Sync ===
<syntaxhighlight lang="bash">
sudo systemctl restart chronyd
sudo chronyc tracking
</syntaxhighlight>

== Failover Automation ==
=== 1. DNS Update Script (/usr/local/bin/update_dns.sh) ===
<syntaxhighlight lang="bash">
#!/bin/bash

CLUSTER_NAME="haproxy_cluster"
VIP="192.168.1.100"
DNS_RECORD="haproxy.example.com"
DNS_SERVER="ns1.example.com"
DNS_KEY="/etc/bind/cluster.key"
TTL=300

ACTIVE_NODE=$(sudo pcs status | grep "Current DC:" | awk '{print $4}'
logger -t haproxy-cluster "Failover detected. Active node: $ACTIVE_NODE"

nsupdate -k $DNS_KEY <<EOF
server $DNS_SERVER
update delete $DNS_RECORD A
update add $DNS_RECORD $TTL A $VIP
send
EOF

dig +short $DNS_RECORD @$DNS_SERVER
</syntaxhighlight>

=== 2. Make Script Executable ===
<syntaxhighlight lang="bash">
sudo chmod 750 /usr/local/bin/update_dns.sh
sudo chown hacluster:haclient /usr/local/bin/update_dns.sh
</syntaxhighlight>

=== 3. Configure Pacemaker Resource ===
<syntaxhighlight lang="bash">
sudo pcs resource create dns_update ocf:pacemaker:ClusterMon \
user=hacluster \
extra_options="-e /usr/local/bin/update_dns.sh" \
op monitor interval=15s timeout=30s \
meta target-role=Started

sudo pcs constraint colocation add dns_update with haproxy_group INFINITY
sudo pcs constraint order haproxy_group then dns_update
</syntaxhighlight>

== Testing Procedures ==
=== 1. Manual Failover Test ===
<syntaxhighlight lang="bash">
sudo pcs resource move haproxy_group node2.cluster.local
sudo pcs status | grep -A5 "haproxy_group"
sudo pcs resource clear haproxy_group
</syntaxhighlight>

=== 2. Simulate Node Failure ===
<syntaxhighlight lang="bash">
sudo systemctl stop corosync
sudo pcs status
watch -n 1 sudo pcs status
</syntaxhighlight>

=== 3. Verify DNS Updates ===
<syntaxhighlight lang="bash">
dig +short haproxy.example.com
</syntaxhighlight>

== Maintenance Operations ==
=== 1. Node Maintenance ===
<syntaxhighlight lang="bash">
sudo pcs node standby node3.cluster.local
sudo pcs node unstandby node3.cluster.local
</syntaxhighlight>

=== 2. Cluster Maintenance ===
<syntaxhighlight lang="bash">
sudo pcs cluster standby --all
sudo pcs cluster unstandby --all
</syntaxhighlight>

=== 3. Adding/Removing Nodes ===
<syntaxhighlight lang="bash">
sudo pcs cluster node add node5.cluster.local
sudo pcs cluster node remove node4.cluster.local
</syntaxhighlight>

== Troubleshooting ==
=== Common Issues ===
'''Corosync fails to start'''
<syntaxhighlight lang="bash">
sudo corosync-cfgtool -s
sudo corosync-cmapctl | grep members
</syntaxhighlight>

'''Split-brain scenario'''
<syntaxhighlight lang="bash">
sudo pcs property set stonith-enabled=true
sudo pcs stonith create fence_ipmi fence_ipmilan ...
</syntaxhighlight>

'''Resource failures'''
<syntaxhighlight lang="bash">
sudo pcs resource debug-start haproxy
sudo pcs resource failcount show
sudo pcs resource cleanup haproxy
</syntaxhighlight>

=== Log Locations ===
* '''Corosync''': /var/log/corosync/corosync.log
* '''Pacemaker''': journalctl -u pacemaker
* '''HAProxy''': /var/log/haproxy.log

== Appendix ==
=== Sample DNS Key File (/etc/bind/cluster.key) ===
<syntaxhighlight lang="conf">
key "cluster-key" {
algorithm hmac-sha256;
secret "Base64EncodedKeyHere==";
};
</syntaxhighlight>

=== Useful Commands ===
<syntaxhighlight lang="bash">
sudo pcs config show
sudo pcs stonith show
sudo pcs resource operations haproxy
</syntaxhighlight>

=== References & Further Reading ===
* [https://clusterlabs.org/pacemaker/doc/ Pacemaker Documentation]
* [https://www.haproxy.org/download/ HAProxy Configuration Manual]
* [https://wiki.debian.org/HighAvailability Debian Cluster Suite]
* [https://corosync.github.io/ Corosync Documentation]
* [https://chrony.tuxfamily.org/ Chrony Documentation]

----

''This comprehensive guide includes: step-by-step instructions, configuration examples, automated failover procedures, maintenance and troubleshooting, and a reference appendix.''

@@ Line 193: / Line 193: @@
 CLUSTER_NAME="haproxy_cluster"
 DNS_RECORD="haproxy.example.com"
 DNS_SERVER="ns1.example.com"
@@ Line 199: / Line 198: @@
 TTL=300
-ACTIVE_NODE=$(sudo pcs status | grep "Current DC:" | awk '{print $4}'
+# List your cluster node IPs here for reference (optional, not used in script logic)
-logger -t haproxy-cluster "Failover detected. Active node: $ACTIVE_NODE"
+NODE_IPS=("192.168.1.101" "192.168.1.102" "192.168.1.103" "192.168.1.104")
 nsupdate -k $DNS_KEY <<EOF
 server $DNS_SERVER
 update delete $DNS_RECORD A
-update add $DNS_RECORD $TTL A $VIP
+update add $DNS_RECORD $TTL A $ACTIVE_NODE_IP
 send
 EOF

@@ Line 199: / Line 199: @@
 TTL=300
-ACTIVE_NODE=$(sudo pcs status | grep "Current DC:" | awk '{print $3}' | cut -d. -f1)
+ACTIVE_NODE=$(sudo pcs status | grep "Current DC:" | awk '{print $4}'
 logger -t haproxy-cluster "Failover detected. Active node: $ACTIVE_NODE"
@@ Line 211: / Line 211: @@
 dig +short $DNS_RECORD @$DNS_SERVER
 </pre>
 === 2. Make Script Executable ===
 <pre>

@@ Line 2: / Line 2: @@
 '''A Comprehensive Guide for Debian Systems'''
 == Introduction ==
@@ Line 212: / Line 215: @@
 TTL=300
-ACTIVE_NODE=$(sudo pcs status | grep "Current DC:" | awk '{print $4}'
+ACTIVE_NODE=$(sudo pcs status | grep "Current DC:" | awk '{print $3}' | cut -d. -f1)
 logger -t haproxy-cluster "Failover detected. Active node: $ACTIVE_NODE"
@@ Line 310: / Line 313: @@
 == Appendix ==
 === Sample DNS Key File (/etc/bind/cluster.key) ===
 <syntaxhighlight lang="conf">
 key "cluster-key" {

High Availability Cluster with Pacemaker, Chronyd - Revision history

Manhal.Mohamed: /* 1. DNS Update Script (/usr/local/bin/update_dns.sh) */

Manhal.Mohamed: /* 1. DNS Update Script (/usr/local/bin/update_dns.sh) */

Manhal.Mohamed at 09:34, 22 July 2025

Manhal.Mohamed at 09:31, 22 July 2025

Manhal.Mohamed: /* Sample DNS Key File (/etc/bind/cluster.key) */

Manhal.Mohamed: Created page with "= High Availability Cluster with Pacemaker, Chronyd, and HAProxy = '''A Comprehensive Guide for Debian Systems''' == Introduction == This guide documents the setup of a 4-no..."