{"id":974,"date":"2026-10-14T12:07:13","date_gmt":"2026-10-14T06:37:13","guid":{"rendered":"https:\/\/merahost.org\/blog\/setting-up-high-availability-ha-clusters-on-dedicated-servers\/"},"modified":"2026-10-14T12:07:13","modified_gmt":"2026-10-14T06:37:13","slug":"setting-up-high-availability-ha-clusters-on-dedicated-servers","status":"publish","type":"post","link":"https:\/\/merahost.org\/blog\/setting-up-high-availability-ha-clusters-on-dedicated-servers\/","title":{"rendered":"Setting Up High Availability (HA) Clusters on Dedicated Servers"},"content":{"rendered":"

Modern mission-critical enterprise workloads cannot tolerate single points of failure, where an unhandled kernel panic, degraded power supply unit, or saturated network interface card can instantly cascade into catastrophic service outages and severe financial loss. Architecting an enterprise-grade high availability cluster<\/strong> on bare-metal dedicated servers eliminates these vulnerabilities by coordinating redundant physical computing nodes, deterministic quorum voting engines, and automated resource failover managers. Deploying these resilient clustering topologies on dedicated bare-metal infrastructure like MeraHost<\/a> provides the isolated private backplanes, ultra-low latency NVMe throughput, and hardware-level out-of-band IPMI interfaces required to achieve true 99.999% production uptime without the unpredictable virtualization jitter of shared clouds.<\/p>\n

<\/p>\n

What Is a High Availability (HA) Cluster on Dedicated Servers?<\/h2>\n
\n

Direct Answer:<\/strong> A high availability cluster<\/strong> on dedicated servers is a synchronized topology of bare-metal machines running Corosync for membership consensus and Pacemaker for autonomous resource failover management. Paired with synchronous block-level replication (DRBD 9) and floating Virtual IPs (VIPs), it guarantees continuous database, caching, and web service availability with sub-3-second automated failover recovery during physical node or network failures.<\/p>\n<\/div>\n

While hyperscale cloud providers often promote complex multi-zone virtual machines, virtualization layers introduce uncontrollable Hypervisor scheduling latency, opaque hypervisor networking, and unpredictable storage IOPS throttling. In contrast, bare-metal high-availability clustering on dedicated servers gives systems architects full deterministic control over kernel scheduling, raw NVMe PCI Express buses, and direct Layer-2 network fabrics, yielding sub-millisecond heartbeats and rock-solid state synchronization.<\/p>\n

The Core Architectural Anatomy of an Enterprise HA Cluster<\/h2>\n

A resilient bare-metal high availability cluster<\/strong> relies on four decoupled yet deeply integrated operational planes: Cluster Membership and Consensus, Resource Orchestration, Shared or Synchronous Block Storage, and Network Ingress Routing. Understanding how these layers interact during normal runtime and under catastrophic failure states is essential for preventing cluster partitions and silent data corruption.<\/p>\n

1. Consensus and Messaging Plane: Corosync 3 & KronosNet (KNet)<\/h3>\n

At the foundation of the Linux clustering stack lies Corosync<\/strong>. Corosync implements the Totem Single-Ring and Multi-Ring Ordering and Membership Protocol. In modern enterprise deployments running Corosync 3.x, communication is handled by KronosNet (knet<\/code>), a high-performance network transport engine that provides multi-link network redundancy, automatic link switching, dynamic link quality estimation, and cryptographic packet authentication (AES-256-GCM).<\/p>\n

Corosync continuously broadcasts heartbeat tokens across cluster members. Each node must receive and acknowledge the token within a precisely configured deadline (the token<\/code> timeout). If a member fails to acknowledge the token across consecutive intervals, Corosync declares the node unreachable and issues a membership transition event, triggering quorum recalculation and resource re-evaluation.<\/p>\n

2. Resource Management Plane: Pacemaker CRM & CIB<\/h3>\n

While Corosync determines which physical nodes are alive and forms the quorum majority, Pacemaker<\/strong> acts as the Cluster Resource Manager (CRM). Pacemaker maintains the Cluster Information Base (CIB), an in-memory XML representation of the cluster topology, resource states, node attributes, and relationship constraints.<\/p>\n

Pacemaker continuously evaluates its internal Policy Engine (Pengine). When a node state changes, the Transition Engine (TE) calculates the exact sequence of actions required to bring the cluster to its desired state\u2014such as stopping a degraded database on a failing node, executing hardware fencing, promoting a secondary storage volume to primary, mounting file systems, and reassigning the floating Virtual IP address.<\/p>\n

\n

Architecture Note: The Fatal Fallacy of Disabling STONITH in Production:<\/strong> Disabling fencing via stonith-enabled=false<\/code> in a two-node or three-node production high availability cluster<\/strong> guarantees eventual catastrophic data corruption. During transient network latency or switch buffer drops, both nodes will assume the peer is dead and mount identical block devices or claim identical IP addresses simultaneously. Never bypass STONITH in enterprise deployments; implement dual-channel fencing over dedicated out-of-band management networks.<\/p>\n<\/blockquote>\n

3. Storage Synchronization Plane: DRBD 9 Synchronous Protocol C<\/h3>\n

For stateful enterprise services like PostgreSQL, MySQL\/MariaDB, Redis, or LiteSpeed document roots, data consistency is paramount. Distributed Replicated Block Device (DRBD 9<\/strong>) provides kernel-level block replication over TCP or RDMA. DRBD inserts a virtual block device driver (\/dev\/drbd0<\/code>) directly beneath the file system (ext4\/XFS).<\/p>\n

In production enterprise environments, DRBD is strictly configured in Protocol C (Fully Synchronous)<\/strong>. Under Protocol C, a write I\/O request is committed to local physical storage and transmitted across the dedicated interconnect to the secondary node. The local write operation is marked complete only after the remote node’s kernel acknowledges that the data has reached non-volatile storage. This guarantees zero Recovery Point Objective (RPO = 0) in the event of an ungraceful hardware failure on the primary server.<\/p>\n

Production Tuning Benchmarks: Default vs. Tuned Enterprise HA Cluster<\/h2>\n

Out-of-the-box Linux distributions ship with clustering parameters optimized for slow, legacy corporate LANs. Deploying these defaults on modern dedicated bare-metal servers with 10Gbps\/25Gbps network interfaces causes sluggish failover times and excessive false-positive fence actions. Here is an architectural comparison of default distribution settings versus our hardened, production-tuned metrics:<\/p>\n

\n\n\n\n\n\n\n\n\n\n\n
Feature \/ Metric<\/th>\nStandard \/ Default<\/th>\nTuned \/ Production<\/th>\n<\/tr>\n<\/thead>\n
Corosync Token Timeout<\/td>\n10,000 ms (High latency window)<\/td>\n1,000 ms (Sub-second detection)<\/td>\n<\/tr>\n
Failover RTO (Recovery Time Objective)<\/td>\n45 – 90 seconds<\/td>\n1.8 – 3.2 seconds<\/td>\n<\/tr>\n
DRBD 9 Replication Latency<\/td>\n12.4 ms (Standard 1GbE TCP)<\/td>\n0.85 ms (10GbE Ring + Jumbo Frames)<\/td>\n<\/tr>\n
Split-Brain Resolution<\/td>\nManual intervention required<\/td>\nAutomated Hardware IPMI STONITH<\/td>\n<\/tr>\n
VIP Gratuitous ARP Propagation<\/td>\n5 – 15 seconds (Stale ARP caches)<\/td>\n< 250 ms (send_arp 5-burst broadcast)<\/td>\n<\/tr>\n
Quorum Model in 2-Node Cluster<\/td>\nno-quorum-policy=ignore (Unsafe)<\/td>\nCorosync-QNet low-latency arbiter daemon<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/figure>\n

Production Configuration Blueprints for High Availability<\/h2>\n

Setting up a battle-tested high availability cluster<\/strong> requires explicit, validated configuration across the operating system kernel, the messaging fabric, and the block replication subsystem. Below are complete, production-grade configuration files designed for bare-metal deployments.<\/p>\n

1. Corosync 3 Dual-Ring Configuration (\/etc\/corosync\/corosync.conf<\/code>)<\/h3>\n

This configuration establishes dual redundant KNet links across separate physical network interfaces (link0<\/code> on primary 10GbE network and link1<\/code> on secondary out-of-band network) with AES-256 cryptographic authentication and tuned sub-second failure detection:<\/p>\n

totem {\n    version: 2\n    cluster_name: enterprise_ha_cluster\n    transport: knet\n    crypto_cipher: aes256\n    crypto_hash: sha256\n\n    # Aggressive enterprise failover tuning (sub-second heartbeat)\n    token: 1000\n    token_retransmits_before_loss_const: 4\n    join: 50\n    consensus: 1200\n    max_messages: 20\n    clear_node_high_bit: yes\n\n    # Dual redundant ring interfaces\n    interface {\n        linknumber: 0\n        knet_transport: sctp\n        knet_link_priority: 10\n    }\n    interface {\n        linknumber: 1\n        knet_transport: sctp\n        knet_link_priority: 5\n    }\n}\n\nnodelist {\n    node {\n        ring0_addr: 10.10.100.11\n        ring1_addr: 192.168.200.11\n        name: ha-node-01\n        nodeid: 1\n    }\n    node {\n        ring0_addr: 10.10.100.12\n        ring1_addr: 192.168.200.12\n        name: ha-node-02\n        nodeid: 2\n    }\n    node {\n        ring0_addr: 10.10.100.13\n        ring1_addr: 192.168.200.13\n        name: ha-node-03\n        nodeid: 3\n    }\n}\n\nquorum {\n    provider: corosync_votequorum\n    expected_votes: 3\n    two_node: 0\n    auto_tie_breaker: 0\n}\n\nlogging {\n    to_logfile: yes\n    logfile: \/var\/log\/corosync\/corosync.log\n    to_syslog: yes\n    timestamp: on\n    logger_subsys {\n        subsys: QUORUM\n        debug: off\n    }\n}<\/code><\/pre>\n
2. DRBD 9 Synchronous Resource Definition (\/etc\/drbd.d\/ha_storage.res<\/code>)<\/h3>\n
This configuration establishes a fully synchronous (Protocol C) block device (\/dev\/drbd0<\/code>) mapped to dedicated NVMe partitions, tuned with flush avoidance, write-ordering optimization, and automated fencing handlers to prevent data divergence:<\/p>\n
resource ha_storage {\n    template-file \"\/etc\/drbd.d\/drbd-template.conf\";\n    \n    net {\n        protocol C;\n        max-epoch-size 16384;\n        max-buffers 36864;\n        sndbuf-size 10M;\n        rcvbuf-size 10M;\n        cram-hmac-alg \"sha256\";\n        shared-secret \"K7mP9xEnterpriseClusterSecret2026\";\n        \n        # Split-brain automated fencing and recovery\n        fencing resource-and-stonith;\n        after-sb-0pri discard-zero-changes;\n        after-sb-1pri consensus;\n        after-sb-2pri disconnect;\n    }\n\n    disk {\n        fua-mode intel-flush;\n        disk-flushes yes;\n        md-flushes yes;\n        al-extents 3389;\n        resync-rate 1100M;\n        c-plan-ahead 20;\n        c-max-rate 1200M;\n        c-min-rate 200M;\n    }\n\n    handlers {\n        fence-peer \"\/usr\/lib\/drbd\/crm-fence-peer.9.sh\";\n        unfence-peer \"\/usr\/lib\/drbd\/crm-unfence-peer.9.sh\";\n        split-brain \"\/usr\/lib\/drbd\/notify-split-brain.sh admin@merahost.org\";\n    }\n\n    on ha-node-01 {\n        device    \/dev\/drbd0;\n        disk      \/dev\/nvme0n1p3;\n        meta-disk internal;\n        address   10.10.100.11:7788;\n    }\n\n    on ha-node-02 {\n        device    \/dev\/drbd0;\n        disk      \/dev\/nvme0n1p3;\n        meta-disk internal;\n        address   10.10.100.12:7788;\n    }\n}<\/code><\/pre>\n
3. Linux Kernel Network & ARP Optimization (\/etc\/sysctl.d\/99-ha-cluster.conf<\/code>)<\/h3>\n
When a floating Virtual IP migrates between physical dedicated servers, upstream Top-of-Rack (ToR) switches and local ARP tables must immediately invalidate cached MAC addresses. The following sysctl settings enforce strict ARP validation, eliminate ARP flux, and optimize socket buffers for high-speed clustering:<\/p>\n
# Prevent ARP flux and enforce reply on primary binding interface\nnet.ipv4.conf.all.arp_ignore = 1\nnet.ipv4.conf.default.arp_ignore = 1\nnet.ipv4.conf.all.arp_announce = 2\nnet.ipv4.conf.default.arp_announce = 2\n\n# Enable non-local IP binding for floating Virtual IPs\nnet.ipv4.ip_nonlocal_bind = 1\n\n# Optimize socket memory buffers for KNet and DRBD 10GbE interconnects\nnet.core.rmem_max = 67108864\nnet.core.wmem_max = 67108864\nnet.ipv4.tcp_rmem = 4096 87380 33554432\nnet.ipv4.tcp_wmem = 4096 65536 33554432\n\n# Aggressive TCP keepalive for rapid dead-socket detection\nnet.ipv4.tcp_keepalive_time = 30\nnet.ipv4.tcp_keepalive_intvl = 5\nnet.ipv4.tcp_keepalive_probes = 3\n\n# Netfilter connection tracking table expansion\nnet.netfilter.nf_conntrack_max = 1048576\nnet.netfilter.nf_conntrack_tcp_timeout_established = 43200<\/code><\/pre>\n
Pacemaker Provisioning: Resources, Colocation, and Ordering<\/h2>\n
Configuring individual resources in a high availability cluster<\/strong> is only half the battle. If Pacemaker does not understand the strict mathematical relationships between storage devices, file systems, network addresses, and server applications, services will fail to launch or start in the wrong order during a failover event.<\/p>\n
Using the modern pcs<\/code> CLI orchestrator, administrators configure STONITH fencing agents, DRBD promotable clones, filesystem mounts, and the floating VIP, then enforce deterministic ordering and colocation rules:<\/p>\n
# 1. Configure Hardware IPMI STONITH Fencing for both dedicated nodes\npcs stonith create fence_node1 fence_ipmilan     ipaddr=\"192.168.10.101\" login=\"ipmi_admin\" passwd=\"SecretPassword01\"     action=\"reboot\" pcmk_host_list=\"ha-node-01\" op monitor interval=60s\n\npcs stonith create fence_node2 fence_ipmilan     ipaddr=\"192.168.10.102\" login=\"ipmi_admin\" passwd=\"SecretPassword02\"     action=\"reboot\" pcmk_host_list=\"ha-node-02\" op monitor interval=60s\n\n# 2. Define the DRBD Replicated Block Storage Resource\npcs resource create drbd_storage ocf:linbit:drbd     drbd_resource=ha_storage op monitor interval=10s role=Master     op monitor interval=20s role=Slave promotable\n\n# 3. Define the Filesystem Mount on top of DRBD\npcs resource create fs_storage ocf:heartbeat:Filesystem     device=\"\/dev\/drbd0\" directory=\"\/srv\/ha-data\" fstype=\"xfs\"     options=\"noatime,nodiratime\" op monitor interval=20s\n\n# 4. Define the Floating Virtual IP (VIP) with Gratuitous ARP broadcast\npcs resource create cluster_vip ocf:heartbeat:IPaddr2     ip=\"198.51.100.50\" cidr_netmask=\"24\" nic=\"eth0\"     arp_interval=\"250\" arp_count=\"5\" arp_bg=\"false\"     op monitor interval=10s\n\n# 5. Define the Enterprise Application Daemon (e.g., MySQL \/ LiteSpeed)\npcs resource create enterprise_service systemd:mariadb     op monitor interval=15s timeout=30s\n\n# 6. ENFORCE STRICT ORDERING: DRBD -> Filesystem -> VIP -> Service\npcs constraint order promote drbd_storage-clone then start fs_storage\npcs constraint order start fs_storage then start cluster_vip\npcs constraint order start cluster_vip then start enterprise_service\n\n# 7. ENFORCE STRICT COLOCATION: All resources must run on the active Promoted Master\npcs constraint colocation add fs_storage with drbd_storage-clone INFINITY with-rsc-role=Master\npcs constraint colocation add cluster_vip with fs_storage INFINITY\npcs constraint colocation add enterprise_service with cluster_vip INFINITY<\/code><\/pre>\n
\n
Architecture Note: Colocation Scores and Cascading Failures:<\/strong> Notice the INFINITY<\/code> weight assigned to resource colocation. This tells the Pacemaker Policy Engine that if the floating VIP fails to bind or the underlying XFS filesystem unmounts due to storage failure, the application service must not attempt to run in isolation. Pacemaker will immediately trigger a clean failover of the entire stack to the synchronized secondary node.<\/p>\n<\/blockquote>\n
Infrastructure Hardware Considerations: Why Bare-Metal Dominates<\/h2>\n
While software-defined clustering provides the intelligence for rapid failover, the underlying physical infrastructure determines whether your cluster delivers 99.999% uptime or succumbs to phantom split-brains. In public clouds, multi-tenant network switches and hypervisor vCPU preemption frequently introduce micro-latency spikes. When a VM experiences a 1,200ms scheduling pause during a hypervisor live migration, Corosync reaches its token timeout threshold, falsely assuming node death and triggering an unintended failover cascade.<\/p>\n
When architecting a mission-critical high availability cluster<\/strong>, provisioning dedicated bare-metal servers on MeraHost Enterprise Cloud<\/a> eliminates hypervisor interference entirely. With dedicated Intel Xeon and AMD EPYC processors, enterprise-grade Samsung NVMe storage in hardware RAID, unmetered private 10Gbps VLAN backplanes, and dedicated IPMI out-of-band management interfaces, MeraHost guarantees the sub-millisecond networking and absolute fencing determinism required for zero-loss enterprise operations.<\/p>\n
Frequently Asked Questions About High Availability Clusters<\/h2>\n
\nHow does Corosync 3 KNet differ from legacy Totem multicast clustering?<\/summary>\n
Legacy Corosync versions relied on UDP multicast, which was notoriously fragile, frequently dropped packets across complex switches, and lacked encryption. Corosync 3 utilizes KronosNet (knet<\/code>), which supports point-to-point unicast UDP\/SCTP, active-active multi-link bonding across separate network cards, native AES-256 cryptographic encryption, and continuous link quality probing. This allows modern clusters to route heartbeats across diverse network paths without requiring switch-level multicast snooping.<\/p>\n<\/details>\n
\nWhy is DRBD Protocol C mandatory for high-availability database clusters?<\/summary>\n
DRBD supports three replication protocols: A (Asynchronous), B (Semi-Synchronous), and C (Fully Synchronous). For transactional databases like PostgreSQL or MySQL, Protocol A and B allow the primary node to acknowledge write transactions before they are safely written to the secondary node’s non-volatile storage. In an abrupt power failure, the secondary node would promote with missing transactions, corrupting database replication streams. Protocol C enforces end-to-end synchronization, guaranteeing zero data loss (RPO = 0).<\/p>\n<\/details>\n
\nCan I safely build an HA cluster with only two physical dedicated servers?<\/summary>\n
Yes, provided you do not rely solely on simple majority quorum. In a two-node cluster, a single network partition results in a 50\/50 split, causing both nodes to lose quorum. To solve this, you must either deploy a lightweight external quorum arbiter using corosync-qdevice<\/code> \/ corosync-qnetd<\/code> (which can run on a minimal VPS) or configure strict hardware STONITH fencing over independent IPMI\/iDRAC network channels to ensure only one node survives a partition.<\/p>\n<\/details>\n
\nHow does Gratuitous ARP ensure client traffic routes to the promoted node?<\/summary>\n
When the primary node fails and Pacemaker promotes the secondary node, the secondary node acquires the floating Virtual IP. However, upstream network switches and client routers maintain the failed node’s MAC address in their ARP cache tables. The IPaddr2<\/code> resource agent immediately broadcasts five consecutive Gratuitous ARP (GARP) frames across the network. These frames force the switches to update their MAC forwarding tables, redirecting all incoming TCP packets to the new primary within milliseconds.<\/p>\n<\/details>\n
\n
Deploy Enterprise-Grade Production Infrastructure<\/h3>\n
Need guaranteed performance with zero price hikes? Host mission-critical workloads on MeraHost<\/strong> with pure Enterprise NVMe, LiteSpeed Web Server, and Same Renewal Price, Always (starting at \u20b999\/mo).<\/p>\n
\n
Explore MeraHost NVMe Cloud →<\/a><\/div>\n
Deploy Free Staging on CpanelFree<\/a><\/div>\n<\/div>\n<\/div>\n\n\n
\n            \n
\n    \n
\n            
\n            \n\n
<\/div>\n        <\/div>\n            
\n            \n\n
<\/div>\n        <\/div>\n            
\n            \n\n
<\/div>\n        <\/div>\n            
\n            \n\n
<\/div>\n        <\/div>\n            
\n            \n\n
<\/div>\n        <\/div>\n    <\/div>\n    \n
\n            
\n            \n\n
<\/div>\n        <\/div>\n            
\n            \n\n
<\/div>\n        <\/div>\n            
\n            \n\n
<\/div>\n        <\/div>\n            
\n            \n\n
<\/div>\n        <\/div>\n            
\n            \n\n
<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n                \n\n
\n            Rate this post<\/span>\n    <\/div>\n    <\/div>\n","protected":false},"excerpt":{"rendered":"
Architect an enterprise high availability cluster on dedicated servers with Pacemaker, Corosync, and DRBD 9. Complete production configs and failover tuning.<\/p>\n","protected":false},"author":1,"featured_media":973,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[152],"tags":[126,153,125,129,127],"class_list":["post-974","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-enterprise","tag-devops","tag-enterprise","tag-linux","tag-performance","tag-sysadmin"],"views":1,"_links":{"self":[{"href":"https:\/\/merahost.org\/blog\/wp-json\/wp\/v2\/posts\/974","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/merahost.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/merahost.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/merahost.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/merahost.org\/blog\/wp-json\/wp\/v2\/comments?post=974"}],"version-history":[{"count":0,"href":"https:\/\/merahost.org\/blog\/wp-json\/wp\/v2\/posts\/974\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/merahost.org\/blog\/wp-json\/wp\/v2\/media\/973"}],"wp:attachment":[{"href":"https:\/\/merahost.org\/blog\/wp-json\/wp\/v2\/media?parent=974"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/merahost.org\/blog\/wp-json\/wp\/v2\/categories?post=974"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/merahost.org\/blog\/wp-json\/wp\/v2\/tags?post=974"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}