HAProxy Lab Setup Guide – Ubuntu-Based Multi-Node Setup (sdnog Edition)

HAProxy Lab Setup Guide – Ubuntu-Based Multi-Node Setup (sdnog Edition)

Overview

This guide documents the setup of HAProxy on Ubuntu servers for load balancing web (HTTP/HTTPS) and MySQL database traffic, using the `sdnog.sd` lab domain. The environment includes multiple backend nodes and complete instructions for configuration, testing, and troubleshooting.

Prerequisites

• 3+ Ubuntu servers/VMs: 1 for HAProxy (load balancer), at least 2 for web/database backends.
• Domain names:

 * `lb.lab.sdnog.sd` (HAProxy/load balancer)
 * `www.lab.sdnog.sd`, `db.lab.sdnog.sd`, `lb.lab.sdnog.sd` (all pointing to the HAProxy IP)
 * Backends: `web01.lab.sdnog.sd`, `web02.lab.sdnog.sd`

• Sudo/root access on all nodes.
• Ability to update `/etc/hosts` or your DNS zone for lab domains.

Local Hosts or DNS Configuration

Set the following entries on your local hosts file or DNS server, pointing to your HAProxy IP (`X.X.X.X`):

 X.X.X.X lb.lab.sdnog.sd
 X.X.X.X www.lab.sdnog.sd
 X.X.X.X db.lab.sdnog.sd

Backends (`web01.lab.sdnog.sd`, `web02.lab.sdnog.sd`) should resolve to their actual server IPs.

Step 1: Install HAProxy on Ubuntu

sudo apt update
sudo apt install haproxy

Step 2: Install Web Servers on Backends

On `web01.lab.sdnog.sd` and `web02.lab.sdnog.sd`, install Nginx on one and Apache on another:

# Nginx (on web01)
sudo apt install nginx
echo "This is web01.lab.sdnog.sd" | sudo tee /var/www/html/index.html

# Apache (on web02)
sudo apt install apache2
echo "This is web02.lab.sdnog.sd" | sudo tee /var/www/html/index.html

Step 3: Install MySQL on Backends

On both `web01.lab.sdnog.sd` and `web02.lab.sdnog.sd`:

sudo apt install mysql-server

Step 4: Configure MySQL User for HAProxy

On both DB nodes (MySQL prompt):

CREATE USER 'sdnoguser'@'%' IDENTIFIED BY 'StrongPassword';
GRANT ALL PRIVILEGES ON *.* TO 'sdnoguser'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;

Step 5: HAProxy Configuration for Web & Database Load Balancing

Edit `/etc/haproxy/haproxy.cfg` on the HAProxy server:

global
    log         127.0.0.1:514 local1 info
    maxconn     4000
    user        haproxy
    group       haproxy

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option                  forwardfor except 127.0.0.0/8
    option                  http-server-close
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

frontend https-in
    bind *:443 ssl crt /home/sdnog/lab/cert.pem
    mode http
    acl host_lab_sdnog hdr(host) -i www.lab.sdnog.sd
    use_backend www_back if host_lab_sdnog
    default_backend www_back

frontend http-in
    bind *:80
    redirect scheme https code 301 if !{ ssl_fc }

backend www_back
    balance roundrobin
    cookie SERVERID insert indirect nocache
    server nginx_server web01.lab.sdnog.sd:80 check cookie web01
    server apache_server web02.lab.sdnog.sd:80 check cookie web02

frontend database_frontend
    bind *:3306
    mode tcp
    default_backend database_backend

backend database_backend
    mode tcp
    balance roundrobin
    server db01 web01.lab.sdnog.sd:3306 check
    server db02 web02.lab.sdnog.sd:3306 check

listen stats
    bind 0.0.0.0:8080
    bind :::8080
    mode http
    stats uri /stats
    stats realm HAProxy\ Statistics
    stats auth admin:StrongPassword
    stats admin if TRUE
    timeout client 5000
    timeout connect 4000
    timeout server 30000

Step 6: SSL Certificate for HAProxy

Generate or copy your SSL certificate and key as `/home/sdnog/lab/cert.pem` (or another path if you adjust your config).

Step 7: Restart and Enable HAProxy

sudo systemctl restart haproxy
sudo systemctl enable haproxy

Step 8: Testing

Web Load Balancing

Test the round-robin backend selection:

curl -I https://www.lab.sdnog.sd | grep SERVERID

You should see alternating results like:

 set-cookie: SERVERID=web01; path=/
 set-cookie: SERVERID=web02; path=/

See for example output.

Web test

Database Load Balancing

Test DB backend switching:

mysql -u sdnoguser -p -h db.lab.sdnog.sd -e "show variables like 'hostname';"

You should see the hostname alternating between `db01.lab.sdnog.sd` and `db02.lab.sdnog.sd`

Layer 4 loadbalancing

HAProxy Stats Page

Browse to: http://lb.lab.sdnog.sd:8080/stats

for an example:

HAProxy Log

How to Enable HAProxy Logging

To enable and view HAProxy logs, follow these steps:

1. Edit your HAProxy configuration file (usually /etc/haproxy/haproxy.cfg) and add the following in the global and/or defaults sections:

global
    log 127.0.0.1 local0

defaults
    log     global
    option  httplog
    option  dontlognull

• log 127.0.0.1 local0: Sends logs to the local syslog server.
• option httplog: Enables detailed HTTP log format.
• option dontlognull: Avoids logging empty connections.

1. Configure your syslog service (such as rsyslog) to receive HAProxy logs:

• Add the following to /etc/rsyslog.conf or /etc/rsyslog.d/haproxy.conf:

$ModLoad imudp
$UDPServerRun 514

local0.*    /var/log/haproxy.log

1. Restart your syslog service:

sudo systemctl restart rsyslog

1. Restart HAProxy:

sudo systemctl restart haproxy

1. Check your HAProxy log output:

sudo tail -f /var/log/haproxy.log

---

Example Log Line

2025-07-18T15:51:04+00:00 localhost haproxy[8400]: 102.117.90.22:57291 [18/Jul/2025:15:51:04.732] https-in~ www_back/nginx_server 0/0/1/1/2 200 210 - - --NI 1/1/0/0/0 0/0 "HEAD https://www.lab.sdnog.sd/ HTTP/2.0"

Explanation of Fields

Timestamp and Host

2025-07-18T15:51:04+00:00 — Date and time (ISO 8601 format)

localhost — Hostname where HAProxy is running

haproxy[8400] — Process name and PID

Client Info

102.117.90.22:57291 — Source IP address and port of the client

Accept Date

[18/Jul/2025:15:51:04.732] — When HAProxy accepted the connection/request

Frontend, Backend, Server

https-in~ — HAProxy frontend handling the request

www_back/nginx_server — Backend and backend server that handled the request

Timers (ms)

0/0/1/1/2

• Tq: Time spent waiting in queue
• Tw: Time waiting for connection to backend server
• Tc: Time to establish connection to backend
• Tr: Time to get the full HTTP request from the client
• Tt: Total time from accept to response

HTTP Status and Bytes

200 — HTTP status code returned to the client (OK)

210 — Number of bytes sent to the client (response size)

Captured Request/Response Cookies

- - — (Dashes mean "not captured" or "not set")

Termination State

--NI — How/why the session ended (see HAProxy documentation for codes)

Connections (ActConn/FeConn/BeConn/SrvConn/Retry)

1/1/0/0/0

• ActConn: Active connections on the frontend
• FeConn: Connections on the frontend
• BeConn: On the backend
• SrvConn: On the server
• Retry: Retries

Queues (SrvQueue/BackendQueue)

0/0

• SrvQueue: Number of queued requests on the server
• BackendQueue: Number of queued requests on the backend

Request Line

"HEAD https://www.lab.sdnog.sd/ HTTP/2.0" — The HTTP method, URL, and protocol

---

Explanation

• Each line is a single request processed by HAProxy.
• The log shows: when it happened, who connected, what request they made, what server handled it, how long each step took, and what the result was.
• If you see different backends/servers (like nginx_server or apache_server), it means HAProxy is load balancing between them.
• Status codes like 200 mean “OK”. If you see 500, 404, etc., that means there was an error.
• Timings help you diagnose where delays are happening (queue, connection, etc.).
• Termination state (--NI) can show if the connection ended normally or with errors/timeouts.

---

Quick Reference Table

Field	Example Value	Meaning
Timestamp	2025-07-18T15:51:04+00:00	When the event happened
Client IP:Port	102.117.90.22:57291	Who made the request
Accept Date	[18/Jul/2025:15:51:04.732]	When HAProxy accepted the request
Frontend~	https-in~	Which frontend handled it
Backend/Server	www_back/nginx_server	Backend/server chosen
Timers	0/0/1/1/2	Time in each HAProxy phase
Status	200	HTTP status code
Bytes	210	Bytes sent to client
Term. State	--NI	How session ended
Connections	1/1/0/0/0	Conn. counts (frontend, backend, etc.)
Queues	0/0	Queued requests
Request	"HEAD ... HTTP/2.0"	HTTP Method, URL, Protocol

---

Tip: For more details, see the HAProxy log format documentation.

Troubleshooting

Common Issues and Solutions

1. HAProxy not starting:
   • Check the configuration file for syntax errors:

haproxy -c -f /etc/haproxy/haproxy.cfg

1. • Verify that the ports HAProxy is trying to bind to are not already in use.

1. Backend servers not responding:
   • Make sure Apache/Nginx/MySQL are running on their respective servers.
   • Check firewall rules (UFW, iptables, or cloud security groups) to allow traffic between HAProxy and backend nodes.
   • Confirm the correct backend IP addresses and ports in the HAProxy configuration.

1. SSL certificate issues:
   • Double-check the certificate and private key path in your HAProxy config.
   • Ensure your `.pem` file has correct permissions and the combined format (cert + key).

1. ACLs/routing not working as expected:
   • Verify that your local `/etc/hosts` or DNS is configured for the lab domains.
   • Use `tcpdump` or `wireshark` to inspect HTTP headers if needed.

1. Stats page not accessible:
   • Confirm the `listen stats` block in your config.
   • Make sure port 8080 is open on the HAProxy machine and not blocked by a firewall.

1. MySQL authentication errors:
   • Ensure user/password is the same on all DB nodes.
   • Make sure MySQL is listening on 0.0.0.0 or the correct interface.

Performance Tuning

1. Increase maximum connections:
   • Adjust the `maxconn` parameter in the `global` and `defaults` sections based on your hardware.
2. Enable kernel TCP keepalive:
   • Add `option tcpka` in the `defaults` section if needed.
3. Enable HTTP/2:
   • Update your SSL binding to:

bind *:443 ssl crt /home/sdnog/lab/cert.pem alpn h2,http/1.1

1. Implement caching:
   • Consider using Varnish in front of HAProxy for static content.

Troubleshooting

Common Issues and Solutions

1. HAProxy not starting:
   • Check the configuration file for syntax errors:

haproxy -c -f /etc/haproxy/haproxy.cfg

1. • Verify that the ports HAProxy is trying to bind to are not in use by other services.

1. Backend servers not responding:
   • Ensure that Apache and Nginx are running on their respective VMs.
   • Check firewall rules to allow traffic between HAProxy and backend servers.
   • Verify the IP addresses and ports in the HAProxy configuration.

1. SSL certificate issues:
   • Double-check the path to the SSL certificate and key in the HAProxy configuration.
   • Ensure the combined PEM file has the correct permissions.

1. ACLs not working as expected:
   • Verify that your local hosts file is correctly configured.
   • Use `tcpdump` or `wireshark` to inspect the HTTP headers and ensure the correct `Host` header is being sent.

Performance Tuning

Optimizing HAProxy

1. Increase maximum connections:
   • Adjust the `maxconn` parameter in the `global` section based on your server's capacity.

1. Enable kernel TCP splicing:
   • Add `option tcpka` to the `defaults` section for keep-alive connections.

1. Use HTTP/2:
   • Update your SSL binding to support HTTP/2:

bind *:443 ssl crt /etc/ssl/certs/haproxy.pem alpn h2,http/1.1

1. Implement caching:
   • Consider adding a caching layer with Varnish in front of HAProxy for static content.

Optimal Configuration Options for Web-Based Frontends

It's crucial to customize the following according to your application's specific requirements.

   frontend http-in
   bind *:80
   bind *:443 ssl crt /etc/haproxy/certs/cert.pem no-sslv3
   mode http
   option httplog
   log global
   
   # Redirect HTTP to HTTPS (enforce HTTPS for all traffic)
   http-request redirect scheme https code 301 if !{ ssl_fc }
  
   # Set default security headers for responses
   # Enforce HSTS for HTTPS (1 year, include subdomains, preload)
   http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
   
   # Clickjacking protection, allow only the same origin to embed this site
   http-response set-header X-Frame-Options "SAMEORIGIN"
   
   # XSS filtering enabled in browsers, block if an attack is detected
   http-response set-header X-XSS-Protection "1; mode=block"
   
   # Prevent MIME type sniffing (force browser to honor content type declared by the server)
   http-response set-header X-Content-Type-Options "nosniff"
  
   # Add Content Security Policy to mitigate XSS and data injection attacks
   http-response set-header Content-Security-Policy "default-src 'self'; script-src 'self'; object-src 'none'"
  
   # Disable referrer information leakage when navigating to a different origin
   http-response set-header Referrer-Policy "no-referrer-when-downgrade"
   
   # Prevent browsers and proxies from caching sensitive data
   http-response set-header Cache-Control "no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0"
   
   # Set secure cookies (only for HTTPS, HttpOnly, and prevent cross-site requests)
   acl secure_cookie hdr_sub(cookie) Secure
   http-response set-header Set-Cookie %[res.hdr(Set-Cookie)] if secure_cookie
   http-response set-header Set-Cookie Secure; HttpOnly; SameSite=Strict if secure_cookie
   
   # Forward client's original IP in X-Forwarded-For header
   http-request add-header X-Forwarded-For %[src]
   
   # Forward the protocol used by the client (HTTP/HTTPS) in X-Forwarded-Proto header
   http-request add-header X-Forwarded-Proto https if { ssl_fc }
   http-request add-header X-Forwarded-Proto http if !{ ssl_fc }
   
   # Preserve the original Host header
   http-request add-header X-Forwarded-Host %[req.hdr(host)]
   
   default_backend servers

Appendix: Useful Commands

• Check HAProxy config:

  haproxy -c -f /etc/haproxy/haproxy.cfg

• Restart HAProxy:

  sudo systemctl restart haproxy

• View HAProxy logs:

  sudo tail -f /var/log/haproxy.log

• Test backend health:

  curl http://web01.lab.sdnog.sd

Author

• Author: Manhal Mohamed , SdNOG Team

---

This guide is based on practical deployment, tested with Ubuntu 22.04+. For additional features and advanced security, consult the HAProxy documentation.