Multicast notes

http://pve.proxmox.com/wiki/Multicast_notes

Introduction

Multicast allows a single transmission to be delivered to multiple servers at the same time.

This is the basis for cluster communications in Proxmox VE 2.0 and higher, which uses corosync and cman, and would apply to any other solution which utilizes those clustering tools.

If multicast does not work in your network infrastructure, you should fix it so that it does. If all else fails, use unicast instead, but beware of the node count limitations with unicast.

IGMP snooping

IGMP snooping prevents flooding multicast traffic to all ports in the broadcast domain by only allowing traffic destined for ports which have solicited such traffic. IGMP snooping is a feature offered by most major switch manufacturers and is often enabled by default on switches. In order for a switch to properly snoop the IGMP traffic, there must be an IGMP querier on the network. If no querier is present, IGMP snooping will actively prevent ALL IGMP/Multicast traffic from being delivered!

If IGMP snooping is disabled, all multicast traffic will be delivered to all ports which may add unnecessary load, potentially allowing a denial of service attack.

IGMP querier

An IGMP querier is a multicast router that generates IGMP queries. IGMP snooping relies on these queries which are unconditionally forwarded to all ports, as the replies from the destination ports is what builds the internal tables in the switch to allow it to know which traffic to forward.

IGMP querier can be enabled on your router, switch, or even linux bridges.

Testing multicast

Note: not all hosting companies allow multicast traffic.

First, check your cluster multicast address:

#pvecm status|grep "Multicast addresses"
Multicast addresses: 239.192.221.35 

Using omping

Install on all nodes

aptitude install omping

start omping on all nodes with the following command and check the output, e.g:

omping -m yourmulticastadress node1 node2 node3

• note to find the multicast address run this:

pvecm status | grep Multicast

Troubleshooting

cman & iptables

In case cman crashes with cpg_send_message failed: 9 add those to your rule set:

iptables -A INPUT -m addrtype --dst-type MULTICAST -j ACCEPT
iptables -A INPUT -p udp -m state --state NEW -m multiport –dports 5404,5405 -j ACCEPT

Use unicast instead of multicast (if all else fails)

Unicast is a technology for sending messages to a single network destination. In corosync, unicast is implemented as UDP-unicast (UDPU). Due to increased network traffic (compared to multicast) the number of supported nodes is limited, do not use it with more that 4 cluster nodes.

• just create the cluster as usual (pvecm create ...)
• follow this howto to create a cluster.conf.new Fencing#General_HowTo_for_editing_the_cluster.conf
• add the new transport="udpu" in /etc/pve/cluster.conf.new (don't forget to increment the version number)

<cman keyfile="/var/lib/pve-cluster/corosync.authkey" transport="udpu"/>

• activate via GUI
• add all nodes you want to join in /etc/hosts and reboot
• before you add a node, make sure you add all other nodes in /etc/hosts