Cluster setup with Aggregates, VLAN's, and IPMP

Configuring clusters is complicated.  It is even more complicated when you throw in complex network configurations demanded by some customers.  This particular config has pretty much everything in it.  Aggregation, VLAN's and IPMP.

   We've got 2 quad-gig Intel cards.  So that puts e1000g0-g3 on one card and e1000g4-g7 on the other.  Aggregations aren't declared dead until all links are down so to protect agains a complete card failure and a switch failure, we'll have each card aggregate it's ports to a single switch instead of between cards.  So we'll create aggr0 on one card and aggr1 on the other.
    On top of the aggregates, we need to create vlan interfaces.  We'll use VLAN ID is 2601 in this example.  This will createaggr2601000 which will ride on aggr0 and aggr2601001 which will be on aggr1.
    We then use IPMP between aggr2601000 and aggr2601001.  We will configure link-based IPMP which will only switch interfaces if the link goes down.  If we wanted probe-based, in which the system will actively ping the local router to determine network functionality, we would put test addresses on aggr2601000 and aggr2601001.

    From NMS:

setup network aggregation create
    - Select e1000g0 & e1000g1
    - Select "off", "passive", or "active" depending on what the customer's network requires
   - call it aggr0
setup network aggregation create
    - select e1000g4 & e1000g5
    - Select same LACP as aggr0
    - call it aggr1
setup network interface vlan aggr0
    - VLAN ID 2601
setup network interface vlan aggr1
    -VLAN ID 2601

    From there we need to drop to bash and edit the /etc/hostname.* files.
    For IPMP, we should create this according to Clearview standards.  Clearview information can be found at http://hub.opensolaris.org/bin/view/Project+clearview/ipmp

    First, we configure the underlying interfaces for our group.  We're going to call this the data0 group.  We'll give the underlying interfaces "test" addresses, which are used to test functionality of the network.  Typically these will ping the local routers.

/etc/hostname.aggr2601000:
    group data0 192.168.0.111 -failover

/etc/hostname.aggr2601001:
    group data0 192.168.0.112 -failover

    Second, we create the IPMP interface.  We're going to give this a vanity name that descriptive of what it is to do.  In our case, datanet0.

/etc/hostname.datanet0    ipmp group data0    192.168.10.110 netmask 255.255.255.0 broadcast + up

    NMS can't tell RSF-1 to use datanet0 (and doesn't handle IPMP very well in the first place) as the interface when we configure a shared volume via NMS.  So just configure the volume to one of the underlying aggregates and then we'll manually edit the config file.

setup group rsf-cluster <clustername> shared-volume add

    Now edit /opt/HAC/RSF-1/etc/config and under the SERVICE section for each shared volume change theIPDEVICE line from:

IPDEVICE "aggr2601000"

to

IPDEVICE "datanet0"

    Distribute that file to the other node.
    Then re-start the cluster services or simply reboot both nodes.
    Test failover between them and we should be good to go.