Sun Fire[TM] 12K/15K/E20K/E25K: IPMP with Dynamic Reconfiguration (DR)

Asset ID:	1-71-1005025.1
Update Date:	2016-06-01
Keywords:

Solution Type Technical Instruction Sure

Solution 1005025.1 : Sun Fire[TM] 12K/15K/E20K/E25K: IPMP with Dynamic Reconfiguration (DR)

Applies to:

Sun Fire 12K Server - Version Not Applicable and later
Sun Fire 15K Server - Version Not Applicable and later
Solaris SPARC Operating System - Version 8.0 and later
Sun Fire E20K Server - Version Not Applicable and later
Sun Fire E25K Server - Version Not Applicable and later
All Platforms

Goal

This doc assumes familiarity with IPMP (IP Multipathing) and it does not contain description on how to configure IPMP.

It only describes the procedures to remove a network interface from an IPMP multipathing group and subsequently add it back using Dynamic Reconfiguration (DR).

Dynamic Reconfiguration (IPMP RCM Module) for IPMP, works with IPMP in.mpathd(1M) daemon to failover or failback network interfaces that are part of a DR operation so that IPMP is aware of the current DR operation.

Restricted by IPMP, a network interface cannot be shared between network groups. It can not combine IPv4 and IPv6 protocols in the same multipathing group although DR with IPMP works for both IPv4 and IPv6.

The DR procedures described here is for IPMP with IPv4.

It is important to note that each network interface in the domain should have a unique MAC address. Please ensure setting local-mac-address? to true in the OpenBoot PROM. Also any network interface that has been certified to work with IPMP would be able to work under DR.

Solaris[TM] releases 8 9 and 10 have been verified to work with IPMP and DR. Below is the software and patch matrix required to run IPMP with DR properly:

Solaris 8

108528 Solaris Kernel Patch (in.mpathd, ip, icmp, ifconfig)

109898 ARP Patch

109900 /etc/init.d/network and /sbin/ifparse Patch

109902 in.ndpd multipathing offlining IPMP capability (Update 3)

110378 mipagent patch needed for offlining capability

111802 Introduce RCM suspend/resume (Update 6)

110839 introduce RCM IPMP module (Update 6)

Solaris 9

112233 Solaris Kernel Patch (ip)

112911 ifconfig, in.mpathd and in.routed Patch

112912 libinetcfg Patch

112854 icmp Patch

113464 IPMP Headers Patch

Solaris 10

118833 Solaris kernel patch

120011 Solaris kernel patch

141532 in.mpathd patch

Solution

Here are the steps to DR attach and detach a Cassini Enternet (CE) network interface in a IPMP multiple active paths without standby configuration (i.e active-active setup). It works for active-standy configuration as well:

1. Verify the first network interface IPMP settings.

# cat /etc/hostname.ce0
129.158.142.253 netmask + broadcast + group mytestgroup up \
addif 129.158.142.61 deprecated -failover netmask + broadcast + up

2. Verify the second network interface IPMP settings.

# cat /etc/hostname.ce2
129.158.142.59 netmask + broadcast + group mytestgroup up \
addif 129.158.142.58 deprecated -failover netmask + broadcast + up

3. Verify domain IPMP failback is enabled.

# cat /etc/default/mpathd | grep FAILBACK
FAILBACK=yes

4. Verfiy the domain IPMP settings.

# ifconfig -a
lo0: flags=1000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=1000843 mtu 1500 index 2
inet 129.158.142.253 netmask ffffff00 broadcast 129.158.142.255
groupname mytestgroup
ether 0:3:ba:e:91:d3
ce0:1: flags=9040843 mtu 1500 index 2
inet 129.158.142.61 netmask ffffff00 broadcast 129.158.142.255
ce2: flags=1000843 mtu 1500 index 3
inet 129.158.142.59 netmask ffffff00 broadcast 129.158.142.255
groupname mytestgroup
ether 0:3:ba:e:91:77
ce2:1: flags=9040843 mtu 1500 index 3
inet 129.158.142.58 netmask ffffff00 broadcast 129.158.142.255
dman0: flags=1008843 mtu 1500 index 4
inet 192.168.102.13 netmask ffffffe0 broadcast 192.168.102.31
ether 0:0:be:ab:3e:c1

5. Obtain the Attachment Point(Ap_Id) of the network interface to be detached.

# cfgadm -l
Ap_Id                          Type         Receptacle   Occupant     Condition
IO7                            MCPU         connected    configured   ok
IO8                            HPCI         connected    configured   ok
SB7                            CPU          connected    configured   ok
SB8                            CPU          connected    configured   ok
c0                             scsi-bus     connected    configured   unknown
c1                             scsi-bus     connected    unconfigured unknown
c2                             scsi-bus     connected    unconfigured unknown
c3                             scsi-bus     connected    unconfigured unknown
pcisch0:e08b1slot1             pci-pci/hp   connected    configured   ok
pcisch1:e08b1slot0             unknown      connected    unconfigured unknown
pcisch2:e08b1slot3             pci-pci/hp   connected    configured   ok
pcisch3:e08b1slot2             unknown      connected    unconfigured unknown
#

6. Execute cfgadm command to unconfigure the network interface, ce2 in

this case (pcisch2:e08b1slot3).

# cfgadm -c  unconfigure pcisch2:e08b1slot3
May  6 15:03:43 v4u-15kb-l in.mpathd[92]: Successfully failed over from NIC ce2 to NIC ce0

7. Review the domain /var/adm/messages file and you should be seeing

the following output.

May 10 14:08:21 v4u-15kb-l in.mpathd[92]: [ID 832587 daemon.error] Successfully failed over from NIC ce2 to NIC ce0
May 10 14:08:21 v4u-15kb-l in.routed[111]: [ID 970160 daemon.notice] unable to get interface flags for ce2: No such device or address
May 10 14:08:21 v4u-15kb-l in.routed[111]: [ID 970160 daemon.notice] unable to get interface flags for ce2:1: No such device or address
May 10 14:08:21 v4u-15kb-l in.routed[111]: [ID 472501 daemon.notice] ce2:1 has no ifIndex: No such device or address
May 10 14:08:22 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/network@0 (ce2) offline
May 10 14:08:22 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/network@1 (ce3) offline
May 10 14:08:22 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/scsi@2,1 (glm3) offline
May 10 14:08:22 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/network@0 (ce2) offline
May 10 14:08:22 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/network@1 (ce3) offline
May 10 14:08:22 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/scsi@2 (glm2) offline
May 10 14:08:22 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/scsi@2,1 (glm3) offline
May 10 14:08:22 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1 (pci_pci1) offline

8. Confirm the network IP has been failed over successfully.

# ifconfig -a
lo0: flags=1000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=1000843 mtu 1500 index 2
inet 129.158.142.253 netmask ffffff00 broadcast 129.158.142.255
groupname mytestgroup
ether 0:3:ba:e:91:d3
ce0:1: flags=9040843 mtu 1500 index 2
inet 129.158.142.61 netmask ffffff00 broadcast 129.158.142.255
ce0:2: flags=1000843 mtu 1500 index 2
inet 129.158.142.59 netmask ffffff00 broadcast 129.158.142.255
dman0: flags=1008843 mtu 1500 index 4
inet 192.168.102.13 netmask ffffffe0 broadcast 192.168.102.31
ether 0:0:be:ab:3e:c1

9. Verify the Ap_Id Occupant is in "unconfigured" state.

# cfgadm -l | grep pcisch2:e08b1slot3
pcisch2:e08b1slot3             unknown      connected    unconfigured unknown

10. Proceed to disconnect the network interface.

# cfgadm -c disconnect pcisch2:e08b1slot3

11. Verify the Ap_Id Receptable is now in "disconnected" state.

# cfgadm -l | grep pcisch2:e08b1slot3
pcisch2:e08b1slot3             unknown      disconnected unconfigured unknown

12. Step 6 and 10 above can be combined onto a single command line.

# cfgadm -c disconnect pcisch2:e08b1slot3

13. Physically remove and replace the network interface.

14. After the network interface has inserted, perform DR connect to connect the network interface.

# cfgadm -c connect pcisch2:e08b1slot3

15. Review the domain /var/adm/messages file you should be seeing the following messages.

May 10 14:35:06 v4u-15kb-l pcisch: [ID 370704 kern.info] PCI-device: pci@1,pci_pci1
May 10 14:35:06 v4u-15kb-l genunix: [ID 936769 kern.info] pci_pci1 is /pci@11d,70000/pci@1
May 10 14:35:06 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1 (pci_pci1) online
May 10 14:35:07 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1 (pci_pci1) offline
May 10 14:35:07 v4u-15kb-l pcisch: [ID 370704 kern.info] PCI-device: pci@1,pci_pci1
May 10 14:35:07 v4u-15kb-l genunix: [ID 936769 kern.info] pci_pci1 is /pci@11d,70000/pci@1
May 10 14:35:07 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1 (pci_pci1) online
May 10 14:35:07 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/network@0 (ce2) online
May 10 14:35:08 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/network@1 (ce3) online
May 10 14:35:08 v4u-15kb-l scsi: [ID 365881 kern.info] /pci@11d,700000/pci@1/scsi@2 (glm2):
May 10 14:35:08 v4u-15kb-l      glm2 supports power management.
May 10 14:35:08 v4u-15kb-l scsi: [ID 365881 kern.info] /pci@11d,700000/pci@1/scsi@2 (glm2):
May 10 14:35:08 v4u-15kb-l      Rev. 7 Symbios 53c896 found.
May 10 14:35:08 v4u-15kb-l pci_pci: [ID 370704 kern.info] PCI-device: scsi@2, glm2
May 10 14:35:08 v4u-15kb-l genunix: [ID 936769 kern.info] glm2 is /pci@11d,70000/pci@/scsi@2
May 10 14:35:08 v4u-15kb-l pci_pci: [ID 370704 kern.info] PCI-device: scsi@2, glm2
May 10 14:35:08 v4u-15kb-l genunix: [ID 936769 kern.info] glm2 is /pci@11d,700000/pci@1/scsi@2
May 10 14:35:08 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/scsi@2 (glm2) online
May 10 14:35:09 v4u-15kb-l genunix: [ID 408822 kern.info] NOTICE: ce2: no fault external to device; service available
May 10 14:35:09 v4u-15kb-l genunix: [ID 611667 kern.info] NOTICE: ce2: xcvr addr:0x01 - link up 100 Mbps full duplex
May 10 14:35:19 v4u-15kb-l scsi: [ID 365881 kern.info] /pci@11d,700000/pci@1/scsi@2,1 (glm3):
May 10 14:35:19 v4u-15kb-l      glm3 supports power management.
May 10 14:35:19 v4u-15kb-l scsi: [ID 365881 kern.info] /pci@11d,700000/pci@1/scsi@2,1 (glm3):
May 10 14:35:19 v4u-15kb-l      Rev. 7 Symbios 53c896 found.
May 10 14:35:19 v4u-15kb-l pci_pci: [ID 370704 kern.info] PCI-device: scsi@2,1, glm3
May 10 14:35:19 v4u-15kb-l genunix: [ID 936769 kern.info] glm3 is /pci@11d,700000/pci@1/scsi@2,1
May 10 14:35:19 v4u-15kb-l pci_pci: [ID 370704 kern.info] PCI-device: scsi@2,1, glm3
May 10 14:35:19 v4u-15kb-l genunix: [ID 936769 kern.info] glm3 is /pci@11d,700000/pci@1/scsi@2,1
May 10 14:35:19 v4u-15kb-l genunix: [ID 408114 kern.info] /pci@11d,700000/pci@1/scsi@2,1 (glm3) online

16. Verify the Ap_Id Receptable is now in "connected" state.

# cfgadm -l | grep pcisch2:e08b1slot3
pcisch2:e08b1slot3             unknown      connected    unconfigured unknown

17. Proceed to DR configure the network interface.

# cfgadm -c configure pcisch2:e08b1slot3

18. Verify the Ap_Id Occupant is now in "configured" state.

# cfgadm -l | grep pcisch2:e08b1slot3
pcisch2:e08b1slot3             pci-pci/hp   connected    configured   ok

19. Step 14 and 17 above can be combined onto a single command line.

20. Verify the network IP has been failed back.

# ifconfig -a
lo0: flags=1000849 mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
ce0: flags=1000843 mtu 1500 index 2
inet 129.158.142.253 netmask ffffff00 broadcast 129.158.142.255
groupname mytestgroup
ether 0:3:ba:e:91:d3
ce0:1: flags=9040843 mtu 1500 index 2
inet 129.158.142.61 netmask ffffff00 broadcast 129.158.142.255
dman0: flags=1008843 mtu 1500 index 4
inet 192.168.102.13 netmask ffffffe0 broadcast 192.168.102.31
ether 0:0:be:ab:3e:c1
ce2: flags=1000843 mtu 1500 index 5
inet 129.158.142.59 netmask ffffff00 broadcast 129.158.142.255
groupname mytestgroup
ether 0:3:ba:e:91:77
ce2:1: flags=9040843 mtu 1500 index 5
inet 129.158.142.58 netmask ffffff00 broadcast 129.158.142.255

IPMP, Internet Protocol Multi-Pathing, DR, Dynamic Reconfiguration, cfgadm

Previously Published As 76037

Attachments

This solution has no attachment