Sun Microsystems, Inc.  Oracle System Handbook - ISO 7.0 May 2018 Internal/Partner Edition
   Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback

Asset ID: 1-72-1905132.1
Update Date:2014-10-09
Keywords:
Solution Type  Problem Resolution Sure

Solution  1905132.1 :   Exadata: ASM start fails with ORA-56865 due to mismatch in MTU interfaces  

Related Items 
    

  • Exadata X4-2 Hardware
    •  
  • Exadata X3-2 Hardware
    •  
  • Exadata X3-8 Hardware
    •  
  • Exadata Database Machine V2
    •  
Related Categories 
    

  • PLA-Support>Eng Systems>Exadata/ODA/SSC>Oracle Exadata>DB: Exadata_EST
    •  




MTU mismatch in interfaces causes error " ORA-56865: INVALID IP ADDRESS IN CELLINIT.ORA" while startup of ASM on all nodes
bondib0 -Interface with MTU = 65520
bondib1, bondib2, bondib3 - Interfaces with MTU 7000

Created from <SR 3-9243957941>

Applies to:

Exadata Database Machine V2 - Version All Versions and later
Exadata X4-2 Hardware - Version All Versions and later
Exadata X3-8 Hardware - Version All Versions and later
Exadata X3-2 Hardware - Version All Versions and later
Information in this document applies to any platform.

Symptoms

Start of ASM fails with the following error:

ORA-56865: Invalid IP address in CELLINIT.ORA

Changes

Change of MTU for interface one interface  "bondib0" from 7000 to  65520

Cause

Change of MTU for interface one interface  "bondib0" from 7000 to  65520

Solution

After the MTU was changed from 7000 to 65520 for "bondib0", the ASM instance failed to start with ORA-56865: invalid ip address in cellinit.ora.

 

Checked the cellinit.ora - it was correct
==========================
ipaddress1=192.168.15.5/22
ipaddress2=192.168.15.6/22
ipaddress3=192.168.15.7/22
ipaddress4=192.168.15.8/22
==========================

Error is misleading as there was not changed to cellinit.ora

 

Running strace on the startup:   strace -a -e -f -o /tmp/strace2.out sqlplus / as sysasm

This shows the interface "bondib0" with MTU = 65520 while rest of the interfaces bondib1, bondib2, bondib3 have MTU = 7000.

 

...
66726 ioctl(18, SIOCGIFFLAGS, {ifr_name="bondib0", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MASTER|IFF_MULTICAST}) = 0
66726 ioctl(18, SIOCGIFMTU, {ifr_name="bondib0", ifr_mtu=65520}) = 0      <<<<<<<<<<<<<<<<<<<<<<< only one interface bondib0 is having MTU=65520
 :
66726 ioctl(18, SIOCGIFFLAGS, {ifr_name="bondib1", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MASTER|IFF_MULTICAST}) = 0
66726 ioctl(18, SIOCGIFMTU, {ifr_name="bondib1", ifr_mtu=7000}) = 0
 :
66726 ioctl(18, SIOCGIFFLAGS, {ifr_name="bondib2", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MASTER|IFF_MULTICAST}) = 0
66726 ioctl(18, SIOCGIFMTU, {ifr_name="bondib2", ifr_mtu=7000}) = 0
 :
66726 ioctl(18, SIOCGIFFLAGS, {ifr_name="bondib3", ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MASTER|IFF_MULTICAST}) = 0
66726 ioctl(18, SIOCGIFMTU, {ifr_name="bondib3", ifr_mtu=7000}) = 0
 :
66726 times({tms_utime=67, tms_stime=72, tms_cutime=0, tms_cstime=1}) = 430534518
66726 write(4, "ossnet_validate_ips_with_skgxp1:"..., 76) = 76
66726 write(4, "\n", 1) = 1
66726 times({tms_utime=67, tms_stime=72, tms_cutime=0, tms_cstime=1}) = 430534518
66726 write(4, "ossnet_validate_ips_with_skgxp1:"..., 51) = 51
66726 lseek(4, 0, SEEK_CUR) = 2025
...

 


From "ifonfig -a" output of bondib0 interface confirmed MTU = 65520 :

 

bondib0   Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  
          inet addr:192.168.15.5  Bcast:192.168.11.255  Mask:255.255.252.0
          inet6 addr: fe80::221:2800:1ef:55bf/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:65520  Metric:1
          RX packets:27456 errors:0 dropped:1233 overruns:0 frame:0
          TX packets:27529 errors:0 dropped:6 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:5088905 (4.8 MiB)  TX bytes:12209640 (11.6 MiB)
bondib0:1 Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  
          inet addr:169.254.31.246  Bcast:169.254.63.255  Mask:255.255.192.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:65520  Metric:1
          
ib0       Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:65520  Metric:1
          RX packets:26227 errors:0 dropped:4 overruns:0 frame:0
          TX packets:27529 errors:0 dropped:6 overruns:0 carrier:0
          collisions:0 txqueuelen:1024
          RX bytes:4967091 (4.7 MiB)  TX bytes:12209640 (11.6 MiB)
          
ib1       Link encap:InfiniBand  HWaddr 80:00:00:49:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:65520  Metric:1
          RX packets:1229 errors:0 dropped:1229 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1024
          RX bytes:121814 (118.9 KiB)  TX bytes:0 (0.0 b)

 

   
Interfaces bondib1, bondib2, bondib3 confirmed MTU = 7000:

 

bondib1   Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  
          inet addr:192.168.15.6  Bcast:192.168.11.255  Mask:255.255.252.0
          inet6 addr: fe80::225:2800:1ef:55c3/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:7000  Metric:1
          RX packets:14529 errors:0 dropped:1078 overruns:0 frame:0
          TX packets:11933 errors:0 dropped:6 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2292558 (2.1 MiB)  TX bytes:1954655 (1.8 MiB)

bondib1:1 Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  
          inet addr:165.155.33.71  Bcast:165.155.33.255  Mask:255.255.192.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:7000  Metric:1

bondib2   Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  
          inet addr:192.168.15.7  Bcast:192.168.11.255  Mask:255.255.252.0
          inet6 addr: fe80::225:2800:1ef:56ef/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:7000  Metric:1
          RX packets:14348 errors:0 dropped:933 overruns:0 frame:0
          TX packets:11971 errors:0 dropped:6 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2169401 (2.0 MiB)  TX bytes:1684160 (1.6 MiB)

bondib2:1 Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  
          inet addr:165.155.33.28  Bcast:165.155.33.255  Mask:255.255.192.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:7000  Metric:1

bondib3   Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  
          inet addr:192.168.15.8  Bcast:192.168.11.255  Mask:255.255.252.0
          inet6 addr: fe80::225:2800:1ef:56d3/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:7000  Metric:1
          RX packets:14053 errors:0 dropped:825 overruns:0 frame:0
          TX packets:11847 errors:0 dropped:6 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2261537 (2.1 MiB)  TX bytes:1960013 (1.8 MiB)

bondib3:1 Link encap:InfiniBand  HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00  
          inet addr:165.155.33.220  Bcast:165.155.33.255  Mask:255.255.192.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:7000  Metric:1

 



Change MTU back to 7000 to resolve the issue.

Login as root user:

ifconfig <interface> mtu <new value>

 

References

<BUG:19153318> - MERGE REQUEST ON TOP OF EXADATA DATABASE BP 11.2.0.3.20 FOR BUGS 12755502
<NOTE:1495104.1> - Recommended Patches for Direct NFS Client
<BUG:19152399> - MULTIPLE PRODUCTION NODES CRASH
<NOTE:16314214.8> - Bug 16314214 - ORA-56865: invalid ip address in cellinit.ora
<BUG:19169513> - EXADATA: ORA-56865: INVALID IP ADDRESS IN CELLINIT.ORA

Attachments 
This solution has no attachment
  Copyright © 2018 Oracle, Inc.  All rights reserved.
 Feedback