Exadata Database Machine V2
 

PLA-Support>Sun Systems>x86>Engineered Systems HW>SN-x64: EXADATA
 

When trying to boot a cell node via PXE, the boot doesn't succeed and fails on all the NICs with this message : "Base-code ROM ID structure was not found.".
Boot on disk works fine.

In this Document

Symptoms

Changes

Cause

Solution

References

Created from <SR 3-7939136892>

Applies to:

Symptoms

On a cell node, after having selected the NIC (Network Interface) as a boot device to boot PXE, the system does not boot with the following errors :

Changes

This symptom occurs when the administrator changes the boot device from disks to the NIC so that the system boots over the network.

Cause

By default, all the PCI option ROMs are enabled on the PCI devices and NICs.

In order to boot on a specific device (HBA, NIC...), these option ROMs need to be loaded into memory during the system initialization process.

The PC memory space to load the option ROMs are limited between the 640KB and 1024KB boundaries of conventional and extended RAM.

On the cell nodes of the Exadata system, we have got 6 additional PCI-E cards (1x HBA to control the disks, 1x HCA for Infiniband purpose, 4x F20 cards hosting the FMODs).

The BIOS scans and attributes space for option ROM to these PCI-E devices first, before scanning the internal NICs.

The issue is that the available memory space for the option ROMs gets exhausted before the NICs can load their option ROM (despite of having the "option ROM" enabled in the BIOS).

This issue is very common across all PC architectures from all vendors as it dates back to the initial design of the x86 processor and how a platform is configured during early boot stages.

As there is not enough space to load the NIC options ROM into memory, the platform's NICs are not bootable.

Solution

In order to be able to boot from the network, we need to disable the option ROM on the devices which are not expected to boot (This will not impact the availability of these devices from the O/S as an O/S driver will still initialize and use the PCI device as intended. This is only for the initial boot process).

In our cell node, we just use the HBA in slot 0 to boot the disks so all the other PCI devices can have their option ROMs can be disabled.

Note: At first, we just disabled 3 option ROMs during tests and the system was still unable to boot on the network. So, don't hesitate to disable the option ROM on all the PCI devices not hosting a bootable device.

When the system boots, hit F2 to enter into the BIOS.

In the Boot tab, select the entry "Option ROM Enable" and then, disable all the PCI slot except the slot 0.

You should get something similar to :

* NET0 Option ROM:               [Enabled]  

* NET1 Option ROM:               [Enabled]  

* NET2 Option ROM:               [Enabled]  

* NET3 Option ROM:               [Enabled]           

* PCI-E Slot 0 Option ROM:       [Enabled]

* PCI-E Slot 1 Option ROM:       [Disabled]

* PCI-E Slot 2 Option ROM:       [Disabled]

* PCI-E Slot 3 Option ROM:       [Disabled]

* PCI-E Slot 4 Option ROM:       [Disabled]

* PCI-E Slot 5 Option ROM:       [Disabled]

Then you can save and exit BIOS (the changes will be persistent).

That should allow you to boot on the NICs (as well as on the disks, following the boot device you choose).

References

This solution has no attachment