Exadata: Cell Patching Fails Because Files Under /boot Are Missing

Asset ID:	1-72-1668197.1
Update Date:	2014-11-17
Keywords:

Solution Type Problem Resolution Sure

Solution 1668197.1 : Exadata: Cell Patching Fails Because Files Under /boot Are Missing

Applies to:

Exadata X3-2 Hardware - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

Patching fails on cell node with below error.

odal08celadm03: Order is ok.
odal08celadm03: Copy root home dir ...
odal08celadm03: Upgrade /boot ...
odal08celadm03: Removing new files based on list /boot/cellboot.backup.11.2.3.2.0.120713.lst ...
odal08celadm03: /boot
odal08celadm03: Restoring old boot files using "tar -x -p -C /boot -f /boot/cellboot.backup.11.2.3.2.0.120713.tar" ...
odal08celadm03: [ERROR] Unable to backup file into /boot/cellboot.backup.11.2.3.2.0.120713.tar
odal08celadm03: ge1_5
odal08celadm03: tar: grub/stage2: Cannot stat: No such file or directory
odal08celadm03: tar: Error exit delayed from previous errors
odal08celadm03: /bin/rm: cannot remove `grub': Is a directory
odal08celadm03: /bin/rm: cannot remove `lost+found': Is a directory
odal08celadm03: grub/iso9660_stage1_5
odal08celadm03: grub/ufs2_stage1_5

[root@odal08celadm03 ~]# cellcli -e alter cell validate configuration

CELL-01514: Connect Error. Verify that Management Server is listening at the specified HTTP port: 8888.
[root@odal08celadm03 ~]# ssh odal08celadm04
Last login: Sun Apr 27 08:10:18 2014 from odal08celadm03.fmr.com

Changes

Patching the cell nodes

Cause

Known Defect: BUG 18470555 - EXADATA : DURING UPGRADE TO 11.2.3.3.0 FILES UNDER /BOOT MISSING

The problem is with the grub.conf that did not have entries for the running kernel. The dostep_cleanup_boot_to_make_space_14261627() cleaned up /boot, but it also wrongly removed the files of current running kernel due to the grub.conf did not have entries for the running kernel.
There was a known Linux OS issue, Bug 13963595, that after new kernel was installed, it failed in updating the grub.conf with the new kernel. This problem was with the previous kernel install on this cell. This grub.conf problem led to the dostep_cleanup_boot_to_make_space_14261627() that cleaned up these files, thinking they are extras and not needed as they are not referenced in the grub.conf.

Solution

1) From successfully patched cell node (say cell node 4) , get the backup file cellboot.backup.11.2.3.2.0.120713.tar.bz2 and copy it to /boot directory on failed node cell node (say cell node 3).

    Copy the file (for example cellboot.backup.11.2.3.2.0.120713.tar.bz2 from cell node 4(working node) from /boot and put it on cell node 3(failing node) under /boot

2) Extract this file under the boot directory in non-working cell(cell node 3) using

    command ===> tar -xvjpf cellboot.backup.11.2.3.2.0.120713.tar.bz2.

3) Validate again using below command.

     cellcli -e alter cell validate configuration

     The output results should show successful ==> Cell odal08celadm03 successfully altered

4) Start with patching and it will complete with success.

References

<BUG:18470555> - EXADATA : DURING UPGRADE TO 11.2.3.3.0 FILES UNDER /BOOT MISSING
<BUG:18667317> - FILES UNDER /BOOT MISSING DURING PATCHING

Attachments

This solution has no attachment