ODA: Kernel Panic INVALIDATE_BH_LRUS during 12.1.2.6.0 Infra Upgrade -- Looping Reboot

Asset ID:	1-72-2140386.1
Update Date:	2016-05-31
Keywords:

Solution Type Problem Resolution Sure

Solution 2140386.1 : ODA: Kernel Panic INVALIDATE_BH_LRUS during 12.1.2.6.0 Infra Upgrade -- Looping Reboot

Applies to:

Oracle Database Appliance - Version All Versions to All Versions [Release All Releases]
Information in this document applies to any platform.

Symptoms

During the 12.1.2.5 -> 12.1.2.6 INFRA upgrade, patching is expected to automatically reboot the nodes.

However during the reboot one node keeps on rebooting in a loop after showing the following in console:

UG: soft lockup - CPU#29 stuck for 22s! [kworker/u:48:704]

Modules linked in: sd_mod(+) crc_t10dif floppy xen_blkfront mpt3sas scsi_transport_sas raid_class pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod

CPU 29

Modules linked in: sd_mod(+) crc_t10dif floppy xen_blkfront mpt3sas scsi_transport_sas raid_class pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod

Pid: 704, comm: kworker/u:48 Not tainted 2.6.39-400.264.13.el6uek.x86_64 #1 Xen HVM domU

RIP: 0010:[<ffffffff810a96a0>] [<ffffffff810a96a0>] smp_call_function_many+0x1f0/0x260

RSP: 0018:ffff883b96b03bd0 EFLAGS: 00000202

RAX: 0000000000000080 RBX: ffff883b96b03b70 RCX: 0000000000000007

RDX: 0000000000000007 RSI: 0000000000000080 RDI: 0000000000000000

RBP: ffff883b96b03c10 R08: ffff883ba5604810 R09: 0000000000000080

R10: 0000000000000002 R11: ffff883b96b03a08 R12: ffffffff81517b0e

R13: ffffffff812fac36 R14: ffff883b96b03b40 R15: ffffffff812fa62e

FS: 0000000000000000(0000) GS:ffff883d0e9a0000(0000) knlGS:0000000000000000

CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b

CR2: 00007fff666bd900 CR3: 0000000001787000 CR4: 00000000001006e0

DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000

DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400

Process kworker/u:48 (pid: 704, threadinfo ffff883b96b02000, task ffff883b96b00340)

Stack:

01ff883b96b03be0 0000000000000000 ffff883b96b03c30 ffffffff8119bad0

0000000000000000 0000000000000001 ffff883b9cde4758 ffff883b969b9800

ffff883b96b03c20 ffffffff810a9732 ffff883b96b03c50 ffffffff810a976b

Call Trace:

[<ffffffff8119bad0>] ? buffer_cpu_notify+0xa0/0xa0

[<ffffffff810a9732>] smp_call_function+0x22/0x30

[<ffffffff810a976b>] on_each_cpu+0x2b/0x70

[<ffffffff8119b5ac>] invalidate_bh_lrus+0x1c/0x20

[<ffffffff811a1dc8>] kill_bdev+0x28/0x50

[<ffffffff811a2f51>] __blkdev_put+0x71/0x1d0

[<ffffffff81099340>] ? async_schedule+0x20/0x20

[<ffffffff811a30e8>] blkdev_put+0x38/0x150

[<ffffffff81246a7c>] register_disk+0x18c/0x1a0

[<ffffffff81099340>] ? async_schedule+0x20/0x20

[<ffffffff81246be5>] add_disk+0x155/0x2f0

[<ffffffffa00b77c1>] sd_probe_async+0x131/0x1f0 [sd_mod]

[<ffffffff810993c4>] async_run_entry_fn+0x84/0x160

[<ffffffff8108a7c6>] process_one_work+0x136/0x450

[<ffffffff8108bda0>] worker_thread+0x170/0x3c0

[<ffffffff8108bc30>] ? manage_workers+0x120/0x120

[<ffffffff81090fe6>] kthread+0x96/0xa0

[<ffffffff81517884>] kernel_thread_helper+0x4/0x10

[<ffffffff81090f50>] ? kthread_worker_fn+0x1a0/0x1a0

[<ffffffff81517880>] ? gs_change+0x13/0x13

Code: e8 a6 4c 46 00 0f ae f0 48 8b 7b 30 ff 15 21 f0 6e 00 80 7d c7 00 0f 84 97 fe ff ff f6 43 20 01 0f 84 8d fe ff ff 0f 1f 44 00 00 <f3> 90 f6 43 20 01 75 f8 e9 7b fe ff ff 0f 1f 00 4c 89 ea 4c 89

Call Trace:

[<ffffffff8119bad0>] ? buffer_cpu_notify+0xa0/0xa0

[<ffffffff810a9732>] smp_call_function+0x22/0x30

[<ffffffff810a976b>] on_each_cpu+0x2b/0x70

[<ffffffff8119b5ac>] invalidate_bh_lrus+0x1c/0x20

[<ffffffff811a1dc8>] kill_bdev+0x28/0x50

[<ffffffff811a2f51>] __blkdev_put+0x71/0x1d0

[<ffffffff81099340>] ? async_schedule+0x20/0x20

[<ffffffff811a30e8>] blkdev_put+0x38/0x150

[<ffffffff81246a7c>] register_disk+0x18c/0x1a0

[<ffffffff81099340>] ? async_schedule+0x20/0x20

[<ffffffff81246be5>] add_disk+0x155/0x2f0

[<ffffffffa00b77c1>] sd_probe_async+0x131/0x1f0 [sd_mod]

[<ffffffff810993c4>] async_run_entry_fn+0x84/0x160

[<ffffffff8108a7c6>] process_one_work+0x136/0x450

[<ffffffff8108bda0>] worker_thread+0x170/0x3c0

[<ffffffff8108bc30>] ? manage_workers+0x120/0x120

[<ffffffff81090fe6>] kthread+0x96/0xa0

[<ffffffff81517884>] kernel_thread_helper+0x4/0x10

[<ffffffff81090f50>] ? kthread_worker_fn+0x1a0/0x1a0

[<ffffffff81517880>] ? gs_change+0x13/0x13

BUG: soft lockup - CPU#30 stuck for 22s! [kworker/u:55:711]

Modules linked in: sd_mod(+) crc_t10dif floppy xen_blkfront m

Changes

12.1.2.5->12.1.2.6

Cause

Unpublished Bug 23002605 - LNX64-121-CMT:CALL TRACES AFTER DEPLOY ODA_BASE AND CAN'T CONFIGURE FIRSTNET

Solution

Go to grub following MOS Note How to reset root password on an Oracle Database Appliance ODA-BASE Virtual Machine <Document 2066895.1>.

Then edit the related grub information: change loglevel=3 ( the original value is 7)

References

<BUG:23316257> - UPGARDE PROCESS KERNEL PANIC AT: INVALIDATE_BH_LRUS WHEN REBOOT.

Attachments

This solution has no attachment