Exalogic Virtual: EC-templates Patching Fails During Patching to PSUs Later than April 2017 Due To EMOC Control vServer Panic and Crash

Asset ID:	1-72-2294632.1
Update Date:	2018-03-27
Keywords:

Solution Type Problem Resolution Sure

Solution 2294632.1 : Exalogic Virtual: EC-templates Patching Fails During Patching to PSUs Later than April 2017 Due To EMOC Control vServer Panic and Crash

Applies to:

Oracle Exalogic Elastic Cloud Software - Version 2.0.6.2.170418 and later
Exalogic Elastic Cloud X5-2 Hardware - Version X5 to X5 [Release X5]
Linux x86-64
Oracle Virtual Server x86-64

Symptoms

When applying April 2017 or later PSU's to Exalogic Virtual racks, after running the following commands to patch the control vServers,

[root@compute-node1]# cd /exalogic-lcdata/patches/Virtual/2XXXXXXXX/Infrastructure
[root@compute-node1]# /exalogic-lctools/bin/exapatch -a patch ectemplates

patching fails with a timeout ( actually the failure shows "TypeError: cannot concatenate 'str' and 'int' objects" due to Bug 23710536 ) and a stack trace similar to Doc ID 2037365.1 - "EC-templates Patching Failing During April 2015 PSU Application Due To EMOC Control vServer Panic and Crash". It can be seen by manually starting the ExalogicControl vserver with the -c console flag:

Loading OpenIB kernel modules:.. Loading XEN_FMR_Slavedivide error: 0000 [#1] SMP
CPU 0
Modules linked in: mlx4_core(+) dm_mirror video sbs sbshc hed acpi_memhotplug acpi_ipmi ipmi_msghandler lp serio_raw pata_acpi parport_pc parport ata_piix floppy pcspkr ata_generic i2c_piix4 i2c_core dm_region_hash dm_log dm_mod ext3 jbd mbcache xen_netfront xen_blkfront

Pid: 1565, comm: insmod Not tainted 2.6.39-400.294.7.el5uek #1 Xen HVM domU
RIP: 0010:[<ffffffffa015b805>] [<ffffffffa015b805>] mlx4_slave_cap+0x145/0x250 [mlx4_core]
RSP: 0018:ffff8807dab4d918 EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 000000000000000f RDI: 0000000000007fff
RBP: ffff8807dab4d948 R08: 0000000000000000 R09: 000000000000000f
R10: ffffffff819b1640 R11: 000000000000000a R12: 000000000000001c
R13: ffff8807de350000 R14: 0000000000000000 R15: ffffffff819b1640
FS: 00007fae3a6606e0(0000) GS:ffff88080fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fae3a602000 CR3: 00000007dc65b000 CR4: 00000000001006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process insmod (pid: 1565, threadinfo ffff8807dab4c000, task ffff8807db6381c0)
Stack:
ffff8807dab4d938 ffff8807de350000 0000000000000000 ffff8807df756000
ffff8807df756090 0000000000080000 ffff8807dab4dc38 ffffffffa015b9c9
ffff8807daf8ef40 ffff8807daf8ef58 ffff8807dab4da40 ffff8807dab4d9c8
Call Trace:
[<ffffffffa015b9c9>] mlx4_init_hca+0xb9/0x420 [mlx4_core]
[<ffffffff810dd965>] ? call_rcu_sched+0x15/0x20
[<ffffffff81142a73>] ? __free_vmap_area+0xd3/0x100
[<ffffffff81049c21>] ? __cpa_process_fault+0x81/0xa0
[<ffffffff8104a546>] ? __change_page_attr+0xc6/0x250
[<ffffffff8104a751>] ? __change_page_attr_set_clr+0x81/0xd0

Changes

Applied April 2017 PSU or later PSU to a Virtual Exalogic environment.

Cause

Compute node 1's patching was skipped. The OpenIB kernel modules are new to Exalogic Virtual starting from the April 2017 PSU. If the compute node (dom0) is still running the older IB modules, the control vServer will panic as it can't communicate with the Dom0 (compute node).

Solution

To resolve this issue patch Compute node 1 per the patching instructions and then bring the control stack back up with exabr and re-run the patching commands for upgrading ectemplates.

References

<BUG:23710536> - TYPEERROR: CANNOT CONCATENATE 'STR' AND 'INT' OBJECTS IF T/O OCCURS
<NOTE:2248213.1> - fwverify fails after restoring the configuration of IB switch
<NOTE:2223662.1> - Master Note For Exalogic Infiniband Switch Replacement – Overview and guide to key articles
<NOTE:2234640.1> - (EX35) Fast node death detection (FNDD) may improperly evict a database server when combining Exadata 12.1.2.3.4 or lower with InfiniBand switch version 2.2.4 or 2.2.2
<NOTE:2156050.1> - Known Issues for Exalogic 2.0.6.2.X Patch Set Updates

Attachments

This solution has no attachment