Sun Fire[TM] 12K - E25K / Sparc Enterprise Mx000: WARNING: fcode_timer: Timeout waiting for interpreter

Asset ID:	1-72-1003550.1
Update Date:	2017-07-10
Keywords:

Solution Type Problem Resolution Sure

Solution 1003550.1 : Sun Fire[TM] 12K - E25K / Sparc Enterprise Mx000: WARNING: fcode_timer: Timeout waiting for interpreter - Interpreter did not pick up request

Applies to:

Sun SPARC Enterprise M5000 Server - Version Not Applicable and later
Sun Fire 15K Server - Version Not Applicable and later
Sun Fire 12K Server - Version Not Applicable and later
Sun Fire E20K Server - Version Not Applicable and later
Sun Fire E25K Server - Version Not Applicable and later
All Platforms

Symptoms

Issue may occur on different platforms; same messages log "fcode_timer: Timeout waiting for interpreter - Interpreter did not pick up request" will appear.

When trying to DR (Dynamic Reconfiguration) a component into Sun Fire[TM] 12K-25K domain, it fails to add the board with the following message:

    # /usr/sbin/cfgadm -v -c configure IO5
    assign IO5
    assign IO5 done
    poweron IO5
    poweron IO5 done
    test IO5
    test IO5 done
    connect IO5
    cfgadm: Hardware specific failure: connect IO5: Solaris[TM] failed to probe: IO5

The Starcat domain's /var/adm/messages file, or the domain console log file on the System Controller shows the following errors:

    Mar 20 11:45:42 Domain-A fcodem: [ID 696114 kern.warning] WARNING: fcode_timer: Timeout waiting for interpreter - Interpreter did not pick up request
    Mar 20 11:45:42 Domain-A gptwo_pci: [ID 687272 kern.warning] WARNING: gptwo_pci: Unable to probe pci leaf bc
    Mar 20 11:46:27 Domain-A fcodem: [ID 696114 kern.warning] WARNING: fcode_timer: Timeout waiting for interpreter -Interpreter did not pick up request
    Mar 20 11:46:27 Domain-A gptwo_pci: [ID 687272 kern.warning] WARNING: gptwo_pci: Unable to probe pci leaf bc
    Mar 20 11:47:12 Domain-A fcodem: [ID 696114 kern.warning] WARNING: fcode_timer: Timeout waiting for interpreter -Interpreter did not pick up request
    Mar 20 11:47:12 Domain-A gptwo_pci: [ID 687272 kern.warning] WARNING: gptwo_pci: Unable to probe pci leaf bd
    Mar 20 11:47:57 Domain-A fcodem: [ID 696114 kern.warning] WARNING: fcode_timer: Timeout waiting for interpreter -Interpreter did not pick up request
    Mar 20 11:47:57 Domain-A gptwo_pci: [ID 687272 kern.warning] WARNING: gptwo_pci: Unable to probe pci leaf bd
    Mar 20 11:47:57 Domain-A genunix: [ID 408114 kern.info] /address-extender-queue@bf,0 (axq9) offline
    Mar 20 11:48:00 Domain-A dcs: [ID 397954 daemon.error] <21638> config_change_state: Hardware specific failure:
    connect IO5: Solaris failed to probe: IO5

Performing an addboard on a Mx000 system will produce the following output:

&nbsp;&nbsp;&nbsp; XSCF&gt; <span class="hiddenSpellError" pre="">addboard</span> -c configure -d 0 01-0
&nbsp;&nbsp;&nbsp; XSB#01-0 will be configured into DomainID 0. Continue?[y|n] :y
&nbsp;&nbsp;&nbsp; Initial diagnosis started. [1800sec]
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 0..... 30..... 60..... 90.....120.....150.....180.....210..end
&nbsp;&nbsp;&nbsp; Initial diagnosis has completed.
&nbsp;&nbsp;&nbsp; Start configuring XSB to domain.

    connect SB4: Firmware probe failed
    XSB#01-0 could not be configured into DomainID 0 due to operating system error.

The Mx000 domain /var/adm/messages will have messages:

    Dec 30 03:20:40 domain fcodem: [ID 696114 kern.warning] WARNING: fcode_timer: Timeout waiting for interpreter - Interpreter did not pick up request
    Dec 30 03:20:40 domain opl_cfg: [ID 695635 kern.warning] WARNING: IKP: Unable to probe PCI leaf (1-4-0)
    Dec 30 03:20:40 domain drmach: [ID 702911 kern.warning] WARNING: Firmware probe failed
    Dec 30 03:20:40 domain dcs: [ID 223655 daemon.error] <17045> config_change_state: Hardware specific failure: connect SB1: Firmware probe failed

Changes

Cause

Issue is related to efdaemon, that runs in all Solaris releases.

If efdaemon is not running, the errors on a DR operation as shown above will occur.

The efdaemon can die due to resource problems on the domain or it may have been disabled in error.

In Solaris 9 or below, ps command will allow to check if efdaemon is running

# ps -ef | grep efd
root 1395 1 0 04:02:18 ? 0:00 /usr/lib/efcode/sparcv9/efdaemon

In Solaris[TM] 10 or above, the svcs command will show the status of efdaemon:

# svcs efdaemon
disabled Dec_21 svc:/platform/sun4u/efdaemon:default

Note: Solaris 10 or above is mandatory on Mx000 platforms.

Solution

efdaemon status needs to be checked in order to fix the issue.

In Solaris 9 and below, efdaemon is affected by know bug (efdaemon dying and not being automatically restarted): a fix is provided by Patch ID: 112988-01 for Solaris[TM] 8 and Patch ID: 113224-01 for Solaris[TM] 9. Else, messages and daemon core dumps (if any) will need to be cheked.

In Solaris 10 or above, the service may have been disabled in error: use the svcadm command to re-enable the service.
# svcadm enable /platform/sun4u/efdaemon
Eventually, daemon service logs can be checked:
logfile /var/svc/log/platform-sun4u-efdaemon:default.log

Example of recovery on a SF25K domain with Solaris 9 OS:

# ps -ef | grep efd
root 1563 14146 0 13:29:00 console 0:00 grep efd

The efdaemon is confirmed not to be running, so restart it:

# /usr/lib/efcode/sparcv9/efdaemon

# ps -ef | <span class="hiddenSpellError" pre="">grep</span> efd

root  1533     1  0 13:34:38          0:00 /usr/lib/efcode/sparcv9/efdaemon

root  1563 14146  0 13:35:00 console  0:00 <span class="hiddenSpellError" pre="">grep</span> efd

Once restarted, DR operations should function normally:

# <span class="hiddenSpellError" pre="">cfgadm</span> -v -c configure IO5

assign IO5
assign IO5 done
<span class="hiddenSpellError" pre="">poweron</span> IO5
<span class="hiddenSpellError" pre="">poweron</span> IO5 done
test IO5
test IO5 done
connect IO5
connect IO5 done
configure IO5

The domain's /var/adm/messages file and the domain console log file on the System Controller will show the successful DR operation:

Mar 20 13:36:16 Domain-A dr: OS configure dr@0:IO5::pci0
Mar 20 13:36:16 Domain-A dr: OS configure dr@0:IO5::pci1
Mar 20 13:36:16 Domain-A dr: OS configure dr@0:IO5::pci2
Mar 20 13:36:16 Domain-A dr: OS configure dr@0:IO5::pci3

Internal Section

Reference
Bug ID 4710558 and Bug ID 4633009;

Keywords: 12K, 15K, 20K, 25K, Starcat, fcode_timer, DR, dr, cfgadm, Hardware specific failure

Previously Published As 75023

References

<BUG:15095726> - SUNBT4633009 CONFIGURE A CPCI CARD HUNG ON FC_REQUEST_CV
<BUG:15113933> - SUNBT4710558 DEAD EFDAEMON CAUSES IO CONFIGURE FAIL WITH "SOLARIS FAILED TO PROB

Attachments

This solution has no attachment