Sun Microsystems, Inc.  Oracle System Handbook - ISO 7.0 May 2018 Internal/Partner Edition
   Home | Current Systems | Former STK Products | EOL Systems | Components | General Info | Search | Feedback

Asset ID: 1-72-1665557.1
Update Date:2018-03-02
Keywords:

Solution Type  Problem Resolution Sure

Solution  1665557.1 :   Oracle ZFS Storage: How To Troubleshoot "Cannot Monitor Target. Incorrect Credentials" with Enterprise Manager Monitoring Plug-in for ZFS Storage Appliance  


Related Items
  • Sun ZFS Storage 7320
  •  
  • Enterprise Manager Ops Center
  •  
  • Sun Storage 7210 Unified Storage System
  •  
  • Oracle ZFS Storage ZS3-BA
  •  
  • Oracle ZFS Storage Appliance Racked System ZS4-4
  •  
  • Oracle ZFS Storage ZS5-4
  •  
  • Oracle ZFS Storage ZS3-2
  •  
  • Oracle ZFS Storage ZS3-4
  •  
  • Sun Storage 7410 Unified Storage System
  •  
  • Sun ZFS Storage 7420
  •  
  • Oracle ZFS Storage ZS5-2
  •  
  • Sun Storage 7310 Unified Storage System
  •  
  • Oracle ZFS Storage ZS4-4
  •  
  • Sun Storage 7110 Unified Storage System
  •  
  • Sun ZFS Storage 7120
  •  
Related Categories
  • PLA-Support>Sun Systems>DISK>ZFS Storage>SN-DK: 7xxx NAS
  •  




In this Document
Symptoms
Changes
Cause
Solution
 Troubleshooting Steps
 Verifying Credentials
  (Re)creating EM Monitoring Template
 Creating A New Template
 If  You Are Replacing An Existing Template
References


Created from <SR 3-8804145661>

Applies to:

Enterprise Manager Ops Center - Version 12c (12.1) to 12cR2 (12.2) [Release 12.0]
Sun Storage 7110 Unified Storage System - Version All Versions to All Versions [Release All Releases]
Oracle ZFS Storage ZS3-2 - Version All Versions to All Versions [Release All Releases]
Oracle ZFS Storage ZS3-4 - Version All Versions to All Versions [Release All Releases]
Sun ZFS Storage 7420 - Version All Versions to All Versions [Release All Releases]
7000 Appliance OS (Fishworks)

Symptoms

 

  • Enterprise Manager reports error "Cannot monitor target. Incorrect Credentials."
  • ZFS Storage Appliance see the ssh log such as below in debug.sys.
  • The symptom goes away when you update oracle_agent password from Enterprise Manager, however it comes back in a few days to a week.

 

[ZFSSA debug.sys]

Apr 3 15:32:10 zfssa01 sshd[22377]: [ID 800047 auth.notice] Failed keyboard-interactive for oracle_agent from 192.168.0.142 port 55298 ssh2
Apr 3 15:52:08 zfssa01 sshd[22444]: [ID 800047 auth.notice] Failed keyboard-interactive for oracle_agent from 192.168.0.142 port 55856 ssh2

 
If you have email notification set up, you receive an email such as in the example below.
 

Subject: EM Event: Critical:ZFSStorageAppliance_zfssa01 - Cannot monitor target. Incorrect Credentials

Host=zfssa01.lab.oracle.com
Target type=Oracle ZFS Storage Appliance
Target name=ZFSStorageAppliance_zfssa01
Categories=Availability
Message=Cannot monitor target. Incorrect Credentials
Severity=Critical
Event reported time=Apr 4, 2014 5:12:00 AM UTC 
Platform=x86_64
Operating System=Linux
Associated Incident Id=1896
Associated Incident Status=New
Associated Incident Owner=
Associated Incident Acknowledged By Owner=No
Associated Incident Priority=None
Associated Incident Escalation Level=0
Event Type=Metric Alert
Event name=MonitoringPrerequisite:Status
Metric Group=MonitoringPrerequisite
Metric=Status
Metric value=0
Key Value=
Rule Name=HW_OS_metric_alert,HW_OS_metric_alert
Rule Owner=SYSMAN
Update Details:
Cannot monitor target. Incorrect Credentials
In

 

Changes

 None

Cause

 The Enterprise Manager monitoring template was either stale for the version of software you have or corrupted.

Solution

  • Recreate Enterprise Manager monitoring template and enter the correct password to oracle_agent user.]
  • Make sure you have the latest version of the plug-in.
  • In order to verify if your ZFS Storage Appliance is responding and you are entering correct password, the below troubleshooting steps are helpful.

Troubleshooting Steps

  1. Refer to the Doc ID 1609846.1  Enterprise Manager reports "Cannot monitor target. Incorrect Credentials" in Incidents and Problems for the ZFS Storage Appliance Target  to verify your oracle_agent user is set up in the way it should be set up.
  2. Make sure if you have the laste version of the plug in.  If not, upgrade.
    The latest version of plug-in can be found at:  http://www.oracle.com/technetwork/oem/grid-control/downloads/zfs-storage-plugin-487867.html  or EM Self-upgdate. 
  3. Verify if we have correct credentials by following below steps described in "Verifying Credentials" section.
  4. Recreate the EM template by following the steps in "(Re)create EM Template" section.

Verifying Credentials

The following steps must be executed on the server where Enterprise Manager and Plug-in is installed.
1) Go under directory where Oracle_Grid.jar is. If you are not sure, run find command to look for it under the directory where agent is installed.  (You can usually find it under <agent_install_dir>/agent/scripts/emx/sun_storage_7000.)
$ pwd
/u01/app/agents/scripts/emx/sun_storage_7000
$ find . -name Oracle_Grid_jar -print
(omitted) 
 

2) Please run the following command.  If it does not run, check your Java Runtime version to conform the requirement by the Enterprise Manager.
 

java -classpath ./Oracle_Grid.jar:./ws-commons-java5-1.0.1.jar:./xmlrpc-common-3.1.2.jar:./xmlrpc-client-3.1.2.jar:./ws-commons-util-1.0.2.jar:./commons-logging-1.1.jar:./commons-codec-1.7.jar:./jsch-0.1.49.jar com.sun.s7000.client.RetrieveSSHResult aie-7210e.us.oracle.com “GetLogicalStatus.aksh"

 

The FQDN host name of aie-7210e.us.oracle.com must be replaced with one for the ZFS Storage Appliance that you are monitoring from Enterprise Manager.

 

3) The command above will wait for your input.  Type the followings and press CTRL+D twice.
 

InstanceUser=oracle_agent
InstancePassword=yourpasswordhere
^D
^D

 
4) Wait for a few minutes.  If the result is login denied, it means the credential information is wrong.  It is likely that you specified the same wrong password at the time you added the ZFSSA as a monitoring target to EM.

    If the password contains special characters, quote the password with single quotes (').  If you have the correct credentials, you see something like below.

[Sample Runlog]
java -classpath ./Oracle_Grid.jar:./ws-commons-java5-1.0.1.jar:./xmlrpc-common-3.1.2.jar:./xmlrpc-client-3.1.2.jar:./ws-commons-util-1.0.2.jar:./commons-logging-1.1.jar:./commons-codec-1.7.jar:./jsch-0.1.49.jar com.sun.s7000.client.RetrieveSSHResult aie-7210e.us.oracle.com "GetLogicalStatus.aksh"
InstanceUser=oracle_agent
InstancePassword=oracle
^D
^D
pool-0:GRACE-GRID-CONTROL/Vol-1,2448.77,0.46,0.46,0.02,0,0,0,false
pool-0:GRACE-GRID-CONTROL/Vol-10,2448.77,1.25,1.25,0.05,0,0,0,false
pool-0:GRACE-GRID-CONTROL/Vol-100,2448.77,0.16,0.16,0.01,0,0,0,false
pool-0:GRACE-GRID-CONTROL/Vol-101,2448.77,0.05,0.05,0,0,0,0,false
pool-0:GRACE-GRID-CONTROL/Vol-102,2448.77,0.06,0.06,0,0,0,0,false
pool-0:GRACE-GRID-CONTROL/Vol-103,2448.77,0.05,0.05,0,0,0,0,false 

 
5)  Another command we can use to verify the credential is as in the sample run log below.  This runs a workflow for EM monitoring on ZFSSA and retrieve result from that workflow.
 

[Sample Runlog #2]

java -classpath ./Oracle_Grid.jar:./ws-commons-java5-1.0.1.jar:./xmlrpc-common-3.1.2.jar:./xmlrpc-client-3.1.2.jar:./ws-commons-util-1.0.2.jar:./commons-logging-1.1.jar:./commons-codec-1.7.jar com.sun.s7000.client.RetrieveWorkflowResults aie-7210e.us.oracle.com 215 "Oracle Get Configuration Service Status" 1.0 includeHeader Boolean false
InstanceUser=oracle_agent
InstancePassword=oracle
^D
^D
ad=disabled
smb=online
dns=online
dynrouting=online
ftp=online
http=online
identity=online
idmap=online
ipmp=online
iscsi=online
ldap=maintenance
ndmp=online
nfs=online
nis=disabled
ntp=disabled
replication=online
scrk=disabled
sftp=online
shadow=online
smtp=online
snmp=online
srp=disabled
ssh=online
syslog=disabled
tags=online
tftp=online
vscan=disable

 

 (Re)creating EM Monitoring Template

Creating A New Template
  1. From the OCC login screen, drop down Enterprise-> Monitoring->Monitoring Templates
  2. Click the "Create" button (or click "Create Like" if you already have a template you want to use as a base.)
  3. On the "Create Monitoring Template: Copy Monitoring Settings" screen, click "Target Type" and select target type “Oracle ZFS Storage Appliance".  Then click "Continue".
  4. Give the template a name/description. IMPORTANT: If you want this to become the default for all ZFS Appliances, check the box that says "Make this the default template for this target type."
  5. Click the "Metric Thresholds" tab and change the thresholds you want to change.  You can add and remove metrics from the template, also.
  6. Click "Continue" to see all thresholds. After making changes, click "OK".  The template will be created.
  7. You are returned to the "Monitoring Templates" screen, and your template should appear.  Click the radio button under "Select" and then click "Apply".
  8. Select between the two Apply Options: "Template will completely replace all metric settings in the target." or "Template will only override metrics that are common to both template and target." As you can see, you could create a template that had only one metric,  and if you apply that with "Template will only override metrics that are common to both template and target.", the template only applies to that one metric, while the others are unmodified.
  9. Click "Add" and select the targets to which you want the template to apply, then click "Select".
  10. Now you must select again the targets on the "Apply Monitoring Template" page.  Select and then click OK.
  11. The template is now applied to the targets you selected.
If  You Are Replacing An Existing Template

Please follow the steps below if you are replacing an existing template.

  1. From the OCC login screen, drop down Enterprise-> Monitoring->Monitoring Templates
  2. Search for the existing template by selecting Target Type as “Oracle ZFS Storage Appliance”.  If you don’t find one, try using “Sun ZFS Storage Appliance”.
  3. Select the template. It should say at the bottom: Default Template for a target type.  This template will be applied automatically to newly discovered targets in Enterprise Manager.
  4. Click “Create” Button.
  5. On the "Create Monitoring Template: Copy Monitoring Settings" screen, click "Target Type" and select target type “Oracle ZFS Storage Appliance". Then click "Continue”.
  6. Give the template a **New** name/description.
    IMPORTANT: If you want this to become the default for all ZFS Appliances, check the box that says “Replace TemplateNameHere as  default template for this target type."
  7. Click OK
  8. It should say “Template will completely replace all metric settings in the target”.
  9. In the Main page you will see the new template is marked as a default template.  If you look at the screen, oracle-Test3 is marked as the default template.

 



 

References

<BUG:18597155> - ZFS ALERT RECEIVED MESSAGE=CANNOT MONITOR TARGET. INCORRECT CREDENTIALS
<NOTE:1609846.1> - Enterprise Manager reports "Cannot monitor target. Incorrect Credentials" in Incidents and Problems for the ZFS Storage Appliance Target.
<NOTE:1609612.1> - EM 12c: Setting Host Target Monitoring Credentials from Enterprise Manager 12c Cloud Control Fails: Error Occurred: Cannot Set Monitoring Credentials At Instance

Attachments
This solution has no attachment
  Copyright © 2018 Oracle, Inc.  All rights reserved.
 Feedback