Exalogic Virtual: Uploading Template From EMOC Failing At "Import Images from URL" Task

Asset ID:	1-72-2086285.1
Update Date:	2017-07-19
Keywords:

Solution Type Problem Resolution Sure

Solution 2086285.1 : Exalogic Virtual: Uploading Template From EMOC Failing At "Import Images from URL" Task

Applies to:

Oracle Exalogic Elastic Cloud Software - Version 2.0.6.0.0 and later
Exalogic Elastic Cloud X3-2 Hardware - Version X3 to X3 [Release X3]
Linux x86-64
Oracle Virtual Server x86-64

Symptoms

In Exalogic Virtual releases 2.0.6.X, issue of vServer template upload failing at "Import Images from URL" task is seen.

Below is snippet of template upload job log which shows failure at "Import Images from URL" task.

Task : Import Images from URL
Task Run ID : 2373
Target : /
Status : FAILED
Result : Import VM Template Failed. Exception occurred: [http://192.168.XX.XX:8003/tmpResourceFile.1448987760127.tgz]. (63080)
Logs :
12/01/2015 11:33:12 AM EST INFO vaxvelcxp01.148.2.2373 acquired the local lock on the target - ecu-pc-IPoIB-vserver-shared-storage-primary-ad9e5d46/com.sun.hss.domain:name="sr:///0004fb0000030000f1313bad96c20033",type=StorageRepository for the duration of Task
12/01/2015 11:33:12 AM EST INFO Task is posted to the Proxy Controller (ecu-pc-IPoIB-vserver-shared-storage-primary-ad9e5d46), from the Enterprise Controller. (15011)
12/01/2015 11:33:13 AM EST INFO Task is received by the Proxy Controller. (15012)
12/01/2015 11:33:13 AM EST INFO Importing the VirtualMachineTemplate in OVM
12/01/2015 11:33:13 AM EST INFO About To Launch OVM Job: Import VM Template. (63065)
12/01/2015 11:33:18 AM EST INFO vaxvelcxp01.148.2.2373 released the local lock on the target - ecu-pc-IPoIB-vserver-shared-storage-primary-ad9e5d46/com.sun.hss.domain:name="sr:///0004fb0000030000f1313bad96c20033",type=StorageRepository for the duration of Task
12/01/2015 11:38:53 AM EST INFO vaxvelcxp01.148.2.2373 acquired the local lock on the target - ecu-pc-IPoIB-vserver-shared-storage-primary-ad9e5d46/com.sun.hss.domain:name="sr:///0004fb0000030000f1313bad96c20033",type=StorageRepository for the duration of Task
12/01/2015 11:38:53 AM EST INFO Task is posted to the Proxy Controller (ecu-pc-IPoIB-vserver-shared-storage-primary-ad9e5d46), from the Enterprise Controller. (15011)
12/01/2015 11:38:58 AM EST INFO Task is received by the Proxy Controller. (15012)
12/01/2015 11:38:58 AM EST INFO Importing the VirtualMachineTemplate in OVM
12/01/2015 11:38:58 AM EST INFO About To Launch OVM Job: Import VM Template. (63065)
12/01/2015 11:38:58 AM EST INFO vaxvelcxp01.148.2.2373 released the local lock on the target - ecu-pc-IPoIB-vserver-shared-storage-primary-ad9e5d46/com.sun.hss.domain:name="sr:///0004fb0000030000f1313bad96c20033",type=StorageRepository for the duration of Task

The issue happens when uploading both Exalogic base guest templates and custom templates.

Cause

This issue happens due to incorrect permissions on /export/Exalogic_EnterpriseController share on ZFS appliance. Below is procedure to confirm this as the cause.

As we can see from the Job log snippet in above symptoms section, the issue is coming from the OVM side. Below is the error seen in the cacao logs at the time of the issue.

FINE: thr#453:"JM-HandleMessagesThread" updating task - 886, target - com.sun.hss.domain:name=NORM-ACC-f2e344b8-8d7f-4213-9559-546353b0a41b,type=Tenancy, status - RUNNING
Dec 1, 2015 11:38:58 AM com.sun.scn.jobmanager.common.impl.TaskExecutionThread finishTask
INFO: thr#252467:"JM-vaxvelcxp01.150.1.886.2398_com.sun.hss.domain:name=NORM-ACC-f2e344b8-8d7f-4213-9559-546353b0a41b,type=Tenancy" ************* End Of Task - vDC-Create-Server-Template-Association-Task
TaskID - 886
TaskRunID - 2398
Target - com.sun.hss.domain:name=NORM-ACC-f2e344b8-8d7f-4213-9559-546353b0a41b,type=Tenancy
JobIDrunID - vaxvelcxp01.150.1
Status - FAILED
Result - Template is missing due to failure in the uploading task. (117317)
CauseCode - null
Dec 1, 2015 11:38:58 AM com.sun.scn.jobmanager.common.impl.db.StateRestore saveAllMessages
FINE: thr#252467:"JM-vaxvelcxp01.150.1.886.2398_com.sun.hss.domain:name=NORM-ACC-f2e344b8-8d7f-4213-9559-546353b0a41b,type=Tenancy" total messages to be saved - 1 length - 1092 atomID - 126bb41e-784e-42b0-a4d8-03ab91a03a4f

The failure on the EMOC side happens due to 403 errors thrown from the OVM. This can be confirmed by looking at the below 403 error messages logged in AdminServer.log inside (/u01/app/oracle/ovm-manager-3/machine1/base_adf_domain/servers/AdminServer/logs directory) in OVM Manager WebLogic domain.

Server error message: [Errno 14] curl#22 - "The requested URL returned error: 403"
, on server: ff:20:00:08:ff:ff:ff:ff:ff:ff:00:10:e0:5c:11:b2, associated with object: cfgFile_0004fb0000140000f588f4f9db548e14 [Tue Dec 01 11:38:58 EST 2015]
com.oracle.ovm.mgr.api.exception.JobFailureException: OVMAPI_5001E Job: 1,448,991,386,080/Import VM Template-1448991386079/OpsCenter: Import VM Template, failed. The job has a Job Failure Event: 1,448,991,387,693/Job Error: Server Async Command Failed/(12/01/2015 11:38:50:714 AM)
Async operation failed on server: vaxovmxp01.ciena.com.
Object: cfgFile_0004fb0000140000f588f4f9db548e14,
PID: 18485,
Server error message: [Errno 14] curl#22 - "The requested URL returned error: 403"
, on server: ff:20:00:08:ff:ff:ff:ff:ff:ff:00:10:e0:5c:11:b2, associated with object: cfgFile_0004fb0000140000f588f4f9db548e14 [Tue Dec 01 11:38:58 EST 2015]
at com.oracle.ovm.mgr.api.job.JobEngine.checkJobStatus(JobEngine.java:657)
at com.oracle.ovm.mgr.api.job.JobEngine.objectAsyncStart(JobEngine.java:626)
at com.oracle.ovm.mgr.op.virtual.VirtualMachineImport.action(VirtualMachineImport.java:72)
at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1156)
at sun.reflect.GeneratedMethodAccessor552.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:356)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:333)
at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:869)
at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:244)
at com.oracle.ovm.mgr.api.virtual.VirtualMachineCfgFileProxy.executeCurrentJobOperationAction(Unknown Source)
at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:230)
at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:322)
at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1383)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:356)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:333)
at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:106)
at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:92)
at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:752)
at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:467)
at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:525)
at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
at java.lang.Thread.run(Thread.java:662)

Same 403 errors can also be seen in OVM Job log for template upload as follows.

11:38:58,986: Starting operation 'Virtual Machine Cfg File Configure' on object 'cfgFile_0004fb00001400003131d6373b079dc5<16294>'
11:38:58,986: Completed operation 'Virtual Machine Cfg File Configure' with direction ==> LATER
11:38:58,989: Starting operation 'Import Virtual Machine' on object 'cfgFile_0004fb00001400003131d6373b079dc5<16294>'
Job Internal Error (Operation)com.oracle.ovm.mgr.api.exception.JobFailureException: OVMAPI_5001E Job: 1,449,083,895,754/Import VM Template-1449083895753/OpsCenter: Import VM Template, failed. The job has a Job Failure Event: 1,449,083,896,432/Job Error: Server Async Command Failed/(12/02/2015 02:18:16:472 PM)
Async operation failed on server: vaxovmxp01.ciena.com.
Object: cfgFile_0004fb00001400003131d6373b079dc5,
PID: 27662,
Server error message: [Errno 14] curl#22 - "The requested URL returned error: 403"
, on server: ff:20:00:08:ff:ff:ff:ff:ff:ff:00:10:e0:5c:11:b2, associated with object: cfgFile_0004fb00001400003131d6373b079dc5 [Wed Dec 01 11:38:58 EST 2015]
at com.oracle.ovm.mgr.api.job.JobEngine.checkJobStatus(JobEngine.java:657)
at com.oracle.ovm.mgr.api.job.JobEngine.objectAsyncStart(JobEngine.java:626)
at com.oracle.ovm.mgr.op.virtual.VirtualMachineImport.action(VirtualMachineImport.java:72)
at com.oracle.ovm.mgr.api.collectable.ManagedObjectDbImpl.executeCurrentJobOperationAction(ManagedObjectDbImpl.java:1156)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:356)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:333)
at com.oracle.odof.core.storage.Transaction.invokeMethod(Transaction.java:869)
at com.oracle.odof.core.Exchange.invokeMethod(Exchange.java:244)
at com.oracle.ovm.mgr.api.virtual.VirtualMachineCfgFileProxy.executeCurrentJobOperationAction(Unknown Source)
at com.oracle.ovm.mgr.api.job.JobEngine.operationActioner(JobEngine.java:230)
at com.oracle.ovm.mgr.api.job.JobEngine.objectActioner(JobEngine.java:322)
at com.oracle.ovm.mgr.api.job.InternalJobDbImpl.objectCommitter(InternalJobDbImpl.java:1383)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:356)
at com.oracle.odof.core.AbstractVessel.invokeMethod(AbstractVessel.java:333)
at com.oracle.odof.core.BasicWork.invokeMethod(BasicWork.java:106)
at com.oracle.odof.command.InvokeMethodCommand.process(InvokeMethodCommand.java:92)
at com.oracle.odof.core.BasicWork.processCommand(BasicWork.java:81)
at com.oracle.odof.core.TransactionManager.processCommand(TransactionManager.java:752)
at com.oracle.odof.core.WorkflowManager.processCommand(WorkflowManager.java:467)
at com.oracle.odof.core.WorkflowManager.processWork(WorkflowManager.java:525)
at com.oracle.odof.io.AbstractClient.run(AbstractClient.java:42)
at java.lang.Thread.run(Thread.java:662)

403 error is thrown by OVM Manager due to incorrect permissions on /export/Exalogic_EnterpriseController share on ZFS appliance. The permission on /export/Exalogic_EnterpriseController share should be 711. In this case the share permission was set to 700 on the ZFS appliance.

During the template upload process the VM template is temporarily staged in /export/Exalogic_EnterpriseController/ECtmp directory before it is imported. So not having correct permissions on /export/Exalogic_EnterpriseController share will cause template upload to fail because of 403 error returned from OVM.

Solution

Contact Oracle Support by opening Service Request if you run into this issue.

INTERNAL NOTE TO SUPPORT

To resolve this issue the permissions on the /export/Exalogic_EnterpriseController share on ZFS appliance have to be changed to 711.

To change the permissions of the /export/Exalogic_EnterpriseController share drop to shell prompt of the ZFS appliance and run below command

chmod 711 /export/Exalogic_EnterpriseController

References

<NOTE:1512139.1> - Oracle Exalogic Elastic Cloud Known Issues - Virtualization Release
<NOTE:1520829.1> - Enterprise Manager Operations Center (EMOC) Known Issues On Exalogic

Attachments

This solution has no attachment