Title
Getting ORA-06512 when 10.2.0.1 Agent uploads to 10.2.0.3 OMS

Date
2007-02-18

Summary
1. Installed a 10.2.0.1 Agent on target server TSERVER.thisisahmed.com.

2. OMS is running 10.2.0.3 listening on HTTPS 443 (but not listening on HTTP).

3. Manually removed any legacy TSERVER.thisisahmed.com host and oracle_emd entries from the OMS Console.

4. Securing the Agent worked.

5. Starting the Agent worked.

6. Uploading XML files failed: 
    c:\oracle\agent10g>emctl status agent
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2006 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 10.2.0.1.0
OMS Version       : 10.2.0.3.0
Protocol Version  : 10.2.0.0.0
Agent Home        : C:\oracle\agent10g
Agent binaries    : C:\oracle\agent10g
Agent Process ID  : 3184
Agent URL         : https://TSERVER.thisisahmed.com:3872/emd/main/
Repository URL    : https://oemserver.thisisahmed.com:443/em/upload
Started at        : 2007-02-18 23:10:36
Started by user   : SYSTEM
Last Reload       : 2007-02-18 23:10:36
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far :     0.00
Number of XML files pending upload           :        3
Size of XML files pending upload(MB)         :     0.38
Available disk space on upload filesystem    :    18.54%
Last successful heartbeat to OMS             : 2007-02-18 23:11:06
---------------------------------------------------------------
Agent is Running and Ready

c:\oracle\agent10g>emctl upload
Oracle Enterprise Manager 10g Release 10.2.0.1.0.
Copyright (c) 1996, 2006 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
upload failed with ret=-10
7. Tried running the following, which did not work: 
    begin
  mgmt_admin.cleanup_agent('TSERVER.thisisahmed.com:3872');
  commit;
end;

    Apparently, that works if an entry already exists in MGMT_TARGETS.
8. Since I manually deleted all references to TSERVER.thisisahmed.com from OMS, there were no entries in MGMT_TARGETS: 
    SQL> SELECT target_name
     FROM   sysman.mgmt_targets
     WHERE  target_type = 'oracle_emd';
9. There is no DELETE_COMPLETE_TIME for the Agent I just installed (which I manually deleted from the OMS Console): 
    SQL> SELECT target_name, target_type, delete_complete_time
     FROM   mgmt_targets_delete;

TARGET_NAME    TARGET_TYPE DELETE_COMPLETE_TIME
-------------- ----------- --------------------
TSERVER.thisisahmed.com        host        18-FEB-07
TSERVER.thisisahmed.com:3872   oracle_emd
10. The log file on the Agent server $ORACLE_HOME/sysman/log/emagent.trc shows: 
    2007-02-18 23:12:26 Thread-3648 WARN  TargetManager: Regenerating all Metadata
2007-02-18 23:12:31 Thread-3648 DEBUG TargetManager: In getTypeAndNameForHostTarget
2007-02-18 23:12:31 Thread-3648 DEBUG TargetManager: getTypeAndNameForHostTarget found host, TSERVER.thisisahmed.com
2007-02-18 23:12:31 Thread-3648 DEBUG TargetManager: In getTypeAndNameForHostTarget
2007-02-18 23:12:31 Thread-3648 DEBUG TargetManager: getTypeAndNameForHostTarget found host, TSERVER.thisisahmed.com
2007-02-18 23:12:31 Thread-3648 WARN  collector: Regenerating all DefaultColls
2007-02-18 23:12:32 Thread-2056 ERROR upload: Failed to upload within 60 seconds
2007-02-18 23:12:38 Thread-3648 ERROR upload: Failed to upload file A0000001.xml, ret = -6
2007-02-18 23:12:38 Thread-3648 ERROR upload: 3 Failures in a row for A0000001.xml, we give up
2007-02-18 23:12:38 Thread-3648 ERROR upload: Error in uploadXMLFiles.  Trying again in 72.00 seconds or earlier.
2007-02-18 23:12:54 Thread-2828 WARN  upload: FxferSend: received fatal error in header from repository: https://oemtestapp1.thisisahmed.com:443/em/upload
FATAL_ERROR::500|ORA-20618: The specified agent is in the process of being
deleted from the repository, wait for deletion to complete before restarting the agent.
(agent name = TSERVER.thisisahmed.com:3872)(agent guid = 0324A760A4DD8AA7BA1E44F4CCEE4B75)
ORA-06512: at "SYSMAN.TARGETS_INSERT_TRIGGER", line 30
ORA-04088: error during execution of trigger 'SYSMAN.TARGETS_INSERT_TRIGGER'
2007-02-18 23:12:54 Thread-2828 ERROR upload: number of fatal error exceeds the limit 3
2007-02-18 23:12:54 Thread-2828 ERROR upload: agent will shutdown now
2007-02-18 23:12:54 Thread-2828 ERROR : Signalled to Exit with status 55
2007-02-18 23:12:54 Thread-2956 ERROR main: EMAgent abnormal terminating

Details
1. Run the package below.

If it appears to hang (i.e., no response within 60 seconds), just CTRL-C out of it: 
SQL> EXEC mgmt_admin.delete_target_internal('TSERVER.thisisahmed.com:3872', 'oracle_emd');

BEGIN
mgmt_admin.delete_target_internal('TSERVER.thisisahmed.com:3872',
'oracle_emd'); END;

*
ERROR at line 1:
ORA-01013: user requested cancel of current operation

2. When querying the MGMT_TARGETS_DELETE table, the DELETE_COMPLETE_TIME column is now updated. 
SQL> SELECT target_name, target_type, delete_complete_time
     FROM   mgmt_targets_delete;

TARGET_NAME                   TARGET_TYPE DELETE_COMPLETE_TIME
----------------------------- ----------- --------------------
TSERVER.thisisahmed.com       host        18-FEB-07
TSERVER.thisisahmed.com:3872  oracle_emd  18-FEB-07

3. After clearing the Agent state, securing the Agent, and starting up the Agent, the XML uploads will work successfully.

Applicable Versions
Oracle Enterprise Manager 10g Grid Control (10.2.0.3)
Oracle Enterprise Manager 10g Agent (10.2.0.1)
Ahmed Aboulnaga
.com .com