Understanding the Enterprise Manager 10g Grid Control Loader System

转载自metalink (Doc ID 550615.1)

In this Document

Purpose

Scope

Details

1. Overview

1. Agent Side Processes

2. OMS Side Processes

3. Loader Related Parameters in Agent and OMS Configuration Files

---

Applies to:

Enterprise Manager Base Platform. - Version 10.2.0.1 to 10.2.0.5 [Release 10.2]
Information in this document applies to any platform.
Checked for relevance on 12-APR-2013

Purpose

The goal of this document is to provide an overview of the EM 10g Grid Control Loader Process.
The Loader Process is a subsystem of the Grid Control architecture responsible for processing the data collected by Management Agents and uploading it to the Management Server (OMS) and Repository.

Scope

The document is aimed at experienced Grid Control Administrators.
You should be familiar with Grid Control concepts, installation, architecture and configuration topics.

Resources to get you started:
EM Grid Control Architecture & Deployment Web Page on OTN

Oracle Enterprise Manager Grid Control Installation and Basic Configuration
10g Release 4 (10.2.0.4)
Part Number E10953-02

Oracle Enterprise Manager Advanced Configuration
10g Release 4 (10.2.0.4.0)
Part Number E10954-01

Details

The Loader System is responsible for processing the data collected by the agent, and uploading it to the OMS and Repository.

Note:
There are 3 forms of data the Agent generates:

• Metadata: Definitions of what it is monitoring and how (A- files)
• State information: Condition of the targets and current state of all the metrics (B-files)
• Metric Data: All datapoints generated by the metrics (C-, D- and E- files)

1. Overview

Below you can find a simplified overview of the Loader processes:

1. The Agent stores the data collected and ready to be uploaded in the /sysman/emd/upload directory

Note.
You can find the location of by doing the following:
Set the $ORACLE_HOME environment variable to point to the agent home directory (eg /orabase/agent10g) and run the following:
$> cd $ORACLE_HOME/bin
$> ./emctl getemhome

may be the same as the oracle home of the agent for standalone installations or it may point to / for Cluster Agent Installs

2. The Agent uploads the data to the OMS according to specific rules and scheduling
3. The OMS stores the data typically under /sysman/recv, parses the XML files, logically validates the data and then uploads it to the Repository. If errors are encountered the culprit file will be moved to the /sysman/recv/errors directory.
4. The OMS sends a response code to the Agent indicating the status of the upload attempt. It may prompt the agent to retry uploading a file if errors were encountered or signal successful processing of the file.

As mentioned above this a simplified version of the process. Let's take a closer look at what happens behind the scenes:

1. Agent Side Processes

It's worth understanding the contents of the /sysman/emd/upload directory.
The files fall into the following categories

• *.xml files - Files that are ready to upload to the OMS.
• *.dat files - Files that the agent writes to prior to concatenating them into .xml files. Each file corresponds to a different table in the EM repository schema.
• *.ext; *.ext.suc - These files are a result of external metrics in the agent (eg. host configuration and chronos metrics) . "External metrics" are metrics where the script. owns responsibility for generating the final .xml file to upload to the repository. The agent passes a filename of the form. Fnnn.ext to the script, and the script. is supposed to rename the file to Fnnn.ext.suc once it finishes writing the file. The agent then renames the Fnnn.ext.suc file to a .xml file and uploads it
• In addition to these files, there are optimizations in the ECM inventory code to avoid sending the same host configuration information over and over again. The inventory metric internally makes a copy of the Fnnn.ext file into the configuration_old file. The next time a collection runs, it compares the new Fnnn.ext content to the old file and sends only a marker file if the content is unchanged.

Below is a detailed overview of the processes that take place at the Agent side:

Overview of the processes used by the Upload Manager:

• Upload Manager: Module of the Agent responsible for handling the Upload process
• As data is collected by other agent components, the Upload Manager serializes writing of intermediary .dat files
• .dat files are merged into .xml files on five priority channels:           

Metadata :	A channel
Severity and Blackouts :	B channel
Upload-on-fetch Data :	C channel
Normal metric data :	D channel
External upload on demand:	E channel

• Data is always loaded in this order: First A, then B, C, D and finally E files.

• If there are files of a lower category waiting to get uploaded, and a new file of a higher category gets generated, the new file from the higher category will be loaded first.Files pending upload are stored in the /sysman/emd/upload directory. The names of the files pending upload are named in a sequential order, per channel.

Examples:
A0000001.xml
A0000002.xml
...
B0000001.xml
B0000002.xml
Etc..

• There are 2 types of upload:
  •  Synchronous Upload:
    •  Metadata (A-files) and State information (B-files) are uploaded in a synchronous way, since this info is vital for the overall monitoring of the targets the agent is working for.
    • The Agent will NOT continue to load another file until this information has been successfully loaded into the repository by the OMS.
  • Asynchronous Upload:
    • All the data generated by the metrics (C-, D- and E files) is pushed to the OMS. The OMS has a special thread (the 'receiver'), which accepts these XML files, but does not upload them.
    • They get stored in the 'receive' directory, where the OMS loaders will pick-up these files and load them into the repository in the order they got received.
    • The Agent can continue with other operations if metric data cannot get uploaded. The Agent can even send a new A- or B- file before picking up the remaining C and D files.

• The Upload Manager maintains statistics on pending xml files; it will disable metric collections based on number of files, aggregate size of files, and percentage free disk space on upload filesystem.

Overview of the Upload Schedule

• The Upload interval is dynamic, based on properties located in the emd.properties file and previous upload status.
• On startup, the Agent will see if any of the metadata has changed. For all changes, A files will be created and uploaded immediately.
• For every state change for a target (including blackout settings), a B file get created and loaded immediately.
• Also, if a metric violates a threshold, a B file gets created and loaded immediately, and the Agent will also force the data upload to happen, to make sure the accompanying data for this metric violations is sent to the OMS.
• For normal data, C and D files are created. 
• C files are loaded immediately after they are created.
• D and E files are normally pushed to the OMS every 30 minutes, if there is no violation reported. This interval is determined by the UploadInterval parameter.

2. OMS Side Processes

Once the data has been uploaded by the Agent, the OMS is responsible for processing it and uploading it to the Repository. Below is a high level overview of the processes that take place at the OMS side:

• Agent sends data to the OMS
• OMS checks for version, key validation, and time zone.
• OMS saves the file to the disk for all non-severity uploads
• If asynchronous upload type, then return success to agent. The file is processed by a background loader thread
• If synchronous upload type, then processes file using SAX parser, blocking the agent for the duration of the process
• Load the data to the Repository
• Move files to errors directory if errors encountered
• Send an appropriate response to agent for both sync/async cases.

3. Loader Related Parameters in Agent and OMS Configuration Files

3.1 Agent Side Properties located in the emd.properties configuration file (under /sysman/config)

Property	Description
REPOSITORY_URL	Pointer to the OMS Receiver URL. This is used by the agent to contact the OMS and upload data.
UploadFileSize	Maximum size (in Kb) for the XML file to be uploaded to the OMS. Default value is 2048 (2Mb)
UploadInterval	Number of minutes to wait before the Agent uploads the XML files to the OMS. Default value is 30 (minutes).
UploadMaxBytesXML	Maximum size (in Mb) the upload manager will support in the upload directory before temporarily being disabled. Collections will stop until more free space is available. Default value is 50 (Mb)
UploadMaxDiskUsedPct	The maximum amount (in percentages) of disk space that can be used on the EMD filesystem before the metric collection, logging and tracing get disabled.
UploadMaxDiskUsedPctFloor	The amount (in percentages) of disk space that can be used on the EMD filesystem before the metric collection, logging and tracing are re-enabled after being disabled previously.
UploadMaxNumberXML	The maximum number of files the upload manager will support in the upload directory before temporarily being disabled Default value is 5000.
UploadTimeout	Maximum amount of time, in seconds, upload will wait for response from repository before timing out. 0 means infinite. Default value is 1800 (seconds) = 30 minutes.
UploadFailBackoffPct	If an upload fails, the agent will backoff by 'UploadFailBackoffPct' percentage more each time for each successive upload failure. For example if the upload is retried at 1 minute, and the next upload attempt fails, the agent will retry again 1 minute and 12 seconds later. Default value is 20 (%).
UploadMaxTime	Timeout for uploading a file. The HealthMonitor will abort the EMD process if the UploadManager does not finish uploading one file within this time frame. The unit is in minutes and the default value is 15 (min).

3.2 OMS Side Properties located in the emoms.properties configuration file (under /sysman/config)

Property	Description
em.loader.maxMetadataThreads	Maximum number of Metadata loader threads the OMS will spawn. Default is 10
em.loader.threadPoolSize	Number of threads used for loading data files into the repository. Possible values are in the range of 1 to 10. Default value is 1
ReceiveDir	Directory to place the XML files retrieved from the Agents. Default location is $ORACLE_HOME/sysman/recv.
em.loader.maxDirectLoadFileSz	This property controls the maximum size in bytes that the loader will attempt to load through on direct loads. The default is 5MB (5242880). On a 'big' machine, you might need to increase this. Only increase this parameter when instructed to do so by Oracle Support

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/543979/viewspace-774260/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/543979/viewspace-774260/