Troubleshoot Timezone Issues in Enterprise Manager Grid Agent Setup-330737.1

最新推荐文章于 2021-04-07 16:07:34 发布

cuilian2013

最新推荐文章于 2021-04-07 16:07:34 发布

阅读量231

点赞数

Grid Agent Configuration: How to Troubleshoot Timezone Issues in Enterprise Manager Grid Agent Setup? [ID 330737.1]

In this Document

Purpose

Troubleshooting Steps

Overview

Quick Checklist

1. Incorrect / Invalid TZ Configuration or Manual Changes

2. Validation Failures at the Repository Database-side

3. DST

References

Applies to:

Enterprise Manager Base Platform. - Version 10.1.0.4 to 11.1.0.1 [Release 10.1 to 11.1]
Information in this document applies to any platform.
***Checked for relevance on 20-Feb-2013***

Purpose

This note provides steps for troubleshooting issues related to Timezone settings in a Grid Agent installation. An incorrect Timezone configuration in the Agent setup will prevent the Agent from starting up.

The Timezone setting discovered by the Agent is utilized for multiple operations:
- The Agent collects metric data using timestamps based on the TZ setting of the Agent/target. The Metric data is uploaded to the repository 'as is' by the Loader into the relevant metric tables.
- The Grid Console UI uses the TZ for each target to convert the metric data based on the TZ of the current page internally, etc.

Hence, if the TZ is wrongly configured at the Agent, it can also affect the timestamp related data seen in the Grid Console, for the discovered targets.

Additional References

Document 548928.1: Enterprise Manager Grid Control Agent 10g, Process Control (Start, Stop & Status) Troubleshooting Guide
Document 359524.1: How To Find Agents with time-skew problems

Troubleshooting Steps

Overview

The Agent uses the logic coded in the /bin/emctl file to identify the timezone setting at the Operating system level and stores the value in the <AGENT_HOME>/sysman/config/emd.properties file via the parameter: agentTZRegion. For example:

agentTZRegion=Asia/Calcutta

This parameter is defined during the first-time startup of the Agent after the installation. During the subsequent startups, this parameter value is validated in two phases during the startup of the Agent:

During the ‘Launch’ phase, it is verified whether the OS and Agent timezone are supported/synchronized/valid. This validation is performed at the Agent side itself and compares the TZ set in the OS with the value specified in agentTZregion parameter of the emd.properties file.

We compare:
- the offset calculated for the agentTZregion based on information in /oracore/zoneinfo/timezlrg.dat
  with
- the offset calculated for the OS TZ based on the OS’s understanding of the current OS Timezone setting.

If the OS does not understand the TZ set in the environment, the offset for the OS is set to 0 (effectively GMT).

If the Timezone is wrongly configured, the Agent startup can fail with errors such as:

Timezone mismatch: The agentTZRegion value (US/Central) in
/agent10g/sysman/config/emd.properties does not match the current
environment TZ setting(US/Pacific).
The agent cannot run with this mismatch.

OR

Starting agent ...... failed.
The agentTZRegion value in /path/to/agent10g/sysman/config/emd.properties is not in agreement with what agent thinks it should be.Please verify your environment to make sure that TZ setting has not changed since the last start of the agent.
If you modified the timezone setting in the environment, please stop the agent and exectute 'emctl resetTZ agent' and also execute the script. mgmt_target.set_agent_tzrgn(, ) to get the value propagated to repository.

During the ‘Initialization’ phase, it is verified whether the Agent timezone is supported by the Repository Database. This validation is performed at the Repository Database side using the details sent by the Agent during the initial heartbeat.

When the Agent uploads data the first time after installation, the TZ is validated to verify whether the Repository Database supports the TZ value. After validation, the TZ values are stored for each of the discovered targets in the TIMEZONE_REGION column of the MGMT_TARGETS table. During the subsequent Agent startups, The TIMEZONE_REGION for the oracle_emd target in MGMT_TARGETS is compared with the TZ string passed by the Agent in its heartbeat.

If the validation fails at the Repository Database side, the /sysman/log/emagent.nohup will have errors such as:

(pid=9970): starting emagent version 10.2.0.5.0
OMS does not understand the timezone region of the agent. Either start the OMS using the extended list of time zones supported by the database or pick a value of time zone from /rdbms/app/OracleHomes/agent10g/sysman/admin/nsupportedtzs.lst , update the property 'agentTZRegion' in the file /rdbms/app/OracleHomes/agent10g/sysman/config/emd.properties and
restart the agent. A value which is around an offset of +01:00 from GMT should be picked.
(pid=9970): emagent now exiting abnormally - initialization failure. Consult '.trc' and '.log' files.

OR from 10.2.0.5 onwards:

(pid=20785): starting emagent version 10.2.0.5.0
OMS decided to shutdown the agent because of the following reason sent from OMS: AGENT_TZ_MISMATCH(pid=20785): emagent now exiting abnormally - initialization failure. Consult '.trc' and '.log' files.
----- Wed Mar 24 09:30:25 2010::Checking status of EMAgent : 20785 -----
----- Wed Mar 24 09:30:25 2010::EMAgent exited at Wed Mar 24 09:30:25 2010 with return value 55. -----

Log/Trace files to check

/sysman/log/emagent.nohup
/sysman/log/emagent.trc

Useful Configuration Files:

/sysman/config/emd.properties: The Timezone discovered by the Agent is stored in the this file via the parameter: agentTZRegion.
/sysman/admin/nsupportedtzs.lst: Contains the list of Timezones supported by the Agent. This is same as the list of timezones supported by 9i Oracle database.
/sysman/admin/supportedtzs.lst: Contains the extended list of Timezones supported by the Agent. This is same as the list of timezones supported by 10g+ Oracle database.
/sysman/admin/ tzmappings.lst: Maps unsupported/OS specific TZs to those supported by Java.
/sysman/admin/ossupportedtzs.lst (on AIX OS only)

Useful Environment Variables:

TZ: defines the value of the Timezone setting at the Operating System session level.
ORA_TZFILE: defines the location of the timezone file which is used for calculating the offset of the agentTZRegion parameter value.
Default value is : /oracore/zoneinfo/timezlrg.dat

The troubleshooting steps can be grouped into:

1. Incorrect / Invalid Configuration or Manual Changes (including Operating System Specific issues).
2. Validation Failures at the Repository Database-side.
3. DST, including Operating System Specific issues.

Quick Checklist

1) Check the TZ settings in the Operating System environment, using one or more of the following methods:

As calculated by the Agent:

cd /bin
$ emctl config agent getTZ
At the Session level:

$ echo $TZ

OR

$ env | grep TZ
To get the Operating System TZ settings via shell script.

- For Linux:
  
  $ export ORACLE_HOME=
  $ for i in `/bin/cat $ORACLE_HOME/sysman/admin/supportedtzs.lst` ; do
  if [ -f /usr/share/zoneinfo/${i} ] ; then
  if [ "`/usr/bin/diff /etc/localtime /usr/share/zoneinfo/${i}`" = "" ] ; then
  echo "${i}"
  fi
  fi
  done
  Asia/Calcutta
- For Solaris:
  
  $ grep -v "^#" /etc/TIMEZONE | grep TZ
  TZ=GMT
- For HP-UX:
  
  $ grep -v "^#" /etc/TIMEZONE | grep TZ
  TZ=GMT
  export TZ
- For AIX:
  
  $ grep -v "^#" /etc/environment | grep TZ
  TZ=PST8PDT
To get the Operating System timezone settings via JAVA:
- Create a file called TestTZ.java, which contains the below code:
  
  import java.util.TimeZone;
  
  public class TestTZ
  {
  public TestTZ()
  {
  }
  public void printTimeZone()
  {
  System.out.println((TimeZone.getDefault()).getID());
  }
  /**
  * @param args
  **/
  public static void main(String[] args)
  {
  TestTZ testTZ = new TestTZ();
  testTZ.printTimeZone();
  }
  }
- Compile and execute the java program:
  
  $ export JAVA_HOME=/jdk
  $ export PATH=$JAVA_HOME/bin:$PATH
  $ javac TestTZ.java
  $ java TestTZ
  GMT

2. Collect the Timezone settings being used by the Agent:

$ grep agentTZ /sysman/config/emd.properties

The TZ settings being used by the Agent and that set at the Operating System should match.

3) Manually execute the validation routine:

$ export ORACLE_HOME=
$ export EMDROOT=$ORACLE_HOME
$ export EMSTATE=$ORACLE_HOME
$ emdctl validateTZ agent

This may fail, but the error message is what we need.
-> For a correct execution, there should be no output
-> For a 'bad' validation, the output is going to look something like this:

$ emdctl validateTZ agent US/Samoa
tzOffset for US/Samoa is -660(min), but agent is runnning with tzOffset -360(min)
trying again after waiting for 1 sec to account for daylight transition
tzOffset for US/Samoa is -660(min), but agent is runnning with tzOffset -360(min)

Note:
- For iAS 9.0.4 and 10.1.2 Console agents, this routine does not exist

4) Force the Agent to re-calculate the TZ settings as per the OS settings:

- Stop the Agent:

cd /bin
emctl stop agent

- Set the necessary environment variables for the Agent and execute the below command and note down the output returned

$ export ORACLE_HOME=
$ export PATH=$ORACLE_HOME/bin:$PATH
$ cd $ORACLE_HOME/bin
$ emctl resetTZ agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
Updating /home/em/oracle/gc102/agent10g/sysman/config/emd.properties...
Successfully updated /home/em/oracle/gc102/agent10g/sysman/config/emd.properties.
Login as the em repository user and run the script.:
exec mgmt_target.set_agent_tzrgn('agentmachine.domain:3872','Asia/Calcutta')
and commit the changes
This can be done for example by logging into sqlplus and doing
SQL> exec mgmt_target.set_agent_tzrgn('agentmachine.domain:3872','Asia/Calcutta')
SQL> commit

DO NOT start the Agent yet!

- Login to the Repository database as the SYSMAN user and execute the SQL statements shown by the above command output:

SQL> exec mgmt_target.set_agent_tzrgn('','');
SQL> commit;

For example:
SQL> exec mgmt_target.set_agent_tzrgn('agentmachine.domain:3872','Asia/Calcutta')
SQL> commit

Note: If this is a new Agent installation, then it is not necessary to run the above SQL statements in the Repository Database. As the Agent has not yet uploaded successfully, the Agent details are not present in the Repository tables. Refer to Note 388280.1 Grid Agent Configuration: "ORA-20233: Invalid Agent name" When Running mgmt_target.set_agent_tzrgn Procedure in Repository DB

- After the Repository side routine has completed successfully, start the agent, and make sure it's uploading new data:

$ emctl start agent
$ emctl upload
$ emctl status agent

1. Incorrect / Invalid TZ Configuration or Manual Changes

As the Agent depends on the timezone setting at the Operating System level, it is imperative that the Timezone at the OS level is set correctly according to match the geographic location in which the Agent machine is located.
Refer to the Operating System documentation to identify the correct format in which the TZ should be set for that OS.

The Timezone set at the OS level should be supported by the Agent. The list of timezones recognized by the Agent is available in the
- /sysman/admin/nsupportedtzs.lst: Contains the list of Timezones supported by the EM agent. This is same as the list of timezones supported by 9i Oracle database.
- /sysman/admin/supportedtzs.lst: Contains the extended list of Timezones supported by the EM agent. This is same as the list of timezones supported by 10g+ Oracle database.

It is recommended that the timezone value which is available in both of the above files is chosen, in order to ensure that the timezone is recognized by the Repository Database, irrespective of its version. It is also essential to verify whether the ORA_TZFILE environment variable has not been set when the Agent is being started. If this is set, the value should be pointed to the /oracore/zoneinfo/timezlrg.dat and not to any other file in the other Oracle home's of the machine.

Some known issues:

Document 409121.1: Grid Control Agent Startup Fails if ORA_TZFILE Environment Variable is Set to Incorrect Timezone File
Document 560529.1: Grid Control Agent Startup Fails after the Machine Timezone is Changed
Document 825933.1: Grid Control Agent Startup Fails on AIX Server due to Invalid Timezone Setting at the OS

2. Validation Failures at the Repository Database-side

Agent should have picked up a valid timezone which is recognized by the Repository Database. This list of timezones supported by the Repository varies according to the version or the Database and the value of the ORA_TZFILE environment variable set at the time of Database startup.
At the time of the Database startup, this parameter can be set to the
large timezone file: $ORACLE_HOME/oracore/zoneinfo/timezlrg.dat OR
basic timezone file: $ORACLE_HOME/oracore/zoneinfo/timezone.dat.

The performance difference between both files is not very high unless the application uses extensive amount of timezone information. In most cases the benefit of having more timezones available outweights any (small) performance difference, hence the default value (= ORA_TZFILE not set) is then the best choice.

Some key points to note:

Until 9.2.0.5 patchset version, the 'basic' (small) time zone file was the default time zone file that is loaded by the Database. From 9.2.0.5 and above, the large file gets loaded by default.
To switch between the time zone files, and overwrite the default, complete the following tasks:

1. Shut down the database.
2. Set the ORA_TZFILE environment variable to the full path name of the time zone file you want to use. On Unix this needs to be a environment variable, on Windows this needs to be set in the relevant HOME under HKEY_LOCAL_MACHINE/Software/Oracle in the registry.
3. Restart the database.
Be aware that you can neither create nor alter these time zone definition files, they are binary files that only Oracle can build and provide.
In Oracle 11.2 the ORA_TZFILE usage and the naming of the files changed, this is documented in the docset under "Choosing a Time Zone File" in Oracle� Database Globalization Support Guide 11g Release 2 (11.2), Chapter 4 .
Note that the actual result in 11.2 , when compared to 11.1 or below, will not be very different when NOT defining ORA_TZFILE, Oracle RDBMS still uses the large file by default and will use the highest DST version available in $ORACLE_HOME/oracore/zoneinfo when creating the database.
To identify the timezone file being used by the Database:

1. Checking the value of the ORA_TZFILE environment variable.

2. Login to the Database as a DBA user and execute:

SQL> SELECT COUNT(*) FROM v$timezone_names;

For the default 'large' time zone file, the output would be:

COUNT(*)
----------
1393 (1250 in Oracle9)

For the 'small' time zone file, the output would be:

COUNT(*)
---------
636 (616 in Oracle9)

For more details, refer to Document 340512.1: Timestamps & time zones - Frequently Asked Questions

Some known issues:

Document 332131.1: Grid Control Agent Startup Fails When the Repository Database Does Not Support the Timezone of the Agent
Document 865332.1: Grid Control Agent Startup Fails in 10.2.0.5 if Timezone is set to Europe/Jersey

3. DST

DST changes for the geographical region / timezone where the Agent is installed can affect the Agent startup and operations under the following conditions:

- Timezone is not set in the correct format, for DST.
- There is a DST change for the timezone being used but the Operating System is not patched to recognize these changes.
- There is a DST change for the timezone being used and the Operating System has been patched to recognize these changes but the Agent's timezone files do not recognize the changes.

For evaluating the DST impact and solutions, refer to the below documents:

Document 739116.1: DST: Is your Grid Control ready for the next Daylight Saving Time - second half 2010 ?
Document 409225.1: Impact of DST changes on Grid Control Release 2 - Preparation and Patching guidelines.
Document 471451.1:DST 2007/2008/2009/2010 Oracore files Version for Grid Control Components.

References

BUG:4466546 - EMCTL RESETTZ AGENT FAILS WHEN TZ SET TO CET

NOTE:409121.1 - Grid Control Agent Startup Fails if ORA_TZFILE Environment Variable is Set to Incorrect Timezone File