Incorrect SYSDATE shown when connected via Listener in RAC

Oracle Net Services - Version 10.2.0.5 to 11.2.0.3 [Release 10.2 to 11.2]
Oracle Database - Enterprise Edition - Version 10.2.0.5 to 11.2.0.3 [Release 10.2 to 11.2]
Information in this document applies to any platform.
Affected Database: Oracle


SYMPTOMS

Incorrect SYSDATE shown when connected via listener
 
Restart of database and listener did not help
  
OS defined TZ timezone setting is correct, but still not picked up by listener


CAUSE

- Listener might have a different Envs variable setting for Timezone
  
- Listener is started by LSNRCTL instead of SRVCTL or vice versa
  
- PMON only reads the OS environment variables when the database is started. If TZ is changed after the database is started PMON will not pick up the changed value.

When dynamic database service registration is enabled to a listener, PMON both registers the service name and also the environment variables for the database associated with that service name. This means that connections via the dynamically registered service handler will continue to use the old TZ timezone setting. 
If the listener is restarted and also has a static service registration, connection via the listener will use the TZ set in the listener until PMON first re-registers the service handler again. When the listener has two service registrations, one static and one dynamic, then the dynamic one will be used.

SOLUTION

For 11.2.0.2 and above the TZ entry in $GRID_HOME/crs/install/s_crsconfig_<nodename>_env.txt sets to correct time zone! This enables resources started under the Grid Infrastructure home (listener, databases) using the timezone set in here and so has no requirement anymore to set TZ for DB and listener via setenv. For more details see: How To Change Timezone for 11gR2 Grid Infrastructure ( Document 1209444.1)
 
1) Check the environment variables set for database and listener in your CRS configuration

srvctl getenv database -d <dbname>
srvctl getenv nodeapps -n <nodename>

-- If you are 11.2, then use 'getenv listener' instead of 'getenv nodeapps':
srvctl getenv listener

2) Check the current environment variables used for PMON and Listener 

Refer: How to Check the Environment Variables for an Oracle Process Document 373303.1

3) Compare and if required set the TZ setting with the above

Note: On HP-UX, if TZ is not present, then the value (default: EST5EDT) contained in /etc/default/tz is used. Also see Document 1209444.1 to check timezone current settings on the OS.


4) If required set the correct TZ value for the database via SRVCTL as follows

srvctl setenv database -d <dbname> -t 'TZ=<the TZ you want>'
srvctl setenv nodeapps -n crmnode1 -t 'TZ=<the TZ you want>'
-- If you are 11.2, then use 'setenv listener' instead of 'setenv nodeapps':
srvctl setenv listener -l <listenername> -t 'TZ=<the TZ you want>'

5) Restart the CRS 

6) Start the database instance and listener via SRVCTL.

Restarting the database instances will lead to re-registration of the services and the environment variables to the listener.