DB2 "No start database manager command was issued. SQLSTATE=57019" 故障解决

故障现象：

Linux下su - itimldap

db2start启动db2时报错： No start database manager command was issued.  SQLSTATE=57019

db2stop force停止db2时报错： No start database manager command was issued.  SQLSTATE=57019

显示详细报错信息：

tail -f  /home/itimldap/sqllib/db2dump/db2diag.log 

2012-10-25-21.27.55.952334+480 I140604G409        LEVEL: Event
PID     : 9469                 TID  : 2980669120  PROC : db2start
INSTANCE: itimldap             NODE : 000
FUNCTION: DB2 UDB, base sys utilities, sqleIssueStartStop, probe:1100
DATA #1 : String, 52 bytes
/home/itimldap/sqllib/adm/db2rstar db2profile SN 0 0
DATA #2 : Hexdump, 4 bytes
0xBF9FCDF8 : 1000 0000                                  ....


2012-10-25-21.27.55.958078+480 E141014G398        LEVEL: Error (OS)
PID     : 9472                 TID  : 2980669120  PROC : db2start
INSTANCE: itimldap             NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloRemoteShell, probe:50
CALLED  : OS, -, execvp                           OSERR: ENOENT (2)
MESSAGE : Error invoking remote shell program.
DATA #1 : String, 12 bytes
/usr/bin/rsh


2012-10-25-21.27.56.962100+480 E141413G600        LEVEL: Error
PID     : 9469                 TID  : 2980669120  PROC : db2start
INSTANCE: itimldap             NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloPdbInitializeRemoteCommand, probe:110
MESSAGE : ZRC=0x810F0012=-2129723374=SQLO_COMM_ERR "Communication error"
DATA #1 : String, 204 bytes
The remote shell program terminated prematurely.  The most likely causes are either that the DB2RSHCMD registry variable is set to an invalid setting, or the remote command program failed to authenticate.
DATA #2 : String, 12 bytes
/usr/bin/rsh


2012-10-25-21.27.56.963425+480 E142014G485        LEVEL: Error
PID     : 9469                 TID  : 2980669120  PROC : db2start
INSTANCE: itimldap             NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloPdbInitializeRemoteCommand, probe:200
MESSAGE : ZRC=0x810F0012=-2129723374=SQLO_COMM_ERR "Communication error"
DATA #1 : String, 9 bytes
localhost
DATA #2 : String, 15 bytes
linux-7mr7.site
DATA #3 : String, 51 bytes
No diagnostics available from remote shell program.


2012-10-25-21.27.56.966089+480 I142500G279        LEVEL: Event
PID     : 9469                 TID  : 2980669120  PROC : db2start
INSTANCE: itimldap             NODE : 000
FUNCTION: DB2 UDB, base sys utilities, sqleIssueStartStop, probe:80
DATA #1 : signed integer, 4 bytes
-6048

查看IBM官方文档，故障原因及解决办法如下：

Cause

The issue may be caused by one of the following reason:

• Logical MPP instance doesn't have .rhosts file under instance HOME directory.
• Logical MPP has .rhosts file with incorrect entries
• A password change may be required for the instance user
• Host name not a fully qualified name in /etc/hosts file.

Diagnosing the problem

Verify the following:

- Check if .rhosts file exist under instance HOME directory with correct entries. An alternative to using a .rhosts file is to use /etc/hosts file. The /etc/hosts file would contain the exact same entries as the .rhosts file, but must be created on each
computer.

- the node has the proper authorization defined in the .rhosts or the /etc/hosts files.

- the application is not using more than (500 + (1995 - 2 * total_number_of_nodes)) file descriptors at the same time.

- the Enterprise Server Edition environment variables are defined in the profile file.

- the profile file is written in the Korn Shell script format.

- all the host names defined in the db2nodes.cfg file in the sqllib directory are defined on the network and are running.

- the DB2FCMCOMM registry variable is set correctly.

- all the nodes defined in the sqllib/db2nodes.cfg file including the new node you attempted to add are correct.

- the server is in the same domain as the IBM® data server client

Resolving the problem

There are three possible resolutions to the problem:

1) Create a .rhosts or /etc/hosts file with correct entries .

2) Login and change the password of the instance user .

The change password is required on servers which forces the user to set the password during first login to reinforce security. This is usually required on AIX servers .

Following are the detailed steps for resolving the issue 

a) Create a .rhosts file under instance HOME directory in the following format

<hostname> <instance_Name>

Where <instance_name> is name of instance for which logical MPP is configured and hostname should be the name of DB2 server(Issue the hostname command to get the hostname)

Example: 
machine1.in.ibm.com db2inst1 

b) Login to the instance user and set a new password.

Example: 
Step 1 : su - <instance_Name>

Step 2 : passwd

Changing password for "db2inst1"
db2inst1's Old password:
db2inst1's New password:
Enter the new password again: 

After doing the suggested resolution in point (a) and (b). Please run the following command as an instance user to verify logical MPP has been configured correctly

Example: 

db2inst1 is the instance configured with logical MPP.

#su - db2inst1

db2inst1@vminstlnx64test16:~/sqllib> db2_all date

Wed Aug 24 06:52:21 EDT 2011
vminstlnx64test16: date completed ok

Wed Aug 24 06:52:22 EDT 2011
vminstlnx64test16: date completed ok

The above output conveys that logical MPP is configured correctly and db2start should work fine now . 

If the command "db2_all date" gives permission denied as shown below ,it means there is some configuration issues

db2inst1@vminstlnx64test16:~> db2_all date

Permission denied.

Permission denied.

db2inst5@vminstlnx64test16:~>

In case you receive a permission denied error , it means that the rsh to the host is not configured properly. This can be due to many system related issues. I am listing few of them below .

i. rsh service is not running on the system.

ii. Hostname specified in db2nodes.cfg or .rhosts is not correct. 

3) if you are using /etc/hosts file, the /etc/hosts file should use the fully-qualified name. If the fully-qualified name is not used in the db2nodes.cfg file and in the/etc/hosts file, you might even receive error message SQL30082N RC=3.

Such as myserver.spifnet.ibm.com instead of myserver

For Example : change myserver to myserver.spifnet.ibm.com.