SBL-OSD-00220: Internal: pthread_mutex_trylock () failed with error.

Applies to: Siebel System Software - Version: 7.5.2.100 [15252] to 8.1.1.1 SIA [21211] - Release: V7 to V8
Generic UNIX
Generic Linux
Product Release: V7 (Enterprise)
Version: 7.5.2.100 [15252]
Database: IBM DB2 8.1 FixPack 6b
Application Server OS: IBM AIX 5L 5.1
Database Server OS: IBM AIX 5L 5.1

This document was previously published as Siebel SR 38-3078571521.

Symptoms

SBL-GEN-00255, SBL-OSD-00220, SBL-OSD-02006, SBL-OSD-02011, SBL-OSD-00204, SBL-ADM-02049, SBL-GEN-00001, OSDmprotect

The Siebel Server was running fine until a few weeks ago. Now we are unable to start the Siebel Server. The Siebel Gateway Name Server is running fine.

When we start the Siebel Server by using start_server all command, the main thread starts (siebsrvr) but when it spawns the child threads (siebmtsh), they are not staying alive.

No Siebel multithreaded server processes are started after running start_server all and no Application Object Manager log files are being generated.

The SiebSrvr.log under siebsrvr/log directory may contain the following error:

    SBL-OSD-00220: Internal: pthread_mutex_trylock () failed with error. (22)

In some situations the following error may also be seen:

   OSDmprotect: mprotect(0xf7400000, 0, 0x00000001) failed with error 22 

In addition to SBL-OSD-00220 error messages in SiebSrvr.log log file, error messages like the following are being logged in the Enterprise Server log file (<enterprise_name>.<server_name>.log) under $SIEBEL_ROOT/enterprises/<enterprise_name>/<server_name>/log directory:

<NoCompName> 64527 SBL-GEN-00002 Process exited with error - Error code SBL-GEN-00002

Changes

 

Cause

A) This situation has been reported when the Unix machine has been shutdown without having run the stop_server script. The stop_server script should remove the osdf.<enterprise_name>.<server_name> file that resides in the $SIEBEL_ROOT/siebsrvr/sys directory.

B) This situation has also been reported when the svc.siebsrvr.<enterprise_name>:<server_name> was duplicated under $SIEBEL_ROOT/siebsrvr/sys directory.

When you specify the ALL argument, the start_server script searches for all files under $SIEBEL_ROOT/siebsrvr/sys in the format svc.siebsrvr.* and parses it to identify the Enterprise name and the Server name.
The first string after svc.siebsrvr. is the Enterprise name and the second one is the Server name.
The start_server script skips all filenames ended with the .bak extension, because they are considered backup files.
Then it starts the service and creates the OSDF file based on these parsed names.

Customer has three svc* files on his system:

- $SIEBEL_ROOT/siebsrvr/sys/svc.siebsrvr.<enterprise_name>:<server_name>
- $SIEBEL_ROOT/siebsrvr/sys/svc.siebsrvr.<enterprise_name>:<server_name>.bak
- $SIEBEL_ROOT/siebsrvr/sys/svc.siebsrvr.<enterprise_name>:<server_name>.OLD1

The second file is considered as a backup file by start_server script.
However, the other two files are used by start_server to start up the Siebel Server, and both point to the same Enterprise and Server names.
Therefore, the start_server script is starting the same Siebel Server twice.

The problem is that, when the second instance is started, the OSDF file is overwritten, and the information used by Siebel about the memory structures used by the first instance is lost. The second instance will conflict with the first one, and the processes end up failing.

Solution

In order to fix this behavior and guarantee you have a clean environment, please do the following:

1. Shutdown the Siebel Server and run the following commands to clean all structures that may remain:

    % stop_server ALL
    % reset_server -e <enterprise_name> <server_name>
    % siebclean -f $SIEBEL_ROOT/siebsrvr/admin/<enterprise_name>.<server_name>.shm -q
    % cleansync -f $SIEBEL_ROOT/siebsrvr/sys/osdf.<enterprise_name>.<server_name> -d
    % siebctl -r $SIEBEL_ROOT -S siebsrvr -i <enterprise_name>:<server_name> -k -q
    % mwcleanup -silent 2>/dev/null

    % stop_server ALL
    % reset_server -e SBA_81 appcnbj068
    % siebclean -f $SIEBEL_ROOT/siebsrvr/admin/SBA_81.appcnbj068.shm -q
    % cleansync -f $SIEBEL_ROOT/siebsrvr/sys/osdf.SBA_81.appcnbj068 -d
    % siebctl -r $SIEBEL_ROOT -S siebsrvr -i SBA_81:appcnbj068 -k -q
    % mwcleanup -silent 2>/dev/null

2. If using AIX, log into UNIX as root and run the following command:

    % /usr/sbin/slibclean

3. Ensure no Siebel processes are running and check that there are no semaphores or shared memory segments allocated for the UNIX user who owns the Siebel Server installation:

    % ps -ef | grep <username>
    % ipcs -m

4. If there are any processes, shared memory segments or semaphores stuck in memory, they can be killed by using the following commands, respectively:

    % kill -9 <process_ID>
    % ipcrm -m <shared_memory_segment_ID>
    % ipcrm -s <semaphore_ID>

5. Remove the following files if they still exist:

    $SIEBEL_ROOT/siebsrvr/sys/osdf.<enterprise_name>.<server_name>
    $SIEBEL_ROOT/siebsrvr/admin/<enterprise_name>.<server_name>.shm

6. Remove all svc* files or move all of them out from $SIEBEL_ROOT/siebsrvr/sys directory.

7. Recreate the correct svc* file:

   

For Siebel 8.0.x and earlier please use the comand: 

% siebctl -S siebsrvr -i "<enterprise_name>:<server_name>" -a -g "-g <gateway_name> -e <enterprise_name> -s <server_name>"

For Siebel 8.1.x and later please use the comand: 

% siebctl -S siebsrvr -i "<enterprise_name>:<server_name>" -a -g "-g <gateway_name> -e <enterprise_name> -s <server_name> -u <sadmin_user>" -e <sadmin_password>

siebctl -S siebsrvr -i "SBA_81:appcnbj068" -a -g "-g appcnbj067 -e SBA_81 -s appcnbj068 -u SADMIN" -e SADMIN

please note that the siebel server service needs user and password for gateway authentication starting with Siebel version 8.1

8. Restart the Siebel Server:

    % start_server all

Customers should note that if there are still any processes, semaphores or shared memory segments stuck in memory, the machine will need to be rebooted before recreating the svc* file (between steps 6 and 7).

Customers should also note that recreating the svc* file (steps 6 and 7) is not necessary in all situations, and in most cases step 1 only is sufficient to guarantee a clean startup.

KEYWORDS: start_server all, failure to start Siebel on Unix server,  OSDmprotect, err=1300220 sys=22 sys=0 SBL-GEN-00001 SBL-GEN-00002 SBL-GEN-00003 SBL-GEN-00004 SBL-GEN-00005 SBL-GEN-00006 SBL-GEN-00007 pthread_mutex_trylock AIX slibclean ipcrm