expdp、rman 等在cifs文件系统上遇到file lock

环境是linux oracle 11.2.0.4 ，使用expdp备份到samba，linux连接samba使用的autofs，所以映射过来的目录是cifs格式。

[root@rac1 oradb]# df -Th
Filesystem           Type   Size  Used Avail Use% Mounted on
/dev/sda7            ext4   201G  109G   82G  58% /
tmpfs                tmpfs  127G  508M  126G   1% /dev/shm
/dev/sda1            ext4   477M   40M  412M   9% /boot
/dev/sda6            ext4   9.8G  1.4G  7.9G  16% /osw
/dev/sda2            ext4    99G   57G   37G  61% /u01
/dev/sda3            ext4    99G   18G   77G  19% /u02
//192.168.1.34/oradb cifs    49T   16T   33T  32% /database_backup/oradb

原来的配置是

[root@rac1 ~]# cat /etc/auto.cifs 
oradb   -fstype=cifs,rw,noperm,user=xxxxx,pass=xxxxx       ://192.168.1.23/oradb
ldpfile -fstype=cifs,rw,noperm,user=xxxxx,pass=xxxxx      ://192.168.1.23/SMSHistoryFile

原来的expdp备份正常，但是最近出现了：

ORA-39000: bad dump file specification
ORA-31641: unable to create dump file "/database_backup/oradb/expdpbak/orcl/orcl_20200412_01.expdp"
ORA-27086: unable to lock file - already in use
Linux-x86_64 Error: 5: Input/output error
Additional information: 8
Additional information: 58406

提示文件正在使用，unable to lock file

经过测试，只有expdp出现了该问题，正常的文件复制等操作均不受影响

到oracle support查资料，发现如下文章很类似：

RMAN and/or Datapump Export Backup fails with ORA-27086 on NFS / NAS (Doc ID 1187014.1)

提到：

APPLIES TO:
Oracle Database - Enterprise Edition - Version 10.2.0.1 to 11.2.0.1.0 [Release 10.2 to 11.2]
Information in this document applies to any platform.
***Checked for relevance on 15-Apr-2014***


SYMPTOMS
RMAN and/or Datapump Export Backup to NFS fails with ORA-27086 error.

The error stack looks like the following (exact error stack / messages depend on OS platform):

RMAN BACKUP Error Stack Example:

Starting Control File Autobackup at 20-AUG-10
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of Control File Autobackup command on ORA_DISK_1 channel at 08/20/2010 12:05:47
ORA-01580: error creating control backup file /backups/u01/fs/bkp/fs1/rman_snapshot_controlfile_fs1.f
ORA-27086: unable to lock file - already in use
HPUX-ia64 Error: 22: Invalid argument
Additional information: 9
Additional information: 22


OR

RMAN-10038: database session for channel default terminated unexpectedly
ORACLE error from target database:
ORA-03127: no new operations allowed until the active operation ends

Datapump Export Backup Error Stack Example:

Export: Release 10.2.0.4.0 - 64bit Production on Monday, 23 August, 2010 20:14:51

Copyright (c) 2003, 2007, Oracle. All rights reserved.

Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORA-39001: invalid argument value
ORA-39000: bad dump file specification
ORA-31641: unable to create dump file "/backups/u01/fs/exp/fs1/fs1_expdp/fs1_expdp01.20100823_201451.dmp"
ORA-27086: unable to lock file - already in use
HPUX-ia64 Error: 22: Invalid argument
Additional information: 9
Additional information: 22



CHANGES
RMAN and/or Datapump Export is taken on a NFS / NAS storage.

CAUSE
There can be several causes for this issue:

a) The first thing to check is if the storage solution provided by the Vendor is certified by Oracle.

b) Check if the mount points have been mounted with correct NFS mount options.

c) Check if the NFS locking daemons are running. They are also known as rpc.lockd and the rpc.statd on most UNIX flavours. We can also check the service 'nfslock' if it is running:

# service nfslock status

d) On some platforms, the NFS locking is not supported or having issues.

SOLUTION
Case a) We should verify if the storage configuration provided by the Vendor is certified by Oracle. This should be checked with Storage Vendor Support, as Oracle Support may have limited information on this.


Case b) For recommended NFS mount options, we can refer "Oracle Database Installation Guide" for our Database version and platform from the Online Documentation available from http://tahiti.oracle.com . Following Article should also be referred:

Note 359515.1 Mount Options for Oracle files when used with NAS devices


Case c) Check if the service 'nfslock' is running:

# service nfslock status

rpc.statd (pid 16489) is running...

If it is not running, then start the service:

# service nfslock start

We should also consider to enable this service via chkconfig also so that it is automatically started at OS restarts.


Case d) In case we suspect that NFS locking is an issue, consider adding "nolock" option in mount point. E.g:

rw,noac,hard,rsize=32768,wsize=32768,nolock

On some platforms like Solaris, HP Unix, option "llock" (local locking) can also be used. Be aware that it only enables locking locally for a single client and will not be visible to other clients.

Please also check with System Administrator and/or Platform/Storage Vendor before setting these options. Specifically locking options (nolock & llock) should be used with care as these options might violate integrity of the system if not appropriate for the environment setup.

 

尝试加了nolock选项，恢复正常，文章中也提到Solaris或者HPUX，应该使用llock，但是手头没有对应环境，无法测试。

[root@rac1 ~]# cat /etc/auto.cifs 
oradb -fstype=cifs,rw,noperm,nolock,user=lgdomain/backup,pass=xxx   ://192.168.1.23/oradb

添加nolock选项后，得到解决

注意文档中提到没有file lock的话可能会破坏文件完整性环境，造成文件损坏