Advanced Programming in UNIX Environment Episode 73

We need to use caution when locking or unlocking byte ranges relative to the end of file. Most implementations convert an l_whence value of SEEK_CUR or SEEK_END into an absolute file offset, using l_start and the file’s current position or current length.

Consider the following sequence of steps:

writew_lock(fd, 0, SEEK_END, 0);
write(fd, buf, 1);
un_lock(fd, 0, SEEK_END);
write(fd, buf, 1);

This sequence of code might not do what you expect. It obtains a write lock from the current end of the file onward, covering any future data we might append to the file.

Mandatory locking causes the kernel to check every open, read, and write to verify that the calling process isn’t violating a lock on the file being accessed. Mandatory locking is sometimes called enforcement-mode locking.

Linux 3.2.0 and Solaris 10 provide mandatory record locking, but FreeBSD 8.0 and Mac OS X 10.6.8 do not. Mandatory record locking is not part of the Single UNIX Specification. On Linux, if you want mandatory locking, you need to enable it on a per file system basis by using the -o mand option to the mount command.

But if the file being opened has outstanding mandatory record locks (either read locks or write locks), and if the flags in the call to open specify either O_TRUNC or O_CREAT, then open returns an error of EAGAIN immediately, regardless of whether O_NONBLOCK is specified.

Only Solaris treats the O_CREAT flag as an error case. Linux allows the O_CREAT flag to be specified when opening a file with an outstanding mandatory lock. Generating the open error for O_TRUNC makes sense, because the file cannot be truncated if it is read locked or write locked by another process. Generating the error for O_CREAT, however, makes little sense; this flag says to create the file only if it doesn’t already exist, but it has to exist to be record locked by another process.

During this sleep period, the following behavior was seen in other typical UNIX System programs.

• The same file could be edited with the ed editor, and the results written back to disk! The mandatory record locking had no effect at all.

Under FreeBSD 8.0 and Solaris 10, we can obtain the system call trace of a process with the truss(1) command. Linux 3.2.0 provides the strace(1) command for the same purpose. Mac OS X 10.6.8 provides the dtruss(1m) command to trace system calls, but its use requires superuser privileges.

• The vi editor was never able to edit the file.
• Using the Korn shell’s > and >> operators to overwrite or append to the file resulted in the error ‘‘cannot create.’’
• Using the same two operators with the Bourne shell resulted in an error for >, but the >> operator just blocked until the mandatory lock was removed, and then proceeded. (The difference in the handling of the append operator occurs because the Korn shell opens the file with O_CREAT and O_APPEND, and we mentioned earlier that specifying O_CREAT generates an error. The Bourne shell, however, doesn’t specify O_CREAT if the file already exists, so the open succeeds but the next write blocks.)

Mandatory record locking can also be used by a malicious user to hold a read lock on a file that is publicly readable. This can prevent anyone from writing to the file. (Of course, the file has to have mandatory record locking enabled for this to occur, which may require the user to be able to change the permission bits of the file.)

The normal UNIX System text editors do not use record locking, so the answer is still that the final result of the file corresponds to the last process that wrote the file.

Some versions of the vi editor use advisory record locking. Even if we were using one of these versions of vi, it still doesn’t prevent users from running another editor that doesn’t use advisory record locking.

If the system provides mandatory record locking, we could modify our favorite editor to use it (if we have the editor’s source code). Not having the source code for the editor, we might try the following. We write our own program that is a front end to vi. This program immediately calls fork, and the parent just waits for the child to complete. The child opens the file specified on the command line, enables mandatory locking, obtains a write lock on the entire file, and then executes vi. While vi is running, the file is write locked, so other users can’t modify it. When vi terminates, the parent’s wait returns and our front end terminates.