Linux下使用pthread实现进程锁的注意事项

1. 背景

最近通过共享内存mmap实现多进程的数据共享，使用PTHREAD_PROCESS_SHARED时写错了，导致进程锁不起作用。

2. 接口

接口说明如下，用法与多线程的互斥锁类似，但需要额外设置一下PTHREAD_PROCESS_SHARED

SYNOPSIS
       #include <pthread.h>

       int pthread_mutexattr_getpshared(
                             const pthread_mutexattr_t *restrict attr,
                             int *restrict pshared);
       int pthread_mutexattr_setpshared(pthread_mutexattr_t *attr,
                             int pshared);

       Compile and link with -pthread.

DESCRIPTION
       These  functions  get  and  set  the process-shared attribute in a mutex attributes object.  This attribute must be appropriately set to ensure correct, efficient operation of a
       mutex created using this attributes object.

       The process-shared attribute can have one of the following values:

       PTHREAD_PROCESS_PRIVATE
              Mutexes created with this attributes object are to be shared only among threads in the same process that initialized the  mutex.   This  is  the  default  value  for  the
              process-shared mutex attribute.

       PTHREAD_PROCESS_SHARED
              Mutexes  created  with  this attributes object can be shared between any threads that have access to the memory containing the object, including threads in different pro‐
              cesses.

3 代码

实现一个N个进程同时操作一块共享内存的代码，8个进程每个对变量进行val++操作10w次，所以是期望最终结果是80w次就为正确。

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

#include <pthread.h>

#include <sys/stat.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <sys/mman.h>
#include <fcntl.h>

#include <assert.h>

#if 1
struct ut_mmap {
    int val;
    pthread_mutex_t mutex;  // must be here !!!!!!!!
};
# define MUTEX &pmap->mutex
#else
struct ut_mmap {
    int val;
};
pthread_mutex_t g_mutex;
# define MUTEX &g_mutex
#endif

/***************************************************************************************
 *   Name: Main
 *   Desc: main entrance
 *  Input: -
 * Output: -
 * Return: int, EXIT_SUCCESS on success;
 *              EXIT_FAILURE on failure;
 * Others: -
 ***************************************************************************************/
int main(int argc, char *argv[])
{
    int res = -1;
    struct ut_mmap *pmap = NULL;
    pthread_mutexattr_t mutexattr;

    {
        int fd = open("ut_mmap.shm", O_CREAT | O_RDWR, 0777);
        (void)ftruncate(fd, sizeof(*pmap));
        pmap = (struct ut_mmap *)mmap(NULL, sizeof(*pmap), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
        close(fd);
    }

    /* Reset */
    memset(pmap, 0, sizeof(*pmap));
    pthread_mutexattr_init(&mutexattr);
    pthread_mutexattr_setpshared(&mutexattr, PTHREAD_PROCESS_SHARED);
    pthread_mutex_init(MUTEX, &mutexattr);

    const int num = 8;
    const int times = 100000;

    for (int ix = 0; ix < num; ++ix) {
        pid_t pid = fork();
        if (pid == 0) {
            for (int jx = 0; jx < times; jx++) {
                pthread_mutex_lock(MUTEX);
                pmap->val++;  // 变量互斥操作加一
                pthread_mutex_unlock(MUTEX);
            }
            res = 0;
            goto out;
        }
        assert(pid > 0);
    }

    for (int ix = 0; ix < num; ++ix) {
        wait(NULL);
    }

    printf("Result: %d\n", pmap->val);
    assert(pmap->val == num * times);   // !!!!!!!

    pthread_mutex_destroy(MUTEX);

    munmap(pmap, sizeof(*pmap));
    unlink("ut_mmap.shm");

    res = 0;
out:
    printf("[PID:%8lu] Result:\t\t\t\t[%s]\n", (long)getpid(), res ? "Failure" : "Success");
    exit(res ? EXIT_FAILURE : EXIT_SUCCESS);
}

注意这个地方，设置1的时候是正确的写法，设置为0是错误的写法

#if 1
struct ut_mmap {
    int val;
    pthread_mutex_t mutex;  // must be here !!!!!!!!
};
# define MUTEX &pmap->mutex
#else
struct ut_mmap {
    int val;
};
pthread_mutex_t g_mutex;
# define MUTEX &g_mutex
#endif

正确的执行结果如下：

[PID:    2779] Result:                          [Success]
[PID:    2776] Result:                          [Success]
[PID:    2782] Result:                          [Success]
[PID:    2781] Result:                          [Success]
[PID:    2780] Result:                          [Success]
[PID:    2778] Result:                          [Success]
[PID:    2777] Result:                          [Success]
[PID:    2783] Result:                          [Success]
Result: 800000
[PID:    2775] Result:                          [Success]

错误的执行结果就加不到800000了，由于没有设置sleep，更容易出现竞争锁的情况出错

[PID:    2798] Result:                          [Success]
[PID:    2801] Result:                          [Success]
[PID:    2803] Result:                          [Success]
[PID:    2799] Result:                          [Success]
[PID:    2800] Result:                          [Success]
[PID:    2802] Result:                          [Success]
[PID:    2804] Result:                          [Success]
[PID:    2805] Result:                          [Success]
Result: 148835
lock: lock.cc:95: int main(int, char**): Assertion `pmap->val == num * times' failed.
Aborted (core dumped)

4. 结论

1. 进程间共享内存，pthread_mutex变量需要包含在共享内存中，否则不生效。
2. 压测多进程需要给大规模数据，增加竞争可以容易出效果来。
3. mmap配合文件锁fcntl也可以得到类似效果，但性能比不上pthread。