Unix/C/C++进阶--线程 pthread_creat() return 1失败

1 介绍

1.1 man pthread_create

PTHREAD_CREATE(3)                                                                     Linux Programmer's Manual                                                                     PTHREAD_CREATE(3)

NAME
       pthread_create - create a new thread

SYNOPSIS
       #include <pthread.h>

       int pthread_create(pthread_t *thread, const pthread_attr_t *attr,
                          void *(*start_routine) (void *), void *arg);

       Compile and link with -pthread.

DESCRIPTION
       The pthread_create() function starts a new thread in the calling process.  The new thread starts execution by invoking start_routine(); arg is passed as the sole argument of start_routine().

       The new thread terminates in one of the following ways:

       * It calls pthread_exit(3), specifying an exit status value that is available to another thread in the same process that calls pthread_join(3).

       * It returns from start_routine().  This is equivalent to calling pthread_exit(3) with the value supplied in the return statement.

       * It is canceled (see pthread_cancel(3)).

       * Any of the threads in the process calls exit(3), or the main thread performs a return from main().  This causes the termination of all threads in the process.

       The  attr  argument  points  to  a  pthread_attr_t  structure  whose contents are used at thread creation time to determine attributes for the new thread; this structure is initialized using
       pthread_attr_init(3) and related functions.  If attr is NULL, then the thread is created with default attributes.

       Before returning, a successful call to pthread_create() stores the ID of the new thread in the buffer pointed to by thread; this identifier is used to refer to the thread in subsequent calls
       to other pthreads functions.

       The  new  thread  inherits a copy of the creating thread's signal mask (pthread_sigmask(3)).  The set of pending signals for the new thread is empty (sigpending(2)).  The new thread does not
       inherit the creating thread's alternate signal stack (sigaltstack(2)).

       The new thread inherits the calling thread's floating-point environment (fenv(3)).

       The initial value of the new thread's CPU-time clock is 0 (see pthread_getcpuclockid(3)).

   Linux-specific details
       The new thread inherits copies of the calling thread's capability sets (see capabilities(7)) and CPU affinity mask (see sched_setaffinity(2)).

RETURN VALUE
       On success, pthread_create() returns 0; on error, it returns an error number, and the contents of *thread are undefined.

ERRORS
       EAGAIN Insufficient resources to create another thread.

       EAGAIN A system-imposed limit on the number of threads was encountered.  There are a number of limits that may trigger this error:  the  RLIMIT_NPROC  soft  resource  limit  (set  via  setr‐
              limit(2)),  which  limits  the  number  of processes and threads for a real user ID, was reached; the kernel's system-wide limit on the number of processes and threads, /proc/sys/ker‐
              nel/threads-max, was reached (see proc(5)); or the maximum number of PIDs, /proc/sys/kernel/pid_max, was reached (see proc(5)).

       EINVAL Invalid settings in attr.

1.2 一般失败原因

EAGAIN 创建线程时，资源数不足，或者是遇到了系统对线程数量的限制
EINVAL arrt 中使用了无效参数
EPERM 系统不允许设置调度策略 和 在attr中使用特定的参数

EPERM的定义，在 erron.h 头文件中的定义，就是1

1.3 最大进程/线程数（理论）

以imx6为例：

cat /proc/sys/kernel/pid_max
32768

可以使用ulimit –u 查看系统限制某用户最大可以运行多少线程max_user_process

ulimit -u 65535
sysctl -w kernel.pid_max=65535

1.3 最大进程/线程数（实际）

理论上linux 上最大线程数是 = 总虚拟内存（用户空间） / 线程栈大小；

一般32bit PC机系统上，进程空间是4G，其中0—3G 是用户空间，3G —4G 是内核空间，所以理论上最大线程数 = 3*1024/ 8M = 384个，考虑系统主线程占用情况，故可创建的最大线程大概为 384 个；如何进程或者线程占用的栈空间小于8M，实际能创建更多的进程/线程。

1.4 查看Linux实际运行的总线程数

ps -eL | wc –l
176

1.5 查看运行的进程

所有：
ps aux

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.3   5480  3708 ?        Ss   May20   0:04 /sbin/init
root         2  0.0  0.0      0     0 ?        S    May20   0:00 [kthreadd]
root         3  0.1  0.0      0     0 ?        R    May20   2:32 [ksoftirqd/0]
root         4  0.1  0.0      0     0 ?        S    May20   1:48 [ktimersoftd/0]
root         6  0.0  0.0      0     0 ?        S<   May20   0:00 [kworker/0:0H]
root         7  0.0  0.0      0     0 ?        S    May20   0:00 [kworker/u8:0]
root         8  0.0  0.0      0     0 ?        S    May20   0:36 [rcu_preempt]
root         9  0.0  0.0      0     0 ?        S    May20   0:00 [rcu_sched]
root        10  0.0  0.0      0     0 ?        S    May20   0:00 [rcu_bh]

筛选：
ps aux | grep 某进程名

root     18741  0.0  0.2   3832  2248 pts/0    Sl   07:41   0:00 ./ubase

1.6 查看进程信息

打印信息中Threads后面就是线程数量。

cat /proc/{pid}/status

Name:   ubase
State:  S (sleeping)
Tgid:   18741
Ngid:   0
Pid:    18741
PPid:   9740
TracerPid:      0
Uid:    0       0       0       0
Gid:    0       0       0       0
FDSize: 256
Groups: 0
VmPeak:     3832 kB
VmSize:     3832 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:      2248 kB
VmRSS:      2248 kB
VmData:      716 kB
VmStk:       136 kB
VmExe:        92 kB
VmLib:      2512 kB
VmPTE:        14 kB
VmPMD:         0 kB
VmSwap:        0 kB
Threads:        2
SigQ:   0/5431
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: 0000000180000000
CapInh: 0000000000000000
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
Cpus_allowed:   f
Cpus_allowed_list:      0-3
Mems_allowed:   1
Mems_allowed_list:      0
voluntary_ctxt_switches:        206
nonvoluntary_ctxt_switches:     18

1.7 top -H 一行显示一个线程

top -H
top - 07:54:14 up 1 day,  2:29,  1 user,  load average: 0.42, 0.23, 0.17
Threads: 176 total,   1 running, 175 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.7 us,  2.0 sy,  0.0 ni, 97.3 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem :  1023424 total,   664420 free,   162856 used,   196148 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   823124 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
27866 root      20   0    5068   2344   1816 R  1.3  0.2   0:00.32 top
    1 root      20   0    5480   3708   2552 S  0.0  0.4   0:04.75 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.01 kthreadd
    3 root      20   0       0      0      0 S  0.0  0.0   2:33.51 ksoftirqd/0
    4 root      -2   0       0      0      0 S  0.0  0.0   1:49.58 ktimersoftd/0
    6 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker/0:0H
    7 root      20   0       0      0      0 S  0.0  0.0   0:00.07 kworker/u8:0
    8 root      -2   0       0      0      0 S  0.0  0.0   0:37.30 rcu_preempt
    9 root      -2   0       0      0      0 S  0.0  0.0   0:00.00 rcu_sched
   10 root      -2   0       0      0      0 S  0.0  0.0   0:00.00 rcu_bh
   11 root      -2   0       0      0      0 S  0.0  0.0   0:00.00 rcub/0
   12 root      -2   0       0      0      0 S  0.0  0.0   0:49.81 rcuc/0
   13 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kclksetdelayd
   14 root      rt   0       0      0      0 S  0.0  0.0   0:00.00 posixcputmr/0
   15 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kcmosdelayd
   16 root      rt   0       0      0      0 S  0.0  0.0   0:07.62 migration/0
   17 root      rt   0       0      0      0 S  0.0  0.0   0:07.22 migration/1
   18 root      -2   0       0      0      0 S  0.0  0.0   1:13.46 rcuc/1

1.8 top -Hp pid 查看进程信息（资源暂用、线程状态）

 top -Hp 18741

top - 07:57:10 up 1 day,  2:32,  1 user,  load average: 0.09, 0.16, 0.14
Threads:   2 total,   0 running,   2 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.7 us,  2.5 sy,  0.0 ni, 96.6 id,  0.2 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1023424 total,   664040 free,   163208 used,   196176 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   822776 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
18741 root      20   0    3832   2248   2100 S  0.0  0.2   0:00.03 ubase
18743 root      -2   0    3832   2248   2100 S  0.0  0.2   0:00.00 Can0_Motor

1.9 ulimits -s 查看默认栈空间

8192KB(8M)

ulimits -s
8192

2 问题描述

在Linux系统 ubuntu16.04 中，ssh进去，root下，运行进程，pthread_create实时线程创建失败，返回1。

3 问题分析

EPERM的定义，在 erron.h 头文件中的定义，就是1。从解释看，就是说没有权限来设置调度策略及参数定义。
systemd默认将ssh连接放到一个非rt组中，如果试图ssh并运行FIFO/RR任务，它将不起作用。您需要将shell移到允许优先级调度的默认cpu:/ group中。
cgclass -g cpu:/ $$
如果你把ControlGroup=cpu:/放在[service]下，你的systemd单元文件可以启动你的服务并获得优先级调度。

from：https://github.com/coreos/bugs/issues/410

what may or may not be very related, systemd defaults put ssh connections into a non-RT cgroup. if you're trying to ssh in and run FIFO/RR tasks, it just won't work. you need to move your shell into the default cpu:/ group which does permit priority scheduling.

cgclassify -g cpu:/ $$

your systemd unit file can start your service and get priority scheduling if you put ControlGroup=cpu:/ under [Service]

4 解决方案

4.1 service 中删除Delegate=yes

cat lib/systemd/system/user@.service
将Delegate=yes注释掉，就可以了。

4.2 向rt_runtime_us文件中写入数据

向rt_runtime_us文件中写入数据950000。
echo 950000 > /sys/fs/cgroup/cpu,cpuacct/user.slice/cpu.rt_runtime_us
echo 950000 > /sys/fs/cgroup/cpu,cpuacct/user.slice/user-0.slice/cpu.rt_runtime_us
echo 950000 > /sys/fs/cgroup/cpu,cpuacct/user.slice/user-0.slice/session-c1.scope/cpu.rt_runtime_us

参考

1、
2、pthread_create priority thread returns EPERM when run as root
3、Linux pthread_creat() 创建线程失败问题总结
4、Linux创建实时线程失败 （pthread_create SCHED_RR 失败）
5、pthread_create返回值