rpm ps

 ps h -Led -o user | sort | uniq -c | sort -n
      1 chrony
      1 libstoragemgmt
      1 rpc
      2 dbus
      7 mfe
     12 polkitd
     49 splunk
    468 grid
   1789 root
   9572 oracle
 

SYMPTOMS

Observed high CPU utilization in the server by dockerd process.

top - 13:00:51 up 5 days, 22:23,  2 users,  load average: 42.04, 41.74, 41.38

Tasks: 738 total,   1 running, 736 sleeping,   0 stopped,   1 zombie

%Cpu(s): 31.6 us, 62.8 sy,  0.0 ni,  4.4 id,  0.0 wa,  0.7 hi,  0.4 si,  0.0 st

MiB Mem : 128268.7 total,  83097.8 free,  19883.5 used,  25287.4 buff/cache

MiB Swap:      0.0 total,      0.0 free,      0.0 used. 104073.3 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND

778770 root      20   0   21.4g  13.0g  53128 S  3420  10.4 751:54.25 dockerd

The dockerd process owned package is shipped by 3rd party - Docker.

[root@hostname ~]# rpm -qa --last | grep -i docker
docker-ce-20.10.10-3.el8.x86_64 Fri Nov 17 15:08:58 2023
docker-ce-cli-20.10.10-3.el8.x86_64 Fri Nov 17 15:07:50 2023
docker-ce-rootless-extras-24.0.7-1.el8.x86_64 Fri Nov 17 15:08:55 2023
docker-scan-plugin-0.23.0-3.el8.x86_64 Fri Nov 17 15:07:46 2023
[root@hostname ~]#

[root@hostname ~]# rpm -qi docker-ce-20.10.10-3.el8.x86_64 docker-ce-cli-20.10.10-3.el8.x86_64 docker-ce-rootless-extras-24.0.7-1.el8.x86_64 docker-scan-plugin-0.23.0-3.el8.x86_64
Name : docker-ce
Epoch : 3
Version : 20.10.10
Release : 3.el8
Architecture: x86_64
Install Date: Fri 17 Nov 2023 03:08:58 PM IST
Group : Tools/Docker
Size : 100016942
License : ASL 2.0
Signature : RSA/SHA512, Mon 25 Oct 2021 08:39:41 PM IST, Key ID c52feb6b621e9f35
Source RPM : docker-ce-20.10.10-3.el8.src.rpm
Build Date : Mon 25 Oct 2021 01:12:44 PM IST
Build Host : 656de9ed4ab6
Relocations : (not relocatable)
Packager : Docker 
Vendor : Docker
URL : https://www.docker.com
Summary : The open-source application container engine
Description : Docker is a product for you to build, ship and run any application as a lightweight container.
.....

SOLUTION

 To address the issue, engage respective 3rd party software provider to investigate. For this case, it is Docker.

[root@rac1 sbin]# pwd
/usr/sbin
[root@rac1 sbin]# ls -ltr glibc_post_upgrade.x86_64
-rwx------. 1 root root 790576 Oct 15  2021 glibc_post_upgrade.x86_64
[root@rac1 sbin]# rpm -qf glibc_post_upgrade.x86_64
glibc-2.17-325.0.1.el7_9.x86_64
[root@rac1 sbin]# rpm -qi glibc-2.17-325.0.1.el7_9.x86_64
Name        : glibc
Version     : 2.17
Release     : 325.0.1.el7_9
Architecture: x86_64
Install Date: Tue 12 Apr 2022 05:27:09 PM CST
Group       : System Environment/Libraries
Size        : 14103114
License     : LGPLv2+ and LGPLv2+ with exceptions and GPLv2+
Signature   : RSA/SHA256, Fri 15 Oct 2021 03:14:40 AM CST, Key ID 72f97b74ec551f03
Source RPM  : glibc-2.17-325.0.1.el7_9.src.rpm
Build Date  : Fri 15 Oct 2021 03:13:40 AM CST
Build Host  : build-ol7-x86_64.oracle.com
Relocations : (not relocatable)
Vendor      : Oracle America
URL         : http://www.gnu.org/software/glibc/
Summary     : The GNU libc libraries
Description :
The glibc package contains standard libraries which are used by
multiple programs on the system. In order to save disk space and
memory, as well as to make upgrading easier, common system code is
kept in one place and shared between programs. This particular package
contains the most important sets of shared libraries: the standard C
library and the standard math library. Without these two libraries, a
Linux system will not function.
[root@rac1 sbin]# rpm -qa --last | grep glibc
glibc-headers-2.17-325.0.1.el7_9.x86_64       Tue 12 Apr 2022 05:27:22 PM CST
glibc-devel-2.17-325.0.1.el7_9.x86_64         Tue 12 Apr 2022 05:27:22 PM CST
glibc-common-2.17-325.0.1.el7_9.x86_64        Tue 12 Apr 2022 05:27:14 PM CST
glibc-2.17-325.0.1.el7_9.x86_64               Tue 12 Apr 2022 05:27:09 PM CST
[root@rac1 sbin]# 

SYMPTOMS

After upgrading to Oracle Linux 7, the top command output shows lot of systemd-udev processes consuming high CPU:
 

Tasks: 3773 total, 87 running, 3686 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.3 us, 88.2 sy, 0.0 ni, 9.3 id, 0.1 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 52763843+total, 3410620 free, 30898044 used, 49332976+buff/cache
KiB Swap: 94371840 total, 94371840 free, 0 used. 38532902+avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
223212 root rt 0 1089732 127696 81136 D 331.0 0.0 2037:53 osysmond.bin
352793 root 20 0 44396 3228 2568 R 100.0 0.0 0:23.61 systemd-udevd
352920 root 20 0 44552 3384 2568 R 100.0 0.0 0:21.47 systemd-udevd
353331 root 20 0 44568 3400 2568 R 100.0 0.0 0:14.35 systemd-udevd
353573 root 20 0 44376 2300 1668 R 100.0 0.0 0:09.73 systemd-udevd
353966 root 20 0 44948 2872 1668 R 100.0 0.0 0:05.68 systemd-udevd
354138 root 20 0 45208 3112 1668 R 100.0 0.0 0:03.59 systemd-udevd
352721 root 20 0 44376 3208 2568 R 100.0 0.0 0:25.91 systemd-udevd

# udevadm monitor

The above output shows change events for many sd devices, and remove/add events for partitions on those devices.
 

The output may have similar keywords like change/add/remove as below:

KERNEL[1378104532.953656] remove /devices/..sdx1
KERNEL[1378104578.203955] change /devices/../sdx
UDEV [1378104578.252306] change /devices/../sdx
KERNEL[1378104578.007559] add /devices/..sdx1
UDEV [1378104578.196100] add /devices/../sdx1

CHANGES

 No changes.

CAUSE

The audit rules captured the process that is responsible for generating the udev events. Program /usr/sbin/amDaemon is opening the disk devices for writing (O_RDWR). This causes a udev event to be generated on close, so that udev can check to see if the program made any changes to the partition table, or labels, or UUIDs, or filesystem types in the partitions.
 
From audit logs:

type=PROCTITLE msg=audit(06/11/18 11:18:54.383:1078105) : proctitle=/usr/sbin/amDaemon <-------------
type=PATH msg=audit(06/11/18 11:18:54.383:1078105) : item=0 name=/dev/sdv inode=1533 dev=00:06 mode=block,660 ouid=root ogid=disk rdev=41:50 nametype=NORMAL
type=CWD msg=audit(06/11/18 11:18:54.383:1078105) : cwd=/
type=SYSCALL msg=audit(06/11/18 11:18:54.383:1078105) : arch=x86_64 syscall=open success=yes exit=36 a0=0x7f7714001038 a1=O_RDWR a2=0x7f773555e350 a3=0xfffff800 items=1 ppid=1 pid=88427 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=amDaemon exe=/opt/fujitsu/ServerViewSuite/RAIDManager/amDaemon key=blkchg

SOLUTION

Program /usr/sbin/amDaemon is not a part of Oracle Linux. It is included in rpm package from vendor Fujitsu.

Contact Fujitsu for a solution:

$ ls -la /usr/sbin/amDaemon
lrwxrwxrwx. 1 root root 49 Mar 24 2017 /usr/sbin/amDaemon -> /opt/fujitsu/ServerViewSuite/RAIDManager/amDaemon
$ rpm -qf /opt/fujitsu/ServerViewSuite/RAIDManager/amDaemon
SVAS_SVRAID-6.4-11.el7.x86_64