This document gives a brief information about “D“ state of a Linux process.

To check if a system has processes that are on "D" State or D-state, the output of the ps command can be checked. It can be something similar to:

ps aux | awk '$8 ~ /^D/{print}'

 This will display processes that are currently on 'D' state.

 

Linux follows the standard of traditional UNIX and computes its load average as the average number of runnable or running processes (R state) and the number of processes in uninterruptable sleep (D state) over the specified interval. 

"D" state or TASK_UNINTERRUPTIBLE is a state which occurs in a kernel code path where the execution can not be interrupted whilst a task is processed. Normally, this should be a fleeting state and in normal operations, kernel threads

frequently pass in and out of the TASK_UNINTERRUPTIBLE state. An example of this might be a low level driver talking to hardware, perhaps retrieving network packet data from NIC firmware or accessing a block of data on a hard disk drive -- read and write IO.

Normally this happens extremely quickly and threads remain in this state for very short periods of time (thus not normally observed, especially in user space).

The problem arises when a thread enters "D" state and fails to exit this state in a reasonable amount of time. That process is now "stuck" and any process waiting for it (possibly behind it in a queue to access the same hardware), or relying on it is similarly stuck.

In addition, it is in a 'sleep' state that won't handle a signal right away. It will wake only as a result of a waited-upon resource becoming available or after a time-out occurs during that wait (if specified when put to sleep). It is mostly used by device drivers waiting for disk or network IO (input/output). When the process is sleeping uninterruptibly, signals accumulated during the sleep will be noticed when the process returns from the system call or trap.

 

In most cases, this is caused by interrupted access to an I/O resource (typically local or remote storage, network file system, etc).

If a process remains stuck/stalled in D state for too long, the logic for "stalled task" within the kernel will be engaged and messages like below may start to appear in system logs:

INFO: task <process>:<pid> blocked for more than 120 seconds

These messages typically mean that the system is experiencing disk or memory congestion and processes are being starved of available resources. It also serve as a warning that something may not be operating optimally. These do not necessarily indicate a serious problem and any blocked processes should eventually proceed when the system recovers.

 

Actions to consider:

"D" state processes cannot be killed, even with SIGKILL or kill -9. As the name implies, they are uninterruptible. It can only be cleared by rebooting the server or waiting for the I/O to respond. It is normal to see processes in a "D" state when the server performs I/O intensive operations.

The first step towards resolution is to isolate the resource (most likely storage/filesystem) which is causing this condition. It may be helpful to check the common denominator of the processes in D state (e.g. current working directory, etc.)

Once the resource in question has been identified, steps should be taken to recover access to it; depending on the particular situation, it might be possible to re-gain access to the filesystem or Storage online, or most likely a reboot of the machine might be required for a full recovery.