scsi eh(scsi error handle)处理流程

内核文档
Documentation/scsi/scsi_eh.txt

scmd为scsi cmd的简称。

scsi_eh_scmd_add()作用：将发生error的scmd加入到eh中。

发生error有两种情况：

1).scmd执行完成了，但是结果为error。

2).scmd执行超时了，即一直没有返回，发生了timeout。

scsi_eh_scmd_add()在两处被调用：

1).scsi_times_out函数 => scmd超时处理函数，将timeout的scmd加入eh
2).scsi_softirp_done函数 => scmd命令结果处理函数，将结果为error的scmd加入eh

scsi_eh_scmd_add()完成以下工作：

1. Turns on scmd->eh_eflags as requested. It’s 0 for error
   completions and SCSI_EH_CANCEL_CMD for timeouts.

2. Links scmd->eh_entry to shost->eh_cmd_q

3. Sets SHOST_RECOVERY bit in shost->shost_state

4. Increments shost->host_failed

5. Wakes up SCSI EH thread if shost->host_busy == shost->host_failed

scsi-high层
中间层完成命令时通过调用scmd->done()通知scsi-high层SCSI命令处理完毕。
HLD completion callback - sd:sd_rw_intr, sr:rw_intr, st:st_intr.

scsi-mid层
1). LLDD顺利完成SCSI命令 => 调用scsi-mid层提供的回调函数scsi_done()。
2). LLDD没能顺利完成SCSI命令 => scsi-mid层将该scmd time out，调用scsi_times_out()处理该命令。

LLDD层

SCSI EH如何工作
LLDD可以通过以下两种方式实现SCSI EH

• 实现EH回调函数
  int (* eh_abort_handler)(struct scsi_cmnd );
  int ( eh_device_reset_handler)(struct scsi_cmnd );
  int ( eh_bus_reset_handler)(struct scsi_cmnd );
  int ( eh_host_reset_handler)(struct scsi_cmnd *);
  供scsi_unjam_host调用完成SCSI EH的全部工作。
• 实现eh_strategy_handler()回调函数
  transportt->eh_strategy_handler()函数完成全部的EH工作。

英文版
How SCSI EH works
LLDD’s can implement SCSI EH actions in one of the following two
ways.

• Fine-grained EH callbacks
  LLDD can implement fine-grained EH callbacks and let SCSI
  midlayer drive error handling and call appropriate callbacks.
  This will be discussed further in [2-1].

• eh_strategy_handler() callback
  This is one big callback which should perform whole error
  handling. As such, it should do all choirs SCSI midlayer
  performs during recovery. This will be discussed in [2-2].

/**

• scsi_error_handler - SCSI error handler thread
• @data: Host for which we are running.
• Notes:
• This is the main error handling loop. This is run as a kernel thread
• for every SCSI host and handles all error handling activity.
  */
  int scsi_error_handler(void data)
  {
  …
  / 如果eh_strategy_handler在LLDD中有实现，则调用该函数处理，
  否则调用通用处理函数scsi_unjam_host进行处理，scsi_unjam_host
  会通过调用LLDD定义的EH回调函数们来完成工作。
  */
  if (shost->transportt->eh_strategy_handler)
  shost->transportt->eh_strategy_handler(shost);
  else
  scsi_unjam_host(shost);
  …
  }

/**

• scsi_host_alloc - register a scsi host adapter instance.
• @sht: pointer to scsi host template
• @privsize: extra bytes to allocate for driver
• Note:

• Allocate a new Scsi_Host and perform basic initialization.
  

• The host is not published to the scsi midlayer until scsi_add_host
  

• is called.
  

• Return value:

• Pointer to a new Scsi_Host
  

**/
struct Scsi_Host *scsi_host_alloc(struct scsi_host_template sht, int privsize)
{
…
shost->ehandler = kthread_run(scsi_error_handler, shost, / scsi_error_handler在scsi_host_alloc函数中被指定 */
“scsi_eh_%d”, shost->host_no);
…

}

static void scsi_unjam_host(struct Scsi_Host *shost)
{
unsigned long flags;
LIST_HEAD(eh_work_q);
LIST_HEAD(eh_done_q);

spin_lock_irqsave(shost->host_lock, flags);
list_splice_init(&shost->eh_cmd_q, &eh_work_q);
spin_unlock_irqrestore(shost->host_lock, flags);

SCSI_LOG_ERROR_RECOVERY(1, scsi_eh_prt_fail_stats(shost, &eh_work_q));

/* scsi_eh_get_sense函数最后调用list_empty(eh_work_q),如果为空，则返回1，否则返回0，
   继续执行更高级别的EH。即如果 scsi_eh_get_sense函数能解决掉所有eh_work_q中的scmd，
   则不许要下一步调用，否则，继续进行更高级别的EH。
*/
if (!scsi_eh_get_sense(&eh_work_q, &eh_done_q))
    if (!scsi_eh_abort_cmds(&eh_work_q, &eh_done_q))
        scsi_eh_ready_devs(shost, &eh_work_q, &eh_done_q);

scsi_eh_flush_done_q(&eh_done_q);

}

对于error-completed (completed but result is error) scmd，调用scsi_eh_get_sense将它们结束
对于timed-out scmd，
a).调用scsi_eh_abort_cmds让LLDD忽略这个scmd
b).调用scsi_eh_stu 对每个device，执行一遍START_UNIT命令，

1). First call scsi_eh_get_sense
<<scsi_eh_get_sense>> /** 处理error-completed scmd，将它们连同sence data返回给scsi-high层处理 **/
该函数是为那些error-completed（complted，但是结果有问题）的scmd准备的。
如果所有scmd都是error-completed的，且成功获得了sense data，scsi_eh_finish_cmd()被调用将该scmd
移到eh_done_q，不需要进一步操作，EH完成；
否则经过scsi_eh_get_sense函数之后，eh_work_q链表仍然非空（即还有无法处理的scmd）则进行更高级别的EH。
标记为 SCSI_EH_CANCEL_CMD的scmd是因为time out而进入EH链表的，不是scsi_eh_get_sense函数处理对象。

2). If !list_empty(&eh_work_q), invoke scsi_eh_abort_cmds().
<<scsi_eh_abort_cmds>>/** 处理timed-out scmd，让底层LLDD abort（终止）这些scmd **/
该函数是为那些timed-out的scmd准备的。
调用hostt->eh_abort_handler()函数依次处理每一个scmd，如果该函数成功地让LLDD和相关的硬件忽略了这个scmd，
则成功返回SUCCESS，。
如果一个timed-out scmd被成功地终止掉了，那么这个scmd对应的device就应该处于offline或者ready状态。
根据scsi_try_to_abort_cmd函数的返回值移动scmd
返回值：
a).FAST_IO_FAIL，则调用scsi_eh_finish_cmd()将该scmd 移到eh_done_q；
b).SUCCESS，则进行检查，如果scmd对应的device是offline或者ready状态，则调用scsi_eh_finish_cmd()
将它移到eh_done_q，否则不移动。
c).否则不移动scmd，它仍然保留在eh_work_q中
如果有不成功的scmd仍然留在eh_work_q链表中，则调用更高级别的EH。

3). If !list_empty(&eh_work_q), invoke scsi_eh_ready_devs()
<<scsi_eh_ready_devs>> /** **/
void scsi_eh_ready_devs(struct Scsi_Host *shost,
struct list_head *work_q,
struct list_head *done_q)
{
if (!scsi_eh_stu(shost, work_q, done_q))
if (!scsi_eh_bus_device_reset(shost, work_q, done_q))
if (!scsi_eh_target_reset(shost, work_q, done_q))
if (!scsi_eh_bus_reset(shost, work_q, done_q))
if (!scsi_eh_host_reset(work_q, done_q))
scsi_eh_offline_sdevs(work_q,
done_q);
}

<<scsi_eh_stu>>
每个device，执行一遍START_UNIT命令，然后检查执行结果。
如果device是offline或者ready状态，则调用scsi_eh_finish_cmd()将它关联的scmd都移动到eh_done_q，否则不移动。

<<scsi_eh_test_devices>>

测试device状态：offline、online、online && ready
/**

• scsi_eh_test_devices - check if devices are responding from error recovery.
• @cmd_list: scsi commands in error recovery.
• @work_q: queue for commands which still need more error recovery
• @done_q: queue for commands which are finished
• @try_stu: boolean on if a STU command should be tried in addition to TUR.
• Decription:
• Tests if devices are in a working state. Commands to devices now in
• a working state are sent to the done_q while commands to devices which
• are still failing to respond are returned to the work_q for more
• processing.
  **/
  static int scsi_eh_test_devices(struct list_head *cmd_list,
  struct list_head *work_q,
  struct list_head *done_q, int try_stu)
  {undefined
  struct scsi_cmnd *scmd, *next;
  struct scsi_device *sdev;
  int finish_cmds;

while (!list_empty(cmd_list)) {undefined
    scmd = list_entry(cmd_list->next, struct scsi_cmnd, eh_entry);
    sdev = scmd->device;
           /* 1.if scsi_device_online() return value is 0, means the device is offline, then the things
                   after || don't need to be done.
                2.if scsi_device_online() return value is 1, means the device is online, then the things
                   after || need to be done to juge if the device is in a ready state.
                3.the way to test if device is in ready state is to call function scsi_eh_tur() to
                   send TEST_UNIT_READY cmd to the device and see the return value.
             */
    finish_cmds = !scsi_device_online(scmd->device) ||
        (try_stu && !scsi_eh_try_stu(scmd) &&
         !scsi_eh_tur(scmd)) ||
        !scsi_eh_tur(scmd);

    list_for_each_entry_safe(scmd, next, cmd_list, eh_entry)
        if (scmd->device == sdev) {undefined
            if (finish_cmds)
                scsi_eh_finish_cmd(scmd, done_q);
            else
                list_move_tail(&scmd->eh_entry, work_q);
        }
}
return list_empty(work_q);

}

static void scsi_unjam_host(struct Scsi_Host *shost)
{undefined
unsigned long flags;
LIST_HEAD(eh_work_q);
LIST_HEAD(eh_done_q);

spin_lock_irqsave(shost->host_lock, flags);
list_splice_init(&shost->eh_cmd_q, &eh_work_q); /将eh_cmd_q中的scmd复制到eh_work_q中/
spin_unlock_irqrestore(shost->host_lock, flags);

SCSI_LOG_ERROR_RECOVERY(1, scsi_eh_prt_fail_stats(shost, &eh_work_q));
/*eh_work_q中的scmd分为两类：
1.scmd执行完成了，但是执行结果为error的。即errored scmd。
2.scmd执行了就没有返回，知道超时发生time out，即timed-out scmd。
*/
/*1.调用scsi_eh_get_sense()从errored scmd的目标设备处获取sense信息。
2.sense信息用途：作为上层驱动判断scmd发生error的原因的依据。
3.然后将errored scmd处理掉(即加入eh_done_q并调用finish函数将scmd返回上层)。
4.eh_done_q中剩余的全部都是timed-out scmd。
*/
if (!scsi_eh_get_sense(&eh_work_q, &eh_done_q))
/*1.对eh_work_q中剩余的timed-out scmd，执行abort操作(abort，放弃执行)，
2.如果abort之后，eh_work_q还不为空，即还有没处理掉的timed-out scmd，
调用scsi_eh_ready_devs开始逐步升级的reset操作，直到eh_work_q中的scmd处理光。
*/
if (!scsi_eh_abort_cmds(&eh_work_q, &eh_done_q))
scsi_eh_ready_devs(shost, &eh_work_q, &eh_done_q);

scsi_eh_flush_done_q(&eh_done_q);
}

static int scsi_eh_abort_cmds(struct list_head *work_q,
struct list_head *done_q)
{undefined
struct scsi_cmnd *scmd, next;
LIST_HEAD(check_list);
int rtn;
/循环对每个scmd执行一遍abort，放弃该命令的执行/
list_for_each_entry_safe(scmd, next, work_q, eh_entry) {undefined
if (!(scmd->eh_eflags & SCSI_EH_CANCEL_CMD))
continue;
SCSI_LOG_ERROR_RECOVERY(3, printk("%s: aborting cmd:"
“0x%p\n”, current->comm,
scmd));
/ 执行对当前scmd的abort /
rtn = scsi_try_to_abort_cmd(scmd->device->host->hostt, scmd);
/ 如果执行结果为SUCCESS，或者期望通过将IO设置为FAIL快速返回，则进入 */
if (rtn == SUCCESS || rtn == FAST_IO_FAIL) {undefined
scmd->eh_eflags &= ~SCSI_EH_CANCEL_CMD;
if (rtn == FAST_IO_FAIL)
scsi_eh_finish_cmd(scmd, done_q);/将返回FAST_IO_FAIL的scmd移动到done_q从而finish掉/
else
list_move_tail(&scmd->eh_entry, &check_list); /将返回SUCCESS的scmd加入check_list/
} else
SCSI_LOG_ERROR_RECOVERY(3, printk("%s: aborting"
" cmd failed:"
“0x%p\n”,
current->comm,
scmd));
}
/*调用scsi_eh_test_devices()函数，检查返回SUCCESS的scmd们(都保存在check_list中)的目标设备的状态。
如果一个scmd的目标设备的状态要么为offline，要么为ready，则说明对这个scmd的abort操作成功了，
可以将这个scmd移动到done_q中finish掉这个scmd了。
*/
return scsi_eh_test_devices(&check_list, work_q, done_q, 0);
}

static int scsi_eh_test_devices(struct list_head *cmd_list,
struct list_head *work_q,
struct list_head *done_q, int try_stu)
{ /*该函数的改进在于如果一个scmd的目标设备处于offline或者ready，则把check_list中的其他具有该目标设备
的scmd也移动到done_q中finish掉。而不是每个scmd都检测一遍它的目标设备，提高了效率。
即以sdev为单位处理check_list中的scmd，而不是以scmd为单位挨个执行，做很多重复工作。
*/
struct scsi_cmnd *scmd, *next;
struct scsi_device *sdev;
int finish_cmds;

while (!list_empty(cmd_list)) {undefined
scmd = list_entry(cmd_list->next, struct scsi_cmnd, eh_entry);
sdev = scmd->device;
/* 1.if scsi_device_online return value is 0, means the device is offline, then the things
after || don’t need to be done.
2.if scsi_device_online return value is 1, means the device is online, then the things
after || need to be done to juge if the device is in a ready state.
3.the way to test if device is in ready state is to call function scsi_eh_tur to
send TEST_UNIT_READY cmd to the device and see the return value.
*/
finish_cmds = !scsi_device_online(scmd->device) ||
(try_stu && !scsi_eh_try_stu(scmd) &&
!scsi_eh_tur(scmd)) ||
!scsi_eh_tur(scmd);
/到check_list中检测目标设备是上面的sdev的scmd，将它们都处理掉，提高效率。/
list_for_each_entry_safe(scmd, next, cmd_list, eh_entry)
if (scmd->device == sdev) {
if (finish_cmds)
scsi_eh_finish_cmd(scmd, done_q);
else
list_move_tail(&scmd->eh_entry, work_q);
}
}
return list_empty(work_q);
}