Processing SCSI subsystem details (scsishow)¶
The scsishow program available in the mpykdump extension allows users to quickly fetch detailed information about SCSI HBAs, FC, iSCSI targets, SCSI devices, and SCSI commands. It also provides the '--check' option which can detect stalled error handling process, stuck I/O requests or SCSI commands, and other common I/O issues.
Options provided by ‘scsishow’:
crash> scsishow -h
usage: scsishow [-h] [-v] [-p] [-d [FIELDS]] [-s [FIELDS]] [-T [FIELDS]] [-f [FIELDS]] [-i [FIELDS]] [-c [FIELDS]] [-q [FIELDS]] [-r [FIELDS]] [-x] [-j] [--check] [--time] [--relative [RELATIVE]]
optional arguments:
-h, --help show this help message and exit
-v, --verbose verbose output
-p, --proc show /proc/scsi/scsi style information
-d [FIELDS], --devices [FIELDS]
show all devices
-s [FIELDS], --hosts [FIELDS]
show all hosts
-T [FIELDS], --Targets [FIELDS]
show all the scsi targets
-f [FIELDS], --fcrports [FIELDS]
show all the FC rports
-i [FIELDS], --iscsi [FIELDS]
show iSCSI session details
-c [FIELDS], --commands [FIELDS]
show SCSI commands
-q [FIELDS], --queue [FIELDS]
show the IO requests, SCSI commands from request_queue
-r [FIELDS], --requests [FIELDS]
show requests to SCSI devices (INCOMPLETE)
-x, --hex display fields in hex
-j, --jiffies Show time in jiffies. Do not convert to seconds.
--check check for common SCSI issues
--time display time, in seconds, and state information for SCSI commands
--relative [RELATIVE]
show fields relative to the given value/symbol. Uses jiffies without argument
** Execution took 0.01s (real) 0.01s (CPU)
crash>
Show /proc/scsi/scsi style information (-p)¶
The '-p' option prints device information similar to /proc/scsi/scsi. It also displays more details about SCSI HBAs, with Scsi_Host pointer, PCI device path, and adapter status:
crash> scsishow -p
============================== Summary ===============================
-- 106 SCSI Devices, 29 Are Busy --
........................... Vendors/Types ............................
Vendor: LIO-ORG Model: block9 Rev: 4.0
Type: Direct-Access ANSI SCSI revision: 05
Vendor: LIO-ORG Model: block10 Rev: 4.0
Type: Direct-Access ANSI SCSI revision: 05
Vendor: ATA Model: WDC WD3200AAKS-7 Rev: 3E02
Type: Direct-Access ANSI SCSI revision: 05
Vendor: LIO-ORG Model: block11 Rev: 4.0
Type: Direct-Access ANSI SCSI revision: 05
Vendor: LIO-ORG Model: block12 Rev: 4.0
Type: Direct-Access ANSI SCSI revision: 05
Vendor: LIO-ORG Model: block13 Rev: 4.0
Type: Direct-Access ANSI SCSI revision: 05
Vendor: LIO-ORG Model: block22 Rev: 4.0
Type: Direct-Access ANSI SCSI revision: 05
[...]
============================= Scsi_Hosts==============================
*scsi0* <struct Scsi_Host 0xffff880211071000>
/devices/pci0000:00/0000:00:01.0/0000:01:00.0/host0
last_reset=0 host_busy=96 host_failed=96 host_eh_scheduled=0
shost_state=SHOST_RECOVERY
hostt: qla2xxx <struct scsi_qla_host 0xffff880211071740>
*scsi7* <struct Scsi_Host 0xffff880211076000>
/devices/pci0000:00/0000:00:01.0/0000:01:00.1/host7
last_reset=0 host_busy=26 host_failed=26 host_eh_scheduled=0
shost_state=SHOST_RECOVERY
hostt: qla2xxx <struct scsi_qla_host 0xffff880211076740>
*scsi3* <struct Scsi_Host 0xffff880211175800>
/devices/pci0000:00/0000:00:1f.2/ata3/host3
last_reset=0 host_busy=0 host_failed=0 host_eh_scheduled=0
shost_state=SHOST_RUNNING
hostt: ahci <shost_priv(shost) 0xffff880211175f40>
*scsi5* <struct Scsi_Host 0xffff880211176800>
/devices/pci0000:00/0000:00:1f.2/ata5/host5
last_reset=0 host_busy=0 host_failed=0 host_eh_scheduled=0
shost_state=SHOST_RUNNING
hostt: ahci <shost_priv(shost) 0xffff880211176f40>
[...]
Show all devices (-d)¶
The '-d' option can be used to get more detailed information about SCSI devices. It prints device details like scsi_device pointer, vendor/model, I/Os pending on the device, and error count:
crash> scsishow -d
========================================================================================================================================
HOST DRIVER
NAME NAME Scsi_Host shost_data hostdata
---------------------------------------------------------------------------------------------------
host2 ahci ffff91d4037ff000 0 ffff91d4037ffa58
DEV NAME scsi_device H:C:T:L VENDOR/MODEL DEVICE STATE IOREQ-CNT IODONE-CNT IOERR-CNT
----------------------------------------------------------------------------------------------------------------------------------------
sda ffff91d4a29f9000 2:0:0:0 ATA ST3500312CS SDEV_RUNNING 15408 14673 (735) 38
========================================================================================================================================
HOST DRIVER
NAME NAME Scsi_Host shost_data hostdata
---------------------------------------------------------------------------------------------------
host5 ahci ffff91d4a29fc000 0 ffff91d4a29fca58
DEV NAME scsi_device H:C:T:L VENDOR/MODEL DEVICE STATE IOREQ-CNT IODONE-CNT IOERR-CNT
----------------------------------------------------------------------------------------------------------------------------------------
sr0 ffff91da379dc000 5:0:0:0 ATAPI iHAS124 F SDEV_RUNNING 1528 1510 ( 18) 4
========================================================================================================================================
HOST DRIVER
NAME NAME Scsi_Host shost_data hostdata
---------------------------------------------------------------------------------------------------
host4 qla2xxx ffff91d405f18000 ffff91da3585f000 ffff91d405f18a58
DEV NAME scsi_device H:C:T:L VENDOR/MODEL DEVICE STATE IOREQ-CNT IODONE-CNT IOERR-CNT
----------------------------------------------------------------------------------------------------------------------------------------
sdb ffff91d4a2ba9000 4:0:0:0 LIO-ORG fclun0 SDEV_RUNNING 322 322 ( 0) 2
sdc ffff91d4a2baf000 4:0:0:9 LIO-ORG fclun9 SDEV_RUNNING 317 317 ( 0) 2
sdd ffff91d43a493000 4:0:0:8 LIO-ORG fclun8 SDEV_RUNNING 14330 14266 ( 64) 2
sde ffff91d43a492000 4:0:0:7 LIO-ORG fclun7 SDEV_RUNNING 319 319 ( 0) 2
sdf ffff91d4a2a29000 4:0:0:6 LIO-ORG fclun6 SDEV_RUNNING 324 324 ( 0) 2
sdg ffff91d4a2baa000 4:0:0:5 LIO-ORG fclun5 SDEV_RUNNING 318 318 ( 0) 2
sdh ffff91d4a1fea000 4:0:0:4 LIO-ORG fclun4 SDEV_RUNNING 322 322 ( 0) 2
sdi ffff91d4a2ae2000 4:0:0:3 LIO-ORG fclun3 SDEV_RUNNING 322 322 ( 0) 2
sdj ffff91d43a494000 4:0:0:2 LIO-ORG fclun2 SDEV_RUNNING 322 322 ( 0) 2
sdk ffff91d4a2a28000 4:0:0:1 LIO-ORG fclun1 SDEV_RUNNING 321 321 ( 0) 2
sdl ffff91d4a1fec000 4:0:1:0 LIO-ORG fclun0 SDEV_RUNNING 286 286 ( 0) 1
sdm ffff91d4a2a2e000 4:0:1:9 LIO-ORG fclun9 SDEV_RUNNING 280 280 ( 0) 1
[...]
========================================================================================================================================
HOST DRIVER
NAME NAME Scsi_Host shost_data hostdata
---------------------------------------------------------------------------------------------------
host7 qla2xxx ffff91d43a480000 ffff91d4a2bac000 ffff91d43a480a58
DEV NAME scsi_device H:C:T:L VENDOR/MODEL DEVICE STATE IOREQ-CNT IODONE-CNT IOERR-CNT
----------------------------------------------------------------------------------------------------------------------------------------
sdv ffff91d4a2ae1000 7:0:0:0 LIO-ORG fclun0 SDEV_RUNNING 281 281 ( 0) 1
sdw ffff91d4a3875000 7:0:0:9 LIO-ORG fclun9 SDEV_RUNNING 275 275 ( 0) 1
sdx ffff91d43a495000 7:0:0:8 LIO-ORG fclun8 SDEV_RUNNING 275 275 ( 0) 1
sdy ffff91d43a491000 7:0:0:7 LIO-ORG fclun7 SDEV_RUNNING 274 274 ( 0) 1
sdz ffff91d4a2a2f000 7:0:0:6 LIO-ORG fclun6 SDEV_RUNNING 275 275 ( 0) 1
sdaa ffff91da35886000 7:0:0:5 LIO-ORG fclun5 SDEV_RUNNING 275 275 ( 0) 1
sdab ffff91d4a41b9000 7:0:0:4 LIO-ORG fclun4 SDEV_RUNNING 280 280 ( 0) 1
sdac ffff91d4a41b8000 7:0:0:3 LIO-ORG fclun3 SDEV_RUNNING 280 280 ( 0) 1
sdad ffff91d4a41bf000 7:0:0:2 LIO-ORG fclun2 SDEV_RUNNING 280 280 ( 0) 1
sdae ffff91d4a3873000 7:0:0:1 LIO-ORG fclun1 SDEV_RUNNING 280 280 ( 0) 1
sdaf ffff91d4a3876000 7:0:1:0 LIO-ORG fclun0 SDEV_RUNNING 281 281 ( 0) 1
sdag ffff91d4a41be000 7:0:1:9 LIO-ORG fclun9 SDEV_RUNNING 274 274 ( 0) 1
sdah ffff91d4a1fed000 7:0:1:8 LIO-ORG fclun8 SDEV_RUNNING 276 276 ( 0) 1
sdai ffff91d4a1feb000 7:0:1:7 LIO-ORG fclun7 SDEV_RUNNING 276 276 ( 0) 1
[...]
Show all host bus adapters (optional -s option)¶
Running scsishow program with '-s' (or no options at all) displays the SCSI adapter details:
crash> scsishow
=========================================================================================================================
HOST DRIVER
NAME NAME Scsi_Host shost_data hostdata
---------------------------------------------------------------------------------------------------
host0 ahci ffff91da3593b000 0 ffff91da3593ba58
Driver version : 3.0
Taints bitmask : 0x0
host_busy : 0
host_blocked : 0
host_failed : 0
host_self_blocked : 0
shost_state : SHOST_RUNNING
eh_deadline : -1 (off, not supported by driver)
max_lun : 1
cmd_per_lun : 0
work_q_name :
nr_hw_queues : 0
[...]
=========================================================================================================================
HOST DRIVER
NAME NAME Scsi_Host shost_data hostdata
---------------------------------------------------------------------------------------------------
host4 qla2xxx ffff91d405f18000 ffff91da3585f000 ffff91d405f18a58
Driver version : 10.02.00.106-k
Taints bitmask : 0x0
host_busy : 64
host_blocked : 0
host_failed : 0
host_self_blocked : 0
shost_state : SHOST_RUNNING
eh_deadline : -1 (off)
max_lun : 65535
cmd_per_lun : 3
work_q_name : scsi_wq_4
nr_hw_queues : 0
=========================================================================================================================
HOST DRIVER
NAME NAME Scsi_Host shost_data hostdata
---------------------------------------------------------------------------------------------------
host7 qla2xxx ffff91d43a480000 ffff91d4a2bac000 ffff91d43a480a58
Driver version : 10.02.00.106-k
Taints bitmask : 0x0
host_busy : 0
host_blocked : 0
host_failed : 0
host_self_blocked : 0
shost_state : SHOST_RUNNING
eh_deadline : -1 (off)
max_lun : 65535
cmd_per_lun : 3
work_q_name : scsi_wq_7
nr_hw_queues : 0
*** NOTE: More detailed HBA information available, use '-v' or '--verbose' to view.
** Execution took 0.07s (real) 0.06s (CPU)
crash>
Using '-v' or '--verbose' will print even more detailed information, e.g. adapter model, PCI device slot, WWPN, WWNN (for FC HBAs), and firmware version. At this stage the verbose details are printed for HBAs managed by the lpfc, qla2xxx, hpsa, vmw_pvscsi, qedf, fnic, bfa, zfcp and iscsi_tcp drivers.
Example output for QLogic FC HBA:
=========================================================================================================================
HOST DRIVER
NAME NAME Scsi_Host shost_data hostdata
---------------------------------------------------------------------------------------------------
host7 qla2xxx ffff91d43a480000 ffff91d4a2bac000 ffff91d43a480a58
Driver version : 10.02.00.106-k
Taints bitmask : 0x0
host_busy : 0
host_blocked : 0
host_failed : 0
host_self_blocked : 0
shost_state : SHOST_RUNNING
eh_deadline : -1 (off)
max_lun : 65535
cmd_per_lun : 3
work_q_name : scsi_wq_7
nr_hw_queues : 0
FC/FCoE HBA attributes
----------------------
fc_host_attrs : ffff91d4a2bac000
node_name (wwnn) : 500143802428b543
port_name (wwpn) : 500143802428b542
QLogic HBA specific details
---------------------------
scsi_qla_host : ffff91d43a480a58
qla_hw_data : ffff91d405ef6000
pci_dev : ffff91da379df000
pci_dev slot : 0000:01:00.1
operating_mode : 1
model_desc : HP 8Gb Dual Channel PCI-e 2.0 FC HBA
optrom_state : 0
fw_major_version : 8
fw_minor_version : 7
fw_subminor_version : 0
fw_dumped : False
ql2xmaxqdepth : 64
** Execution took 0.48s (real) 0.48s (CPU)
crash>
For HP Smart Array RAID controller:
crash> scsishow -v
=========================================================================================================================
HOST DRIVER
NAME NAME Scsi_Host shost_data hostdata
---------------------------------------------------------------------------------------------------
host0 hpsa ffff88a80464e000 ffff88a807d5b660 ffff88a80464ea58
Driver version : 3.4.20-200-RH1
Taints bitmask : 0x0
host_busy : 0
host_blocked : 0
host_failed : 0
host_self_blocked : 0
shost_state : SHOST_RUNNING
eh_deadline : -1 (off, not supported by driver)
max_lun : 1024
cmd_per_lun : 1013
work_q_name :
nr_hw_queues : 0
HPSA HBA specific details
---------------------------
ctlr_info : ffff88a809688000
pci_dev : ffff88a804ad3000
pci_dev slot : 0000:07:00.0
devname : hpsa0
product_name : Smart Array P244br
board_id : 0x21bd103c
fwrev : 7.00B
CommandList : ffff88a75b500000
nr_cmds : 1024
max_commands : 1024
commands_outstanding: 0
interrupts_enabled : 1
intr_mode : 0
remove_in_progress : 0
reset_in_progress : 0
=========================================================================================================================
[...]
Show all SCSI targets (-T)¶
The '-T' option prints details of SCSI targets through which the local and SAN devices are connected:
crash> scsishow -T
===============================================================================================================
HOST DRIVER
NAME NAME Scsi_Host shost_data hostdata
---------------------------------------------------------------------------------------------------
host2 ahci ffff91d4037ff000 0 ffff91d4037ffa58
TARGET DEVICE scsi_target CHANNEL ID TARGET STATUS TARGET_BUSY TARGET_BLOCKED
--------------------------------------------------------------------------------------------------------
target2:0:0 ffff91d4a1302000 0 0 STARGET_RUNNING 0 0
===============================================================================================================
HOST DRIVER
NAME NAME Scsi_Host shost_data hostdata
---------------------------------------------------------------------------------------------------
host5 ahci ffff91d4a29fc000 0 ffff91d4a29fca58
TARGET DEVICE scsi_target CHANNEL ID TARGET STATUS TARGET_BUSY TARGET_BLOCKED
--------------------------------------------------------------------------------------------------------
target5:0:0 ffff91d4a1302800 0 0 STARGET_RUNNING 0 0
===============================================================================================================
HOST DRIVER
NAME NAME Scsi_Host shost_data hostdata
---------------------------------------------------------------------------------------------------
host4 qla2xxx ffff91d405f18000 ffff91da3585f000 ffff91d405f18a58
TARGET DEVICE scsi_target CHANNEL ID TARGET STATUS TARGET_BUSY TARGET_BLOCKED
--------------------------------------------------------------------------------------------------------
target4:0:0 ffff91d4a2a97800 0 0 STARGET_RUNNING 0 0
target4:0:1 ffff91d4a1fe3800 0 1 STARGET_RUNNING 0 0
===============================================================================================================
HOST DRIVER
NAME NAME Scsi_Host shost_data hostdata
---------------------------------------------------------------------------------------------------
host7 qla2xxx ffff91d43a480000 ffff91d4a2bac000 ffff91d43a480a58
TARGET DEVICE scsi_target CHANNEL ID TARGET STATUS TARGET_BUSY TARGET_BLOCKED
--------------------------------------------------------------------------------------------------------
target7:0:0 ffff91d4a38f0000 0 0 STARGET_RUNNING 0 0
target7:0:1 ffff91d4a4230800 0 1 STARGET_RUNNING 0 0
** Execution took 0.07s (real) 0.07s (CPU)
crash>
Show FC remote port details (-f)¶
To view more information about local and remote FC ports, use '-f'. It displays host HBA WWPN, WWNN, port_id, remote port state, and timeouts:
crash> scsishow -f
==========================================================================================================================================================
HOST DRIVER
NAME NAME Scsi_Host shost_data hostdata
---------------------------------------------------------------------------------------------------
host4 qla2xxx ffff91d405f18000 ffff91da3585f000 ffff91d405f18a58
TARGET DEVICE scsi_target fc_rport node_name port_name port_id port_state fast_io_fail_tmo dev_loss_tmo
----------------------------------------------------------------------------------------------------------------------------------------------------------
target4:0:0 ffff91d4a2a97800 ffff91d4a2a90000 20000024ff3de464 21000024ff3de464 0x6f1700 FC_PORTSTATE_ONLINE 5s 30s
target4:0:1 ffff91d4a1fe3800 ffff91d4a1c59000 20000024ff3de465 21000024ff3de465 0x6f1300 FC_PORTSTATE_ONLINE 5s 30s
==========================================================================================================================================================
HOST DRIVER
NAME NAME Scsi_Host shost_data hostdata
---------------------------------------------------------------------------------------------------
host7 qla2xxx ffff91d43a480000 ffff91d4a2bac000 ffff91d43a480a58
TARGET DEVICE scsi_target fc_rport node_name port_name port_id port_state fast_io_fail_tmo dev_loss_tmo
----------------------------------------------------------------------------------------------------------------------------------------------------------
target7:0:0 ffff91d4a38f0000 ffff91d4392db000 20000024ff3de464 21000024ff3de464 0x6f1700 FC_PORTSTATE_ONLINE 5s 30s
target7:0:1 ffff91d4a4230800 ffff91d4392de000 20000024ff3de465 21000024ff3de465 0x6f1300 FC_PORTSTATE_ONLINE 5s 30s
** Execution took 0.13s (real) 0.13s (CPU)
crash>
Show iSCSI session details (-i)¶
The '-i' option can be used to view iSCSI session details, e.g. IQN name, initiator and target IP, connection state, ping timeout, etc.:
crash> scsishow -i
==========================================================================================================================================================
HOST DRIVER
NAME NAME Scsi_Host shost_data hostdata
---------------------------------------------------------------------------------------------------
host3 iscsi_tcp ffff965397331000 ffff96538ba7b5c0 ffff965397331a58
iSCSI Initiator details
-----------------------
Initiator IQN : iqn.1994-05.com.redhat:24142eeaa5e6
iscsi_host : ffff965397331a58
iscsi_cls_host : ffff96538ba7b5c0
iscsi_sw_tcp_host : ffff965397331aa0
ifacename : default
n/w interface: : ens3
sw_tcp_conn.sock : ffff9653b9385b80
initiator/tgt socket: tcp 192.168.0.20:57114 192.168.0.10:3260 ESTABLISHED
iSCSI Target details
--------------------
Target IQN : iqn.2017-09.org.rhel8.target-2
targetalias : None
persistent_port : 3260
tgt persistent_addr : 192.168.0.10
Session specific details
------------------------
state : ISCSI_STATE_LOGGED_IN
iscsi_session : ffff965397336610
iscsi_cls_session : ffff965397336000
iscsi_conn : ffff965397332478
iscsi_cls_conn : ffff965397332000
iscsi_pool : ffff9653973367e0
cmdsn : 1668
exp_cmdsn : 1668
max_cmdsn : 1731
queued_cmdsn : 1668
abort_timeout : 15
lu_reset_timeout : 30
tgt_reset_timeout : 30
def_taskmgmt_tmo : 0
scsi_cmds_max : 113
cmds_max : 128
age : 0
last_ping : 5503597056
ping_timeout : 5
last_recv : 5503597199
recv_timeout : 5
eh_abort_cnt : 0
*** NOTE: Target 'persistent address' is the network address where iSCSI initiator send initial request.
When iSCSI redirection in use, this address might not be the network address used for actual
iSCSI transaction.
** Execution took 0.76s (real) 0.75s (CPU)
crash>
Show in-flight SCSI commands (-c)¶
Users can quickly get a list of in-flight SCSI commands pending on various devices using the '-c' option. It also prints the timestamp when the command was allocated (jiffies_at_alloc):
crash> scsishow -c
scsi_cmnd ffff91d4a28cf8b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297690987
scsi_cmnd ffff91d4a28df8b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297690941
scsi_cmnd ffff91d4a28d48b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691196
scsi_cmnd ffff91d4a28d5938 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691181
scsi_cmnd ffff91d4a28a5938 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691166
scsi_cmnd ffff91d4a28d69b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691409
scsi_cmnd ffff91d4a28a7a38 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691679
scsi_cmnd ffff91d4a28d8ab8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691055
scsi_cmnd ffff91d4a28a8ab8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691766
scsi_cmnd ffff91d4a28cd7b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691736
scsi_cmnd ffff91d4a28d9b38 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691759
scsi_cmnd ffff91d4a28dabb8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691498
scsi_cmnd ffff91d4a28e0c38 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691755
scsi_cmnd ffff91d4a28dbc38 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691739
scsi_cmnd ffff91d4a28d0c38 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297690976
scsi_cmnd ffff91d4a28e1cb8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297690996
scsi_cmnd ffff91d4a28dccb8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691451
[...]
Show commands queued in request queue of device (-q)¶
This option is similar to the '-c' option, but it prints even more detailed information, e.g. pointer to the associated request, bio struct, SCSI command opcode, command age, sector number, and I/O scheduler used by the device:
crash> scsishow -q
=======================================================================================================================
### DEVICE : sda
---------------------------------------------------------------------------------------
gendisk : ffff91d4a1307000 | scsi_device : ffff91d4a29f9000
request_queue : ffff91d4a2adaa40 | H:C:T:L : 2:0:0:0
elevator_name : mq-deadline | VENDOR/MODEL : ATA ST3500312CS
---------------------------------------------------------------------------------------
NO. request bio scsi_cmnd OPCODE COMMAND AGE SECTOR
-----------------------------------------------------------------------------------------------------------------
<<< NO I/O REQUESTS FOUND ON THE DEVICE! >>>
[...]
=======================================================================================================================
### DEVICE : sdd
---------------------------------------------------------------------------------------
gendisk : ffff91d40ade9000 | scsi_device : ffff91d43a493000
request_queue : ffff91d4a31d1520 | H:C:T:L : 4:0:0:8
elevator_name : mq-deadline | VENDOR/MODEL : LIO-ORG fclun8
---------------------------------------------------------------------------------------
NO. request bio scsi_cmnd OPCODE COMMAND AGE SECTOR
-----------------------------------------------------------------------------------------------------------------
1 ffff91d4a28cf790 ffff91d4d19c1490 ffff91d4a28cf8b8 WRITE(10) 0.863s 440936
2 ffff91d4a28df790 ffff91d4d19c1610 ffff91d4a28df8b8 WRITE(10) 0.909s 438376
3 ffff91d4a28d4790 ffff91d43851c9d0 ffff91d4a28d48b8 WRITE(10) 0.654s 371304
4 ffff91d4a28d5810 ffff91d405cf4f10 ffff91d4a28d5938 WRITE(10) 0.669s 367208
5 ffff91d4a28a5810 ffff91d4070afd90 ffff91d4a28a5938 WRITE(10) 0.684s 360552
6 ffff91d4a28d6890 ffff91d405cf4610 ffff91d4a28d69b8 WRITE(10) 0.441s 385128
7 ffff91d4a28a7910 ffff91d4070afb50 ffff91d4a28a7a38 WRITE(10) 0.171s 624640
8 ffff91d4a28d8990 ffff91d43851c010 ffff91d4a28d8ab8 WRITE(10) 0.795s 442472
9 ffff91d4a28a8990 ffff91d4070af910 ffff91d4a28a8ab8 READ(10) 0.084s 440936
10 ffff91d4a28cd690 ffff91d4d19c1910 ffff91d4a28cd7b8 WRITE(10) 0.114s 645120
11 ffff91d4a28d9a10 ffff91d4d19c1f10 ffff91d4a28d9b38 WRITE(10) 0.091s 657408
12 ffff91d4a28daa90 ffff91d405cf4790 ffff91d4a28dabb8 WRITE(10) 0.352s 391784
13 ffff91d4a28e0b10 ffff91d405cf43d0 ffff91d4a28e0c38 WRITE(10) 0.095s 645120
14 ffff91d4a28dbb10 ffff91d4d19c1310 ffff91d4a28dbc38 WRITE(10) 0.111s 642560
15 ffff91d4a28d0b10 ffff91d4d19c1e50 ffff91d4a28d0c38 WRITE(10) 0.874s 438376
[...]
Show requests to SCSI devices (-r)¶
The '-r' option can be used to list the I/O requests pending on various devices:
crash> scsishow -r
ffff91d4a28cf790 (4:0:0:8) timeout: 30000 deadline: 4297720987
ffff91d4a28df790 (4:0:0:8) timeout: 30000 deadline: 4297720941
ffff91d4a28d4790 (4:0:0:8) timeout: 30000 deadline: 4297721196
ffff91d4a28d5810 (4:0:0:8) timeout: 30000 deadline: 4297721181
ffff91d4a28a5810 (4:0:0:8) timeout: 30000 deadline: 4297721166
ffff91d4a28d6890 (4:0:0:8) timeout: 30000 deadline: 4297721409
ffff91d4a28a7910 (4:0:0:8) timeout: 30000 deadline: 4297721679
ffff91d4a28d8990 (4:0:0:8) timeout: 30000 deadline: 4297721055
ffff91d4a28a8990 (4:0:0:8) timeout: 30000 deadline: 4297721766
ffff91d4a28cd690 (4:0:0:8) timeout: 30000 deadline: 4297721736
ffff91d4a28d9a10 (4:0:0:8) timeout: 30000 deadline: 4297721759
[...]
To print the output in hex format, use '-x':
crash> scsishow -r -x
ffff91d4a28cf790 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a049b
ffff91d4a28df790 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a046d
ffff91d4a28d4790 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a056c
ffff91d4a28d5810 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a055d
ffff91d4a28a5810 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a054e
ffff91d4a28d6890 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a0641
ffff91d4a28a7910 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a074f
ffff91d4a28d8990 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a04df
ffff91d4a28a8990 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a07a6
ffff91d4a28cd690 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a0788
ffff91d4a28d9a10 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a079f
[...]
Check for common SCSI issues (--check)¶
The '--check' option runs built-in heuristics to review the status of SCSI adapters, devices, targets, and error handler threads to verify if there are any issues which could contribute to stalled I/Os:
crash> scsishow --check
WARNING: Scsi_Host 0xffff880211071000 (host0) is running error recovery!
WARNING: Scsi_Host 0xffff880211076000 (host7) is running error recovery!
Warning: scsi_cmnd 0xffff88009d796000 on scsi_device 0xffff880211003000 (0:0:0:0) older than its timeout: EH or stalled queue?
Warning: scsi_cmnd 0xffff8801c7a06fc0 on scsi_device 0xffff880211003000 (0:0:0:0) older than its timeout: EH or stalled queue?
Warning: scsi_cmnd 0xffff8801e029f340 on scsi_device 0xffff880211003000 (0:0:0:0) older than its timeout: EH or stalled queue?
[...]
Warning: scsi_cmnd 0xffff8800962fe380 on scsi_device 0xffff88003578d000 (0:0:0:16) older than its timeout: EH or stalled queue?
Warning: scsi_cmnd 0xffff8801e029efc0 on scsi_device 0xffff88003578d000 (0:0:0:16) older than its timeout: EH or stalled queue?
Error: cannot determine timeout!
Warning: scsi_cmnd 0xffff880034ed6c40 on scsi_device 0xffff880035f35000 (0:0:0:15) older than its timeout: EH or stalled queue?
Warning: scsi_cmnd 0xffff880034ed7500 on scsi_device 0xffff880035f34800 (0:0:0:14) older than its timeout: EH or stalled queue?
Warning: scsi_cmnd 0xffff880093f5ea80 on scsi_device 0xffff88003578f000 (0:0:0:13) older than its timeout: EH or stalled queue?
Warning: scsi_cmnd 0xffff8800982c8a80 on scsi_device 0xffff88003578f000 (0:0:0:13) older than its timeout: EH or stalled queue?
Warning: scsi_cmnd 0xffff8800982c96c0 on scsi_device 0xffff88003578f000 (0:0:0:13) older than its timeout: EH or stalled queue?
Warning: scsi_cmnd 0xffff8800982c8fc0 on scsi_device 0xffff88003578f000 (0:0:0:13) older than its timeout: EH or stalled queue?
Warning: scsi_cmnd 0xffff8800982c88c0 on scsi_device 0xffff88003578f000 (0:0:0:13) older than its timeout: EH or stalled queue?
Warning: scsi_cmnd 0xffff8800982c9a40 on scsi_device 0xffff88003578f000 (0:0:0:13) older than its timeout: EH or stalled queue?
Warning: scsi_cmnd 0xffff8801b9641180 on scsi_device 0xffff88003578f000 (0:0:0:13) older than its timeout: EH or stalled queue?
Warning: scsi_cmnd 0xffff8801b9640fc0 on scsi_device 0xffff88003578f000 (0:0:0:13) older than its timeout: EH or stalled queue?
[...]
Display time and state information for SCSI commands (--time)¶
The '--time' or '--t' options provide even more information about the in-flight I/O requests and SCSI commands. Along with command age, it also provides details about when the corresponding 'request' struct was allocated (rq-alloc):
crash> scsishow --time
scsi_cmnd ffff91d4a28cf8b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.137 (30) cmnd-alloc: -0.863 rq-alloc: -0.863
scsi_cmnd ffff91d4a28df8b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.091 (30) cmnd-alloc: -0.909 rq-alloc: -0.909
scsi_cmnd ffff91d4a28d48b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.346 (30) cmnd-alloc: -0.654 rq-alloc: -0.654
scsi_cmnd ffff91d4a28d5938 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.331 (30) cmnd-alloc: -0.669 rq-alloc: -0.669
scsi_cmnd ffff91d4a28a5938 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.316 (30) cmnd-alloc: -0.684 rq-alloc: -0.683
scsi_cmnd ffff91d4a28d69b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.559 (30) cmnd-alloc: -0.441 rq-alloc: -0.441
scsi_cmnd ffff91d4a28a7a38 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.829 (30) cmnd-alloc: -0.171 rq-alloc: -0.171
scsi_cmnd ffff91d4a28d8ab8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.205 (30) cmnd-alloc: -0.795 rq-alloc: -0.795
scsi_cmnd ffff91d4a28a8ab8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.916 (30) cmnd-alloc: -0.084 rq-alloc: -0.083
scsi_cmnd ffff91d4a28cd7b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.886 (30) cmnd-alloc: -0.114 rq-alloc: -0.114
scsi_cmnd ffff91d4a28d9b38 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.909 (30) cmnd-alloc: -0.091 rq-alloc: -0.091
scsi_cmnd ffff91d4a28dabb8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.648 (30) cmnd-alloc: -0.352 rq-alloc: -0.351
[...]
crash> scsishow --t
scsi_cmnd ffff91d4a28cf8b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.137 (30) cmnd-alloc: -0.863 rq-alloc: -0.863
scsi_cmnd ffff91d4a28df8b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.091 (30) cmnd-alloc: -0.909 rq-alloc: -0.909
scsi_cmnd ffff91d4a28d48b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.346 (30) cmnd-alloc: -0.654 rq-alloc: -0.654
scsi_cmnd ffff91d4a28d5938 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.331 (30) cmnd-alloc: -0.669 rq-alloc: -0.669
scsi_cmnd ffff91d4a28a5938 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.316 (30) cmnd-alloc: -0.684 rq-alloc: -0.683
scsi_cmnd ffff91d4a28d69b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.559 (30) cmnd-alloc: -0.441 rq-alloc: -0.441
scsi_cmnd ffff91d4a28a7a38 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.829 (30) cmnd-alloc: -0.171 rq-alloc: -0.171
scsi_cmnd ffff91d4a28d8ab8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.205 (30) cmnd-alloc: -0.795 rq-alloc: -0.795
scsi_cmnd ffff91d4a28a8ab8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.916 (30) cmnd-alloc: -0.084 rq-alloc: -0.083
scsi_cmnd ffff91d4a28cd7b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.886 (30) cmnd-alloc: -0.114 rq-alloc: -0.114
scsi_cmnd ffff91d4a28d9b38 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.909 (30) cmnd-alloc: -0.091 rq-alloc: -0.091
scsi_cmnd ffff91d4a28dabb8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.648 (30) cmnd-alloc: -0.352 rq-alloc: -0.351
[...]
crash> scsishow --t -j
scsi_cmnd ffff91d4a28cf8b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29137 (30000) cmnd-alloc: -863 rq-alloc: -863
scsi_cmnd ffff91d4a28df8b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29091 (30000) cmnd-alloc: -909 rq-alloc: -909
scsi_cmnd ffff91d4a28d48b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29346 (30000) cmnd-alloc: -654 rq-alloc: -654
scsi_cmnd ffff91d4a28d5938 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29331 (30000) cmnd-alloc: -669 rq-alloc: -669
scsi_cmnd ffff91d4a28a5938 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29316 (30000) cmnd-alloc: -684 rq-alloc: -684
scsi_cmnd ffff91d4a28d69b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29559 (30000) cmnd-alloc: -441 rq-alloc: -441
scsi_cmnd ffff91d4a28a7a38 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29829 (30000) cmnd-alloc: -171 rq-alloc: -171
scsi_cmnd ffff91d4a28d8ab8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29205 (30000) cmnd-alloc: -795 rq-alloc: -795
scsi_cmnd ffff91d4a28a8ab8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29916 (30000) cmnd-alloc: -84 rq-alloc: -84
scsi_cmnd ffff91d4a28cd7b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29886 (30000) cmnd-alloc: -114 rq-alloc: -114
scsi_cmnd ffff91d4a28d9b38 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29909 (30000) cmnd-alloc: -91 rq-alloc: -91
scsi_cmnd ffff91d4a28dabb8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29648 (30000) cmnd-alloc: -352 rq-alloc: -352
[...]