Processing SCSI subsystem details (scsishow) ============================================ The scsishow program available in the mpykdump extension allows users to quickly fetch detailed information about SCSI HBAs, FC, iSCSI targets, SCSI devices, and SCSI commands. It also provides the '--check' option which can detect stalled error handling process, stuck I/O requests or SCSI commands, and other common I/O issues. Options provided by ‘scsishow’:: crash> scsishow -h usage: scsishow [-h] [-v] [-p] [-d [FIELDS]] [-s [FIELDS]] [-T [FIELDS]] [-f [FIELDS]] [-i [FIELDS]] [-c [FIELDS]] [-q [FIELDS]] [-r [FIELDS]] [-x] [-j] [--check] [--time] [--relative [RELATIVE]] optional arguments: -h, --help show this help message and exit -v, --verbose verbose output -p, --proc show /proc/scsi/scsi style information -d [FIELDS], --devices [FIELDS] show all devices -s [FIELDS], --hosts [FIELDS] show all hosts -T [FIELDS], --Targets [FIELDS] show all the scsi targets -f [FIELDS], --fcrports [FIELDS] show all the FC rports -i [FIELDS], --iscsi [FIELDS] show iSCSI session details -c [FIELDS], --commands [FIELDS] show SCSI commands -q [FIELDS], --queue [FIELDS] show the IO requests, SCSI commands from request_queue -r [FIELDS], --requests [FIELDS] show requests to SCSI devices (INCOMPLETE) -x, --hex display fields in hex -j, --jiffies Show time in jiffies. Do not convert to seconds. --check check for common SCSI issues --time display time, in seconds, and state information for SCSI commands --relative [RELATIVE] show fields relative to the given value/symbol. Uses jiffies without argument ** Execution took 0.01s (real) 0.01s (CPU) crash> * `Show /proc/scsi/scsi style information (-p)`_ * `Show all devices (-d)`_ * `Show all host bus adapters (optional -s option)`_ * `Show all SCSI targets (-T)`_ * `Show FC remote port details (-f)`_ * `Show iSCSI session details (-i)`_ * `Show in-flight SCSI commands (-c)`_ * `Show commands queued in request queue of device (-q)`_ * `Show requests to SCSI devices (-r)`_ * `Check for common SCSI issues (--check)`_ * `Display time and state information for SCSI commands (--time)`_ Show /proc/scsi/scsi style information (-p) ------------------------------------------- The '-p' option prints device information similar to /proc/scsi/scsi. It also displays more details about SCSI HBAs, with Scsi_Host pointer, PCI device path, and adapter status:: crash> scsishow -p ============================== Summary =============================== -- 106 SCSI Devices, 29 Are Busy -- ........................... Vendors/Types ............................ Vendor: LIO-ORG Model: block9 Rev: 4.0 Type: Direct-Access ANSI SCSI revision: 05 Vendor: LIO-ORG Model: block10 Rev: 4.0 Type: Direct-Access ANSI SCSI revision: 05 Vendor: ATA Model: WDC WD3200AAKS-7 Rev: 3E02 Type: Direct-Access ANSI SCSI revision: 05 Vendor: LIO-ORG Model: block11 Rev: 4.0 Type: Direct-Access ANSI SCSI revision: 05 Vendor: LIO-ORG Model: block12 Rev: 4.0 Type: Direct-Access ANSI SCSI revision: 05 Vendor: LIO-ORG Model: block13 Rev: 4.0 Type: Direct-Access ANSI SCSI revision: 05 Vendor: LIO-ORG Model: block22 Rev: 4.0 Type: Direct-Access ANSI SCSI revision: 05 [...] ============================= Scsi_Hosts============================== *scsi0* /devices/pci0000:00/0000:00:01.0/0000:01:00.0/host0 last_reset=0 host_busy=96 host_failed=96 host_eh_scheduled=0 shost_state=SHOST_RECOVERY hostt: qla2xxx *scsi7* /devices/pci0000:00/0000:00:01.0/0000:01:00.1/host7 last_reset=0 host_busy=26 host_failed=26 host_eh_scheduled=0 shost_state=SHOST_RECOVERY hostt: qla2xxx *scsi3* /devices/pci0000:00/0000:00:1f.2/ata3/host3 last_reset=0 host_busy=0 host_failed=0 host_eh_scheduled=0 shost_state=SHOST_RUNNING hostt: ahci *scsi5* /devices/pci0000:00/0000:00:1f.2/ata5/host5 last_reset=0 host_busy=0 host_failed=0 host_eh_scheduled=0 shost_state=SHOST_RUNNING hostt: ahci [...] Show all devices (-d) --------------------- The '-d' option can be used to get more detailed information about SCSI devices. It prints device details like scsi_device pointer, vendor/model, I/Os pending on the device, and error count:: crash> scsishow -d ======================================================================================================================================== HOST DRIVER NAME NAME Scsi_Host shost_data hostdata --------------------------------------------------------------------------------------------------- host2 ahci ffff91d4037ff000 0 ffff91d4037ffa58 DEV NAME scsi_device H:C:T:L VENDOR/MODEL DEVICE STATE IOREQ-CNT IODONE-CNT IOERR-CNT ---------------------------------------------------------------------------------------------------------------------------------------- sda ffff91d4a29f9000 2:0:0:0 ATA ST3500312CS SDEV_RUNNING 15408 14673 (735) 38 ======================================================================================================================================== HOST DRIVER NAME NAME Scsi_Host shost_data hostdata --------------------------------------------------------------------------------------------------- host5 ahci ffff91d4a29fc000 0 ffff91d4a29fca58 DEV NAME scsi_device H:C:T:L VENDOR/MODEL DEVICE STATE IOREQ-CNT IODONE-CNT IOERR-CNT ---------------------------------------------------------------------------------------------------------------------------------------- sr0 ffff91da379dc000 5:0:0:0 ATAPI iHAS124 F SDEV_RUNNING 1528 1510 ( 18) 4 ======================================================================================================================================== HOST DRIVER NAME NAME Scsi_Host shost_data hostdata --------------------------------------------------------------------------------------------------- host4 qla2xxx ffff91d405f18000 ffff91da3585f000 ffff91d405f18a58 DEV NAME scsi_device H:C:T:L VENDOR/MODEL DEVICE STATE IOREQ-CNT IODONE-CNT IOERR-CNT ---------------------------------------------------------------------------------------------------------------------------------------- sdb ffff91d4a2ba9000 4:0:0:0 LIO-ORG fclun0 SDEV_RUNNING 322 322 ( 0) 2 sdc ffff91d4a2baf000 4:0:0:9 LIO-ORG fclun9 SDEV_RUNNING 317 317 ( 0) 2 sdd ffff91d43a493000 4:0:0:8 LIO-ORG fclun8 SDEV_RUNNING 14330 14266 ( 64) 2 sde ffff91d43a492000 4:0:0:7 LIO-ORG fclun7 SDEV_RUNNING 319 319 ( 0) 2 sdf ffff91d4a2a29000 4:0:0:6 LIO-ORG fclun6 SDEV_RUNNING 324 324 ( 0) 2 sdg ffff91d4a2baa000 4:0:0:5 LIO-ORG fclun5 SDEV_RUNNING 318 318 ( 0) 2 sdh ffff91d4a1fea000 4:0:0:4 LIO-ORG fclun4 SDEV_RUNNING 322 322 ( 0) 2 sdi ffff91d4a2ae2000 4:0:0:3 LIO-ORG fclun3 SDEV_RUNNING 322 322 ( 0) 2 sdj ffff91d43a494000 4:0:0:2 LIO-ORG fclun2 SDEV_RUNNING 322 322 ( 0) 2 sdk ffff91d4a2a28000 4:0:0:1 LIO-ORG fclun1 SDEV_RUNNING 321 321 ( 0) 2 sdl ffff91d4a1fec000 4:0:1:0 LIO-ORG fclun0 SDEV_RUNNING 286 286 ( 0) 1 sdm ffff91d4a2a2e000 4:0:1:9 LIO-ORG fclun9 SDEV_RUNNING 280 280 ( 0) 1 [...] ======================================================================================================================================== HOST DRIVER NAME NAME Scsi_Host shost_data hostdata --------------------------------------------------------------------------------------------------- host7 qla2xxx ffff91d43a480000 ffff91d4a2bac000 ffff91d43a480a58 DEV NAME scsi_device H:C:T:L VENDOR/MODEL DEVICE STATE IOREQ-CNT IODONE-CNT IOERR-CNT ---------------------------------------------------------------------------------------------------------------------------------------- sdv ffff91d4a2ae1000 7:0:0:0 LIO-ORG fclun0 SDEV_RUNNING 281 281 ( 0) 1 sdw ffff91d4a3875000 7:0:0:9 LIO-ORG fclun9 SDEV_RUNNING 275 275 ( 0) 1 sdx ffff91d43a495000 7:0:0:8 LIO-ORG fclun8 SDEV_RUNNING 275 275 ( 0) 1 sdy ffff91d43a491000 7:0:0:7 LIO-ORG fclun7 SDEV_RUNNING 274 274 ( 0) 1 sdz ffff91d4a2a2f000 7:0:0:6 LIO-ORG fclun6 SDEV_RUNNING 275 275 ( 0) 1 sdaa ffff91da35886000 7:0:0:5 LIO-ORG fclun5 SDEV_RUNNING 275 275 ( 0) 1 sdab ffff91d4a41b9000 7:0:0:4 LIO-ORG fclun4 SDEV_RUNNING 280 280 ( 0) 1 sdac ffff91d4a41b8000 7:0:0:3 LIO-ORG fclun3 SDEV_RUNNING 280 280 ( 0) 1 sdad ffff91d4a41bf000 7:0:0:2 LIO-ORG fclun2 SDEV_RUNNING 280 280 ( 0) 1 sdae ffff91d4a3873000 7:0:0:1 LIO-ORG fclun1 SDEV_RUNNING 280 280 ( 0) 1 sdaf ffff91d4a3876000 7:0:1:0 LIO-ORG fclun0 SDEV_RUNNING 281 281 ( 0) 1 sdag ffff91d4a41be000 7:0:1:9 LIO-ORG fclun9 SDEV_RUNNING 274 274 ( 0) 1 sdah ffff91d4a1fed000 7:0:1:8 LIO-ORG fclun8 SDEV_RUNNING 276 276 ( 0) 1 sdai ffff91d4a1feb000 7:0:1:7 LIO-ORG fclun7 SDEV_RUNNING 276 276 ( 0) 1 [...] Show all host bus adapters (optional -s option) ----------------------------------------------- Running scsishow program with '-s' (or no options at all) displays the SCSI adapter details:: crash> scsishow ========================================================================================================================= HOST DRIVER NAME NAME Scsi_Host shost_data hostdata --------------------------------------------------------------------------------------------------- host0 ahci ffff91da3593b000 0 ffff91da3593ba58 Driver version : 3.0 Taints bitmask : 0x0 host_busy : 0 host_blocked : 0 host_failed : 0 host_self_blocked : 0 shost_state : SHOST_RUNNING eh_deadline : -1 (off, not supported by driver) max_lun : 1 cmd_per_lun : 0 work_q_name : nr_hw_queues : 0 [...] ========================================================================================================================= HOST DRIVER NAME NAME Scsi_Host shost_data hostdata --------------------------------------------------------------------------------------------------- host4 qla2xxx ffff91d405f18000 ffff91da3585f000 ffff91d405f18a58 Driver version : 10.02.00.106-k Taints bitmask : 0x0 host_busy : 64 host_blocked : 0 host_failed : 0 host_self_blocked : 0 shost_state : SHOST_RUNNING eh_deadline : -1 (off) max_lun : 65535 cmd_per_lun : 3 work_q_name : scsi_wq_4 nr_hw_queues : 0 ========================================================================================================================= HOST DRIVER NAME NAME Scsi_Host shost_data hostdata --------------------------------------------------------------------------------------------------- host7 qla2xxx ffff91d43a480000 ffff91d4a2bac000 ffff91d43a480a58 Driver version : 10.02.00.106-k Taints bitmask : 0x0 host_busy : 0 host_blocked : 0 host_failed : 0 host_self_blocked : 0 shost_state : SHOST_RUNNING eh_deadline : -1 (off) max_lun : 65535 cmd_per_lun : 3 work_q_name : scsi_wq_7 nr_hw_queues : 0 *** NOTE: More detailed HBA information available, use '-v' or '--verbose' to view. ** Execution took 0.07s (real) 0.06s (CPU) crash> Using '-v' or '--verbose' will print even more detailed information, e.g. adapter model, PCI device slot, WWPN, WWNN (for FC HBAs), and firmware version. At this stage the verbose details are printed for HBAs managed by the lpfc, qla2xxx, hpsa, vmw_pvscsi, qedf, fnic, bfa, zfcp and iscsi_tcp drivers. Example output for QLogic FC HBA:: ========================================================================================================================= HOST DRIVER NAME NAME Scsi_Host shost_data hostdata --------------------------------------------------------------------------------------------------- host7 qla2xxx ffff91d43a480000 ffff91d4a2bac000 ffff91d43a480a58 Driver version : 10.02.00.106-k Taints bitmask : 0x0 host_busy : 0 host_blocked : 0 host_failed : 0 host_self_blocked : 0 shost_state : SHOST_RUNNING eh_deadline : -1 (off) max_lun : 65535 cmd_per_lun : 3 work_q_name : scsi_wq_7 nr_hw_queues : 0 FC/FCoE HBA attributes ---------------------- fc_host_attrs : ffff91d4a2bac000 node_name (wwnn) : 500143802428b543 port_name (wwpn) : 500143802428b542 QLogic HBA specific details --------------------------- scsi_qla_host : ffff91d43a480a58 qla_hw_data : ffff91d405ef6000 pci_dev : ffff91da379df000 pci_dev slot : 0000:01:00.1 operating_mode : 1 model_desc : HP 8Gb Dual Channel PCI-e 2.0 FC HBA optrom_state : 0 fw_major_version : 8 fw_minor_version : 7 fw_subminor_version : 0 fw_dumped : False ql2xmaxqdepth : 64 ** Execution took 0.48s (real) 0.48s (CPU) crash> For HP Smart Array RAID controller:: crash> scsishow -v ========================================================================================================================= HOST DRIVER NAME NAME Scsi_Host shost_data hostdata --------------------------------------------------------------------------------------------------- host0 hpsa ffff88a80464e000 ffff88a807d5b660 ffff88a80464ea58 Driver version : 3.4.20-200-RH1 Taints bitmask : 0x0 host_busy : 0 host_blocked : 0 host_failed : 0 host_self_blocked : 0 shost_state : SHOST_RUNNING eh_deadline : -1 (off, not supported by driver) max_lun : 1024 cmd_per_lun : 1013 work_q_name : nr_hw_queues : 0 HPSA HBA specific details --------------------------- ctlr_info : ffff88a809688000 pci_dev : ffff88a804ad3000 pci_dev slot : 0000:07:00.0 devname : hpsa0 product_name : Smart Array P244br board_id : 0x21bd103c fwrev : 7.00B CommandList : ffff88a75b500000 nr_cmds : 1024 max_commands : 1024 commands_outstanding: 0 interrupts_enabled : 1 intr_mode : 0 remove_in_progress : 0 reset_in_progress : 0 ========================================================================================================================= [...] Show all SCSI targets (-T) -------------------------- The '-T' option prints details of SCSI targets through which the local and SAN devices are connected:: crash> scsishow -T =============================================================================================================== HOST DRIVER NAME NAME Scsi_Host shost_data hostdata --------------------------------------------------------------------------------------------------- host2 ahci ffff91d4037ff000 0 ffff91d4037ffa58 TARGET DEVICE scsi_target CHANNEL ID TARGET STATUS TARGET_BUSY TARGET_BLOCKED -------------------------------------------------------------------------------------------------------- target2:0:0 ffff91d4a1302000 0 0 STARGET_RUNNING 0 0 =============================================================================================================== HOST DRIVER NAME NAME Scsi_Host shost_data hostdata --------------------------------------------------------------------------------------------------- host5 ahci ffff91d4a29fc000 0 ffff91d4a29fca58 TARGET DEVICE scsi_target CHANNEL ID TARGET STATUS TARGET_BUSY TARGET_BLOCKED -------------------------------------------------------------------------------------------------------- target5:0:0 ffff91d4a1302800 0 0 STARGET_RUNNING 0 0 =============================================================================================================== HOST DRIVER NAME NAME Scsi_Host shost_data hostdata --------------------------------------------------------------------------------------------------- host4 qla2xxx ffff91d405f18000 ffff91da3585f000 ffff91d405f18a58 TARGET DEVICE scsi_target CHANNEL ID TARGET STATUS TARGET_BUSY TARGET_BLOCKED -------------------------------------------------------------------------------------------------------- target4:0:0 ffff91d4a2a97800 0 0 STARGET_RUNNING 0 0 target4:0:1 ffff91d4a1fe3800 0 1 STARGET_RUNNING 0 0 =============================================================================================================== HOST DRIVER NAME NAME Scsi_Host shost_data hostdata --------------------------------------------------------------------------------------------------- host7 qla2xxx ffff91d43a480000 ffff91d4a2bac000 ffff91d43a480a58 TARGET DEVICE scsi_target CHANNEL ID TARGET STATUS TARGET_BUSY TARGET_BLOCKED -------------------------------------------------------------------------------------------------------- target7:0:0 ffff91d4a38f0000 0 0 STARGET_RUNNING 0 0 target7:0:1 ffff91d4a4230800 0 1 STARGET_RUNNING 0 0 ** Execution took 0.07s (real) 0.07s (CPU) crash> Show FC remote port details (-f) -------------------------------- To view more information about local and remote FC ports, use '-f'. It displays host HBA WWPN, WWNN, port_id, remote port state, and timeouts:: crash> scsishow -f ========================================================================================================================================================== HOST DRIVER NAME NAME Scsi_Host shost_data hostdata --------------------------------------------------------------------------------------------------- host4 qla2xxx ffff91d405f18000 ffff91da3585f000 ffff91d405f18a58 TARGET DEVICE scsi_target fc_rport node_name port_name port_id port_state fast_io_fail_tmo dev_loss_tmo ---------------------------------------------------------------------------------------------------------------------------------------------------------- target4:0:0 ffff91d4a2a97800 ffff91d4a2a90000 20000024ff3de464 21000024ff3de464 0x6f1700 FC_PORTSTATE_ONLINE 5s 30s target4:0:1 ffff91d4a1fe3800 ffff91d4a1c59000 20000024ff3de465 21000024ff3de465 0x6f1300 FC_PORTSTATE_ONLINE 5s 30s ========================================================================================================================================================== HOST DRIVER NAME NAME Scsi_Host shost_data hostdata --------------------------------------------------------------------------------------------------- host7 qla2xxx ffff91d43a480000 ffff91d4a2bac000 ffff91d43a480a58 TARGET DEVICE scsi_target fc_rport node_name port_name port_id port_state fast_io_fail_tmo dev_loss_tmo ---------------------------------------------------------------------------------------------------------------------------------------------------------- target7:0:0 ffff91d4a38f0000 ffff91d4392db000 20000024ff3de464 21000024ff3de464 0x6f1700 FC_PORTSTATE_ONLINE 5s 30s target7:0:1 ffff91d4a4230800 ffff91d4392de000 20000024ff3de465 21000024ff3de465 0x6f1300 FC_PORTSTATE_ONLINE 5s 30s ** Execution took 0.13s (real) 0.13s (CPU) crash> Show iSCSI session details (-i) ------------------------------- The '-i' option can be used to view iSCSI session details, e.g. IQN name, initiator and target IP, connection state, ping timeout, etc.:: crash> scsishow -i ========================================================================================================================================================== HOST DRIVER NAME NAME Scsi_Host shost_data hostdata --------------------------------------------------------------------------------------------------- host3 iscsi_tcp ffff965397331000 ffff96538ba7b5c0 ffff965397331a58 iSCSI Initiator details ----------------------- Initiator IQN : iqn.1994-05.com.redhat:24142eeaa5e6 iscsi_host : ffff965397331a58 iscsi_cls_host : ffff96538ba7b5c0 iscsi_sw_tcp_host : ffff965397331aa0 ifacename : default n/w interface: : ens3 sw_tcp_conn.sock : ffff9653b9385b80 initiator/tgt socket: tcp 192.168.0.20:57114 192.168.0.10:3260 ESTABLISHED iSCSI Target details -------------------- Target IQN : iqn.2017-09.org.rhel8.target-2 targetalias : None persistent_port : 3260 tgt persistent_addr : 192.168.0.10 Session specific details ------------------------ state : ISCSI_STATE_LOGGED_IN iscsi_session : ffff965397336610 iscsi_cls_session : ffff965397336000 iscsi_conn : ffff965397332478 iscsi_cls_conn : ffff965397332000 iscsi_pool : ffff9653973367e0 cmdsn : 1668 exp_cmdsn : 1668 max_cmdsn : 1731 queued_cmdsn : 1668 abort_timeout : 15 lu_reset_timeout : 30 tgt_reset_timeout : 30 def_taskmgmt_tmo : 0 scsi_cmds_max : 113 cmds_max : 128 age : 0 last_ping : 5503597056 ping_timeout : 5 last_recv : 5503597199 recv_timeout : 5 eh_abort_cnt : 0 *** NOTE: Target 'persistent address' is the network address where iSCSI initiator send initial request. When iSCSI redirection in use, this address might not be the network address used for actual iSCSI transaction. ** Execution took 0.76s (real) 0.75s (CPU) crash> Show in-flight SCSI commands (-c) --------------------------------- Users can quickly get a list of in-flight SCSI commands pending on various devices using the '-c' option. It also prints the timestamp when the command was allocated (jiffies_at_alloc):: crash> scsishow -c scsi_cmnd ffff91d4a28cf8b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297690987 scsi_cmnd ffff91d4a28df8b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297690941 scsi_cmnd ffff91d4a28d48b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691196 scsi_cmnd ffff91d4a28d5938 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691181 scsi_cmnd ffff91d4a28a5938 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691166 scsi_cmnd ffff91d4a28d69b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691409 scsi_cmnd ffff91d4a28a7a38 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691679 scsi_cmnd ffff91d4a28d8ab8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691055 scsi_cmnd ffff91d4a28a8ab8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691766 scsi_cmnd ffff91d4a28cd7b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691736 scsi_cmnd ffff91d4a28d9b38 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691759 scsi_cmnd ffff91d4a28dabb8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691498 scsi_cmnd ffff91d4a28e0c38 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691755 scsi_cmnd ffff91d4a28dbc38 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691739 scsi_cmnd ffff91d4a28d0c38 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297690976 scsi_cmnd ffff91d4a28e1cb8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297690996 scsi_cmnd ffff91d4a28dccb8 on scsi_device 0xffff91d43a493000 (4:0:0:8) jiffies_at_alloc: 4297691451 [...] Show commands queued in request queue of device (-q) ---------------------------------------------------- This option is similar to the '-c' option, but it prints even more detailed information, e.g. pointer to the associated request, bio struct, SCSI command opcode, command age, sector number, and I/O scheduler used by the device:: crash> scsishow -q ======================================================================================================================= ### DEVICE : sda --------------------------------------------------------------------------------------- gendisk : ffff91d4a1307000 | scsi_device : ffff91d4a29f9000 request_queue : ffff91d4a2adaa40 | H:C:T:L : 2:0:0:0 elevator_name : mq-deadline | VENDOR/MODEL : ATA ST3500312CS --------------------------------------------------------------------------------------- NO. request bio scsi_cmnd OPCODE COMMAND AGE SECTOR ----------------------------------------------------------------------------------------------------------------- <<< NO I/O REQUESTS FOUND ON THE DEVICE! >>> [...] ======================================================================================================================= ### DEVICE : sdd --------------------------------------------------------------------------------------- gendisk : ffff91d40ade9000 | scsi_device : ffff91d43a493000 request_queue : ffff91d4a31d1520 | H:C:T:L : 4:0:0:8 elevator_name : mq-deadline | VENDOR/MODEL : LIO-ORG fclun8 --------------------------------------------------------------------------------------- NO. request bio scsi_cmnd OPCODE COMMAND AGE SECTOR ----------------------------------------------------------------------------------------------------------------- 1 ffff91d4a28cf790 ffff91d4d19c1490 ffff91d4a28cf8b8 WRITE(10) 0.863s 440936 2 ffff91d4a28df790 ffff91d4d19c1610 ffff91d4a28df8b8 WRITE(10) 0.909s 438376 3 ffff91d4a28d4790 ffff91d43851c9d0 ffff91d4a28d48b8 WRITE(10) 0.654s 371304 4 ffff91d4a28d5810 ffff91d405cf4f10 ffff91d4a28d5938 WRITE(10) 0.669s 367208 5 ffff91d4a28a5810 ffff91d4070afd90 ffff91d4a28a5938 WRITE(10) 0.684s 360552 6 ffff91d4a28d6890 ffff91d405cf4610 ffff91d4a28d69b8 WRITE(10) 0.441s 385128 7 ffff91d4a28a7910 ffff91d4070afb50 ffff91d4a28a7a38 WRITE(10) 0.171s 624640 8 ffff91d4a28d8990 ffff91d43851c010 ffff91d4a28d8ab8 WRITE(10) 0.795s 442472 9 ffff91d4a28a8990 ffff91d4070af910 ffff91d4a28a8ab8 READ(10) 0.084s 440936 10 ffff91d4a28cd690 ffff91d4d19c1910 ffff91d4a28cd7b8 WRITE(10) 0.114s 645120 11 ffff91d4a28d9a10 ffff91d4d19c1f10 ffff91d4a28d9b38 WRITE(10) 0.091s 657408 12 ffff91d4a28daa90 ffff91d405cf4790 ffff91d4a28dabb8 WRITE(10) 0.352s 391784 13 ffff91d4a28e0b10 ffff91d405cf43d0 ffff91d4a28e0c38 WRITE(10) 0.095s 645120 14 ffff91d4a28dbb10 ffff91d4d19c1310 ffff91d4a28dbc38 WRITE(10) 0.111s 642560 15 ffff91d4a28d0b10 ffff91d4d19c1e50 ffff91d4a28d0c38 WRITE(10) 0.874s 438376 [...] Show requests to SCSI devices (-r) ---------------------------------- The '-r' option can be used to list the I/O requests pending on various devices:: crash> scsishow -r ffff91d4a28cf790 (4:0:0:8) timeout: 30000 deadline: 4297720987 ffff91d4a28df790 (4:0:0:8) timeout: 30000 deadline: 4297720941 ffff91d4a28d4790 (4:0:0:8) timeout: 30000 deadline: 4297721196 ffff91d4a28d5810 (4:0:0:8) timeout: 30000 deadline: 4297721181 ffff91d4a28a5810 (4:0:0:8) timeout: 30000 deadline: 4297721166 ffff91d4a28d6890 (4:0:0:8) timeout: 30000 deadline: 4297721409 ffff91d4a28a7910 (4:0:0:8) timeout: 30000 deadline: 4297721679 ffff91d4a28d8990 (4:0:0:8) timeout: 30000 deadline: 4297721055 ffff91d4a28a8990 (4:0:0:8) timeout: 30000 deadline: 4297721766 ffff91d4a28cd690 (4:0:0:8) timeout: 30000 deadline: 4297721736 ffff91d4a28d9a10 (4:0:0:8) timeout: 30000 deadline: 4297721759 [...] To print the output in hex format, use '-x':: crash> scsishow -r -x ffff91d4a28cf790 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a049b ffff91d4a28df790 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a046d ffff91d4a28d4790 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a056c ffff91d4a28d5810 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a055d ffff91d4a28a5810 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a054e ffff91d4a28d6890 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a0641 ffff91d4a28a7910 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a074f ffff91d4a28d8990 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a04df ffff91d4a28a8990 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a07a6 ffff91d4a28cd690 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a0788 ffff91d4a28d9a10 (4:0:0:8) timeout: 0x7530 deadline: 0x1002a079f [...] Check for common SCSI issues (--check) ---------------------------------------- The '--check' option runs built-in heuristics to review the status of SCSI adapters, devices, targets, and error handler threads to verify if there are any issues which could contribute to stalled I/Os:: crash> scsishow --check WARNING: Scsi_Host 0xffff880211071000 (host0) is running error recovery! WARNING: Scsi_Host 0xffff880211076000 (host7) is running error recovery! Warning: scsi_cmnd 0xffff88009d796000 on scsi_device 0xffff880211003000 (0:0:0:0) older than its timeout: EH or stalled queue? Warning: scsi_cmnd 0xffff8801c7a06fc0 on scsi_device 0xffff880211003000 (0:0:0:0) older than its timeout: EH or stalled queue? Warning: scsi_cmnd 0xffff8801e029f340 on scsi_device 0xffff880211003000 (0:0:0:0) older than its timeout: EH or stalled queue? [...] Warning: scsi_cmnd 0xffff8800962fe380 on scsi_device 0xffff88003578d000 (0:0:0:16) older than its timeout: EH or stalled queue? Warning: scsi_cmnd 0xffff8801e029efc0 on scsi_device 0xffff88003578d000 (0:0:0:16) older than its timeout: EH or stalled queue? Error: cannot determine timeout! Warning: scsi_cmnd 0xffff880034ed6c40 on scsi_device 0xffff880035f35000 (0:0:0:15) older than its timeout: EH or stalled queue? Warning: scsi_cmnd 0xffff880034ed7500 on scsi_device 0xffff880035f34800 (0:0:0:14) older than its timeout: EH or stalled queue? Warning: scsi_cmnd 0xffff880093f5ea80 on scsi_device 0xffff88003578f000 (0:0:0:13) older than its timeout: EH or stalled queue? Warning: scsi_cmnd 0xffff8800982c8a80 on scsi_device 0xffff88003578f000 (0:0:0:13) older than its timeout: EH or stalled queue? Warning: scsi_cmnd 0xffff8800982c96c0 on scsi_device 0xffff88003578f000 (0:0:0:13) older than its timeout: EH or stalled queue? Warning: scsi_cmnd 0xffff8800982c8fc0 on scsi_device 0xffff88003578f000 (0:0:0:13) older than its timeout: EH or stalled queue? Warning: scsi_cmnd 0xffff8800982c88c0 on scsi_device 0xffff88003578f000 (0:0:0:13) older than its timeout: EH or stalled queue? Warning: scsi_cmnd 0xffff8800982c9a40 on scsi_device 0xffff88003578f000 (0:0:0:13) older than its timeout: EH or stalled queue? Warning: scsi_cmnd 0xffff8801b9641180 on scsi_device 0xffff88003578f000 (0:0:0:13) older than its timeout: EH or stalled queue? Warning: scsi_cmnd 0xffff8801b9640fc0 on scsi_device 0xffff88003578f000 (0:0:0:13) older than its timeout: EH or stalled queue? [...] Display time and state information for SCSI commands (--time) --------------------------------------------------------------- The '--time' or '--t' options provide even more information about the in-flight I/O requests and SCSI commands. Along with command age, it also provides details about when the corresponding 'request' struct was allocated (rq-alloc):: crash> scsishow --time scsi_cmnd ffff91d4a28cf8b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.137 (30) cmnd-alloc: -0.863 rq-alloc: -0.863 scsi_cmnd ffff91d4a28df8b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.091 (30) cmnd-alloc: -0.909 rq-alloc: -0.909 scsi_cmnd ffff91d4a28d48b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.346 (30) cmnd-alloc: -0.654 rq-alloc: -0.654 scsi_cmnd ffff91d4a28d5938 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.331 (30) cmnd-alloc: -0.669 rq-alloc: -0.669 scsi_cmnd ffff91d4a28a5938 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.316 (30) cmnd-alloc: -0.684 rq-alloc: -0.683 scsi_cmnd ffff91d4a28d69b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.559 (30) cmnd-alloc: -0.441 rq-alloc: -0.441 scsi_cmnd ffff91d4a28a7a38 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.829 (30) cmnd-alloc: -0.171 rq-alloc: -0.171 scsi_cmnd ffff91d4a28d8ab8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.205 (30) cmnd-alloc: -0.795 rq-alloc: -0.795 scsi_cmnd ffff91d4a28a8ab8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.916 (30) cmnd-alloc: -0.084 rq-alloc: -0.083 scsi_cmnd ffff91d4a28cd7b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.886 (30) cmnd-alloc: -0.114 rq-alloc: -0.114 scsi_cmnd ffff91d4a28d9b38 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.909 (30) cmnd-alloc: -0.091 rq-alloc: -0.091 scsi_cmnd ffff91d4a28dabb8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.648 (30) cmnd-alloc: -0.352 rq-alloc: -0.351 [...] crash> scsishow --t scsi_cmnd ffff91d4a28cf8b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.137 (30) cmnd-alloc: -0.863 rq-alloc: -0.863 scsi_cmnd ffff91d4a28df8b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.091 (30) cmnd-alloc: -0.909 rq-alloc: -0.909 scsi_cmnd ffff91d4a28d48b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.346 (30) cmnd-alloc: -0.654 rq-alloc: -0.654 scsi_cmnd ffff91d4a28d5938 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.331 (30) cmnd-alloc: -0.669 rq-alloc: -0.669 scsi_cmnd ffff91d4a28a5938 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.316 (30) cmnd-alloc: -0.684 rq-alloc: -0.683 scsi_cmnd ffff91d4a28d69b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.559 (30) cmnd-alloc: -0.441 rq-alloc: -0.441 scsi_cmnd ffff91d4a28a7a38 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.829 (30) cmnd-alloc: -0.171 rq-alloc: -0.171 scsi_cmnd ffff91d4a28d8ab8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.205 (30) cmnd-alloc: -0.795 rq-alloc: -0.795 scsi_cmnd ffff91d4a28a8ab8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.916 (30) cmnd-alloc: -0.084 rq-alloc: -0.083 scsi_cmnd ffff91d4a28cd7b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.886 (30) cmnd-alloc: -0.114 rq-alloc: -0.114 scsi_cmnd ffff91d4a28d9b38 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.909 (30) cmnd-alloc: -0.091 rq-alloc: -0.091 scsi_cmnd ffff91d4a28dabb8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29.648 (30) cmnd-alloc: -0.352 rq-alloc: -0.351 [...] crash> scsishow --t -j scsi_cmnd ffff91d4a28cf8b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29137 (30000) cmnd-alloc: -863 rq-alloc: -863 scsi_cmnd ffff91d4a28df8b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29091 (30000) cmnd-alloc: -909 rq-alloc: -909 scsi_cmnd ffff91d4a28d48b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29346 (30000) cmnd-alloc: -654 rq-alloc: -654 scsi_cmnd ffff91d4a28d5938 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29331 (30000) cmnd-alloc: -669 rq-alloc: -669 scsi_cmnd ffff91d4a28a5938 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29316 (30000) cmnd-alloc: -684 rq-alloc: -684 scsi_cmnd ffff91d4a28d69b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29559 (30000) cmnd-alloc: -441 rq-alloc: -441 scsi_cmnd ffff91d4a28a7a38 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29829 (30000) cmnd-alloc: -171 rq-alloc: -171 scsi_cmnd ffff91d4a28d8ab8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29205 (30000) cmnd-alloc: -795 rq-alloc: -795 scsi_cmnd ffff91d4a28a8ab8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29916 (30000) cmnd-alloc: -84 rq-alloc: -84 scsi_cmnd ffff91d4a28cd7b8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29886 (30000) cmnd-alloc: -114 rq-alloc: -114 scsi_cmnd ffff91d4a28d9b38 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29909 (30000) cmnd-alloc: -91 rq-alloc: -91 scsi_cmnd ffff91d4a28dabb8 on scsi_device 0xffff91d43a493000 (4:0:0:8) is unknown, deadline: 29648 (30000) cmnd-alloc: -352 rq-alloc: -352 [...]