Showing and analyzing nvme information (nvme) ============================================= The nvme command is designed to provide an overview of the properties and state of the nvme devices attached to the system. Several layers are covered, including the ctrl layer, namespace, and subsystem layer. Additionally, the tool will show what sort of transport the nvme device is using, for example, fibre, tcp, or pci. Options provided by ‘nvme’:: crash> nvme -h usage: nvme.py [-h] [-l [NS]] [-c [CTRL]] [-n [NS]] [-d [CTRL]] [-q [CTRL]] [-i [QID]] [-s [SUB]] [-k] optional arguments: -h, --help show this help message and exit -l [NS], --list [NS] list nvme namespaces vpd data and capacity -c [CTRL], --ctrl [CTRL] show nvme controller information (nvme_ctrl) -n [NS], --ns [NS] show nvme namespace information (nvme_ns) -d [CTRL], --dev [CTRL] show nvme device information (nvme_dev, nvme_loop_ctrl, nvme_rdma_ctrl, nvme_fc_ctrl, nvme_tcp_ctrl) -q [CTRL], --queue [CTRL] show nvme queue information (nvme_queue, nvme_loop_queue, nvme_rdma_queue, nvme_fc_queue) -i [QID], --qid [QID] limit output by QID. for use with -q -s [SUB], --subsystem [SUB] show nvme subsystem information (nvme_subsystem) -k, --check check for common NVMe issues * `List nvme namespaces vpd data and capacity (-l)`_ * `Show nvme controller (nvme_ctrl) information (-c)`_ * `Show nvme namespace (nvme_ns) information (-n)`_ * `Show nvme device (nvme_dev, nvme_loop_ctrl, nvme_rdma_ctrl, nvme_fc_ctrl, nvme_tcp_ctrl) information (-d)`_ * `Show nvme queue (nvme_queue, nvme_loop_queue, nvme_rdma_queue, nvme_fc_queue) information (-q)`_ * `Limit output by QID ; for use with -q option (-i)`_ * `Show nvme subsystem (nvme_subsystem) information (-s)`_ * `Check for common nvme issues (-k)`_ List nvme namespaces vpd data and capacity (-l) ----------------------------------------------- The nvme list (-l) option prints the vital product data of the nvme devices in addition to the capacity data. It will also show the format of the device and the firmware revision:: crash> nvme -l Node SN Model Namespace Capacity(gendisk) Format FW Rev ---------------- -------------------- ---------------------------------------- --------- ------------------ ---------------- -------- /dev/nvme0n1 PZDxxxxxxxxxxx HPE NS204i-u Gen11 Boot Controller 1 447 GiB 512 B + 0 B 12141004 /dev/nvme1n1 S70xxxxxxxxxxx VO001920KYDMT 1 2 TiB 512 B + 0 B HPK1 /dev/nvme2n1 S70xxxxxxxxxxx VO001920KYDMT 1 2 TiB 512 B + 0 B HPK1 crash> nvme -l Node SN Model Namespace Capacity(gendisk) Format FW Rev ---------------- -------------------- ---------------------------------------- --------- ------------------ ---------------- -------- /dev/nvme0n1 PHHxxxxxxxxxxxxx SSDPEKKF512G8 NVMe INTEL 512GB 1 477 GiB 512 B + 0 B D05N Show nvme controller (nvme_ctrl) information (-c) ------------------------------------------------- The nvme controller information option (-c) prints various information tied to the nvme_ctrl struct and layer. Included are any quirks present, the controller state, several I/O limit values, and the major other structs that are tied to the nvme_ctrl struct. Among those are a list of namespaces, any admin or connection queue, the device, and the subsystem:: crash> nvme -c Name Ctrl Addr Namespaces(list_head) AdminQ ConnectQ Subsystem Ctrl Device ---------- ---------------- --------------------- ---------------- ---------------- ---------------- ---------------- nvme0 ff14afd243390300 ff14afd243390370 ff14afd248b02a60 0 ff14afdbff056800 ff14afd2433903a8 Quirks: None NumQueues: 4 CtrlState: NVME_CTRL_LIVE MaxHWSectors: 256 MaxSegments: 127 PageSize: Name Ctrl Addr Namespaces(list_head) AdminQ ConnectQ Subsystem Ctrl Device ---------- ---------------- --------------------- ---------------- ---------------- ---------------- ---------------- nvme1 ff14b0324338a300 ff14b0324338a370 ff14b0324cbcbf90 0 ff14b0324cbdf800 ff14b0324338a3a8 Quirks: None NumQueues: 64 CtrlState: NVME_CTRL_LIVE MaxHWSectors: 2048 MaxSegments: 127 PageSize: [...] crash> nvme -c Name Ctrl Addr Namespaces(list_head) AdminQ ConnectQ Subsystem Ctrl Device ---------- ---------------- --------------------- ---------------- ---------------- ---------------- ---------------- nvme0 ffffa0bbde082300 ffffa0bbde082370 ffffa0bbd968e4a0 0 ffffa0bbbc47d800 ffffa0bbde0823a8 Quirks: NVME_QUIRK_IGNORE_DEV_SUBNQN NumQueues: 16 CtrlState: NVME_CTRL_LIVE MaxHWSectors: 512 MaxSegments: 127 PageSize: 4096 Show nvme namespace (nvme_ns) information (-n) ---------------------------------------------- The nvme namespace option (-n) shows detailed information about the individual nvme_ns structs and layer, along with the namespace flags. Included are the request_queue, the nvme_ns_head, gendisk, and siblings:: crash> nvme -n Name NS Addr NS Head RequestQ Gendisk Siblings Flags ---------- ---------------- ---------------- ---------------- ---------------- ---------------- ----- nvme0n1 ff14afdbff1c7000 ff14afd24aaf0000 ff14afd24aacf488 ff14afdbff1c1800 ff14afdbff1c7030 0x10 nvme1n1 ff14b0324c4af800 ff14b032543b0000 ff14b032542e8000 ff14b0324c4ae000 ff14b0324c4af830 0x10 nvme2n1 ff14b032544fd800 ff14b032545c0000 ff14b03254504a28 ff14b032544f8800 ff14b032544fd830 0x10 crash> nvme -n Name NS Addr NS Head RequestQ Gendisk Siblings Flags ---------- ---------------- ---------------- ---------------- ---------------- ---------------- ----- nvme0n1 ffffa0bbc3fd9280 ffffa0bbd6900000 ffffa0bbd9688a10 ffffa0bbd96df800 ffffa0bbc3fd92b0 0x0 Show nvme device (nvme_dev, nvme_loop_ctrl, nvme_rdma_ctrl, nvme_fc_ctrl, nvme_tcp_ctrl) information (-d) --------------------------------------------------------------------------------------------------------- The nvme device option (-d) will show various addresses and values tied to any of the nvme_dev type structs. The actual struct used will vary based on transport. Included are the queues, tagsets, pools, and related controller address. Also, several related queue values are provided:: crash> nvme -d pci:Name DevAddr Queues Tagset AdminTagset Ctrl Addr PagePool SmallPool -------- ---------------- ---------------- ---------------- ---------------- ---------------- ---------------- ---------------- nvme0 ff14afd243390000 ff14afd248a20000 ff14afd243390008 ff14afd243390118 ff14afd243390300 ff14afd24814f480 ff14afd24814fe00 OnlineQueues: 4 MaxQID: 3 IOQueues: {3, 0, 0} NumVecs: 4 QDepth: 1024 DBStride 1 Tagset: AdminTagset: ================ ================ Map: ff14afd243390008 ff14afd243390118 NrMaps: 2 1 NrHWQueues: 3 1 QDepth: 1023 30 CmdSize: 168 168 NumaNode: 3 3 Timeout: 30000 60000 Flags: 0x1 0x40 [...] crash> nvme -d pci:Name DevAddr Queues Tagset AdminTagset Ctrl Addr PagePool SmallPool -------- ---------------- ---------------- ---------------- ---------------- ---------------- ---------------- ---------------- nvme0 ffffa0bbde082000 ffffa0bbdf8d8000 ffffa0bbde082008 ffffa0bbde082118 ffffa0bbde082300 ffffa0bbbc6caa80 ffffa0bbbc6cae80 OnlineQueues: 16 MaxQID: 15 IOQueues: {15, 0, 0} NumVecs: 16 QDepth: 1024 DBStride 1 Tagset: AdminTagset: ================ ================ Map: ffffa0bbde082008 ffffa0bbde082118 NrMaps: 2 1 NrHWQueues: 15 1 QDepth: 1023 30 CmdSize: 176 176 NumaNode: 0 0 Timeout: 30000 60000 Flags: 0x1 0x40 Show nvme queue (nvme_queue, nvme_loop_queue, nvme_rdma_queue, nvme_fc_queue) information (-q) ---------------------------------------------------------------------------------------------- The nvme queue option (-q) will show information relevant to the individual queues in the nvme device. Provided are the command lists, completions, and the tags. The entry points into the queue info are also provided, along with the flags. The structs used are transport specific:: crash> nvme -q pci:Ctrl[qid] Queue Addr DMA Dev NVMe Dev SQ Cmds Completion Tags ------------- ---------------- ---------------- ---------------- ---------------- ---------------- ---------------- nvme0[0] ff14afd248a20000 ff14afd243390000 ff14afd248a0a000 ff14afd248a09000 ff14af42cadaff00 QDepth: 32 CQVector: 0 SQTail: 20 LastSQTail: 20 CQHead: 19 LastCQHead: QiD: 0 CQPhase: 0 Flags: 0x1 QCount: 4 pci:Ctrl[qid] Queue Addr DMA Dev NVMe Dev SQ Cmds Completion Tags ------------- ---------------- ---------------- ---------------- ---------------- ---------------- ---------------- nvme0[1] ff14afd248a200c0 ff14afd243390000 ff14afd248b50000 ff14afd248b48000 ff14af4300927300 QDepth: 1024 CQVector: 1 SQTail: 368 LastSQTail: 368 CQHead: 368 LastCQHead: QiD: 1 CQPhase: 0 Flags: 0x1 QCount: 4 [...] pci:Ctrl[qid] Queue Addr DMA Dev NVMe Dev SQ Cmds Completion Tags ------------- ---------------- ---------------- ---------------- ---------------- ---------------- ---------------- nvme1[0] ff14b03250340000 ff14b0324338a000 ff14b03250093000 ff14b0324c49f000 ff14af42caddf600 QDepth: 32 CQVector: 0 SQTail: 12 LastSQTail: 12 CQHead: 11 LastCQHead: QiD: 0 CQPhase: 0 Flags: 0x1 QCount: 64 pci:Ctrl[qid] Queue Addr DMA Dev NVMe Dev SQ Cmds Completion Tags ------------- ---------------- ---------------- ---------------- ---------------- ---------------- ---------------- nvme1[1] ff14b032503400c0 ff14b0324338a000 ff14b0324d1a0000 ff14b0324d190000 ff14af7247698800 QDepth: 1024 CQVector: 1 SQTail: 6 LastSQTail: 6 CQHead: 6 LastCQHead: QiD: 1 CQPhase: 1 Flags: 0x1 QCount: 64 Limit output by QID ; for use with -q option (-i) ------------------------------------------------- The limit option (-i) can be used with the nvme queue option (-q) to limit the output to a particular queue:: crash> nvme -q nvme2 -i 60 pci:Ctrl[qid] Queue Addr DMA Dev NVMe Dev SQ Cmds Completion Tags ------------- ---------------- ---------------- ---------------- ---------------- ---------------- ---------------- nvme2[60] ff14b0324c4c2d00 ff14b0324338e000 ff14b03251540000 ff14b03251530000 ff14af42cafd5600 QDepth: 1024 CQVector: 60 SQTail: 0 LastSQTail: 0 CQHead: 0 LastCQHead: QiD: 60 CQPhase: 1 Flags: 0x1 QCount: 64 Show nvme subsystem (nvme_subsystem) information (-s) ----------------------------------------------------- The nvme subsystem option (-s) shows the nvme_subsystem related addresses, including the subsystem, the controller, and the namespace. This option is most useful for viewing nvme multipathing information:: crash> nvme -s nvme-subsys0 [0xff14afdbff056800] - NQN=PZDxxxxxxxxxxx HPE NS204i-u Gen11 Boot Controller `-+-nvme0:pcie [0xff14afd243390300] NVME_CTRL_LIVE nvme0n1: [0xff14afdbff1c7000] NVME_ANA_OPTIMIZED nvme-subsys1 [0xff14b0324cbdf800] - NQN=nqn.xxxx-xx.com.samsung:nvme:xxxxxxx:2.5-inch:S70xxxxxxxxxxx `-+-nvme1:pcie [0xff14b0324338a300] NVME_CTRL_LIVE nvme1n1: [0xff14b0324c4af800] NVME_ANA_OPTIMIZED nvme-subsys2 [0xff14b0324d004000] - NQN=nqn.xxxx-xx.com.samsung:nvme:xxxxxxx:2.5-inch:S70xxxxxxxxxxx `-+-nvme2:pcie [0xff14b0324338e300] NVME_CTRL_LIVE nvme2n1: [0xff14b032544fd800] NVME_ANA_OPTIMIZED crash> nvme -s nvme-subsys0 [0xffffa0bd84747800] - NQN=nqn.xxxx-xx.com.dell:powerflex:00:xxxxxxxxxxxxxxxxx `-+-nvme0:tcp [0xffffa0a3036ba348] NVME_CTRL_LIVE nvme0c0n12: [0xffffa0a55fe68800] NVME_ANA_OPTIMIZED +-nvme2:tcp [0xffffa0a346394348] NVME_CTRL_LIVE nvme0c2n12: [0xffffa0bdc9863400] NVME_ANA_OPTIMIZED nvme-subsys0 [0xffffa0bd84747800] - NQN=nqn.xxxx-xx.com.dell:powerflex:00:xxxxxxxxxxxxxxxx `-+-nvme0:tcp [0xffffa0a3036ba348] NVME_CTRL_LIVE nvme0c0n1: [0xffffa0a66792e800] NVME_ANA_OPTIMIZED +-nvme1:tcp [0xffffa0a346390348] NVME_CTRL_LIVE nvme0c1n1: [0xffffa0a462258000] NVME_ANA_OPTIMIZED +-nvme2:tcp [0xffffa0a346394348] NVME_CTRL_LIVE nvme0c2n1: [0xffffa0bf42fa8000] NVME_ANA_OPTIMIZED [...] Check for common nvme issues (-k) --------------------------------- The check option (-k) will check for any common nvme issues:: crash> nvme -k WARNING: nvme1 in state NVME_CTRL_RESETTING Warning: extra flags set for : NVME_NS_STOPPED Warning: extra flags set for : NVME_NS_STOPPED Warning: extra flags set for : NVME_NS_STOPPED