Analyzing device-mapper and LVM information (dmshow) ==================================================== The dmshow program can be used to quickly extract dm-multipath and LVM volume information from a vmcore. It also provides the '--check' option to check for potential LVM or multipath issues and report them. Options provided by ‘dmshow’:: crash> dmshow -h usage: dmshow [-h] [-x] [--check] [-m [FIELDS]] [-ll [FIELDS]] [-d [FIELDS]] [--table] [--lvs] [--lvuuid] [--pvs] optional arguments: -h, --help show this help message and exit -x, --hex display fields in hex --check check for common DM issues (WIP) -m [FIELDS], --multipath [FIELDS] show multipath devices and fields -ll [FIELDS], --list [FIELDS] show multipath device listing similar to "multipath -ll" -d [FIELDS], --mapdev [FIELDS] show mapped_devices and fields --table show "dmsetup table" like output --lvs show LVM volume information similar to "lvs" command --lvuuid show LVM volume and volume group's UUID --pvs show physical volume information similar to "pvs" command ** Execution took 1.27s (real) 1.27s (CPU) crash> * `Show mapped_devices and fields (-d)`_ * `Show multipath devices and fields (-m)`_ * `Show multipath devices similar to 'multipath -ll' output (-ll)`_ * `Show 'dmsetup table' like output (--table)`_ * `Show LVM volume information similar to 'lvs' command (--lvs)`_ * `Show LVM volume and volume group's UUID (--lvuuid)`_ * `Show physical volume information similar to 'pvs' command (--pvs)`_ * `Check for common DM issues (--check)`_ Show mapped_devices and fields (-d) ----------------------------------- Executing the dmshow program with the '-d' option lists the mapped_device structure pointers associated with multipath devices and LVM volumes. It also prints the mapped_device->flags for each device:: crash> dmshow -d NUMBER NAME MAPPED_DEVICE FLAGS dm-0 rhel00-root 0xffff88020f221800 flags: 64 dm-1 rhel-swap 0xffff88020f227800 flags: 64 dm-2 rhel00-swap 0xffff88020f226800 flags: 64 dm-3 mpathi 0xffff880210213000 flags: 64 dm-4 mpathh 0xffff880210216800 flags: 64 dm-5 mpathb 0xffff8800cf748800 flags: 64 dm-6 mpathg 0xffff8800cf74b800 flags: 64 dm-7 mpathd 0xffff8800cf74e800 flags: 64 dm-8 mpathj 0xffff8802159d2000 flags: 64 dm-9 mpathf 0xffff8802159d7000 flags: 64 dm-10 mpathe 0xffff8800cf289800 flags: 64 [...] dm-29 appvg-oraapps_vol 0xffff8800ce86b800 flags: 64 dm-30 prodvg1-lvdata0 0xffff8800cde73800 flags: 64 dm-31 prodvg1-lvdata1 0xffff8800cde70800 flags: 64 dm-32 prodvg1-lvdata2 0xffff8800cde77800 flags: 64 dm-33 prodvg1-lvdata3 0xffff8800cde77000 flags: 64 dm-34 prodvg1-lvdata4 0xffff88021130d800 flags: 64 dm-35 prodvg2-prdbkplv0 0xffff8800cf749000 flags: 64 dm-36 prodvg2-prdbkplv1 0xffff8800ce86b000 flags: 64 dm-37 rhel00-home 0xffff8800cf74c000 flags: 64 dm-38 rhel-root 0xffff880210217000 flags: 64 ** Execution took 0.04s (real) 0.05s (CPU) crash> Show multipath devices and fields (-m) -------------------------------------- The '-m' option prints basic information for the multipath devices e.g. pointer to struct multipath, nr_valid_paths, and queue_if_no_path:: crash> dmshow -m NUMBER NAME MULTIPATH nr_valid_paths queue_if_no_path dm-3 mpathi 0xffff880207b70800 4 Enabled dm-4 mpathh 0xffff880214945600 4 Enabled dm-5 mpathb 0xffff880214dbb200 4 Enabled dm-6 mpathg 0xffff880214945e00 4 Enabled dm-7 mpathd 0xffff880214dbbe00 4 Enabled dm-8 mpathj 0xffff8800b1593200 4 Enabled dm-9 mpathf 0xffff8801d76b7600 4 Enabled dm-10 mpathe 0xffff880214dbb400 4 Enabled dm-11 mpatha 0xffff880214dbaa00 4 Enabled dm-12 mpathr 0xffff8800cd3bd000 4 Enabled dm-13 mpathn 0xffff880214dbc800 4 Enabled dm-14 mpathp 0xffff8801dda08800 4 Enabled [...] Show multipath devices similar to 'multipath -ll' output (-ll) -------------------------------------------------------------- The '-ll' option prints detailed multipath device information similar to 'multipath -ll' command output:: crash> dmshow -ll ------------------------------------------------------------------------------------------ mpathi (3600140530e36ebf3d744e94952d7b048) dm-3 LIO-ORG block16 size=1024.00M (queue_if_no_path enabled) hwhandler=None +- policy='service-time' `- 0:0:0:16 sdp 8:240 [scsi_device: 0xffff88003578d000 sdev_state: SDEV_RUNNING] +- policy='service-time' `- 0:0:1:16 sdbl 67:240 [scsi_device: 0xffff88020f06d000 sdev_state: SDEV_RUNNING] +- policy='service-time' `- 7:0:0:16 sdae 65:224 [scsi_device: 0xffff88017c80b000 sdev_state: SDEV_RUNNING] +- policy='service-time' `- 7:0:1:16 sdcl 69:144 [scsi_device: 0xffff88020f121000 sdev_state: SDEV_RUNNING] ------------------------------------------------------------------------------------------ mpathh (3600140590f29d48c0e445ee92666ce3b) dm-4 LIO-ORG block15 size=1024.00M (queue_if_no_path enabled) hwhandler=None +- policy='service-time' `- 0:0:0:15 sdq 65:0 [scsi_device: 0xffff880035f35000 sdev_state: SDEV_RUNNING] +- policy='service-time' `- 0:0:1:15 sdbm 68:0 [scsi_device: 0xffff8800352cb000 sdev_state: SDEV_RUNNING] +- policy='service-time' `- 7:0:0:15 sdaf 65:240 [scsi_device: 0xffff88020ee3a800 sdev_state: SDEV_RUNNING] +- policy='service-time' `- 7:0:1:15 sdcm 69:160 [scsi_device: 0xffff88020f120800 sdev_state: SDEV_RUNNING] ------------------------------------------------------------------------------------------ mpathb (360014058aaaf65146b3415cbdd7dcb8a) dm-5 LIO-ORG block1 size=1024.00M (queue_if_no_path enabled) hwhandler=None +- policy='service-time' `- 0:0:0:1 sdar 66:176 [scsi_device: 0xffff88020ee3d000 sdev_state: SDEV_RUNNING] +- policy='service-time' `- 0:0:1:1 sdcf 69:48 [scsi_device: 0xffff88020f125000 sdev_state: SDEV_RUNNING] +- policy='service-time' `- 7:0:0:1 sdbh 67:176 [scsi_device: 0xffff88003578e800 sdev_state: SDEV_RUNNING] +- policy='service-time' `- 7:0:1:1 sdda 70:128 [scsi_device: 0xffff8802107f2000 sdev_state: SDEV_RUNNING] ------------------------------------------------------------------------------------------ mpathg (36001405eefdc858a57842c3bec1855b4) dm-6 LIO-ORG block14 size=1024.00M (queue_if_no_path enabled) hwhandler=None +- policy='service-time' `- 0:0:0:14 sds 65:32 [scsi_device: 0xffff880035f34800 sdev_state: SDEV_RUNNING] +- policy='service-time' `- 0:0:1:14 sdbn 68:16 [scsi_device: 0xffff880035f30000 sdev_state: SDEV_RUNNING] +- policy='service-time' `- 7:0:0:14 sdai 66:32 [scsi_device: 0xffff8802111cd800 sdev_state: SDEV_RUNNING] +- policy='service-time' `- 7:0:1:14 sdcn 69:176 [scsi_device: 0xffff88020f120000 sdev_state: SDEV_RUNNING] ------------------------------------------------------------------------------------------ [...] Show 'dmsetup table' like output (--table) -------------------------------------------- The '--table' option prints the device-mapper internal table similar to the 'dmsetup --table' command output:: crash> dmshow --table rhel00-root: 0 104857600 linear 8:5 [sda] 395208704 rhel-swap: 0 15990784 linear 8:2 [sda] 104859648 rhel00-swap: 0 15990784 linear 8:5 [sda] 2048 mpathi: 0 2097152 multipath 1 queue_if_no_path 0 4 1 service-time 0 1 1 8:240 [sdp] 1 service-time 0 1 1 67:240 [sdbl] 1 service-time 0 1 1 65:224 [sdae] 1 service-time 0 1 1 69:144 [sdcl] 1 mpathh: 0 2097152 multipath 1 queue_if_no_path 0 4 1 service-time 0 1 1 65:0 [sdq] 1 service-time 0 1 1 68:0 [sdbm] 1 service-time 0 1 1 65:240 [sdaf] 1 service-time 0 1 1 69:160 [sdcm] 1 mpathb: 0 2097152 multipath 1 queue_if_no_path 0 4 1 service-time 0 1 1 66:176 [sdar] 1 service-time 0 1 1 69:48 [sdcf] 1 service-time 0 1 1 67:176 [sdbh] 1 service-time 0 1 1 70:128 [sdda] 1 mpathg: 0 2097152 multipath 1 queue_if_no_path 0 4 1 service-time 0 1 1 65:32 [sds] 1 service-time 0 1 1 68:16 [sdbn] 1 service-time 0 1 1 66:32 [sdai] 1 service-time 0 1 1 69:176 [sdcn] 1 [...] mpathq: 0 2097152 multipath 1 queue_if_no_path 0 4 1 service-time 0 1 1 8:64 [sde] 1 service-time 0 1 1 67:64 [sdba] 1 service-time 0 1 1 8:224 [sdo] 1 service-time 0 1 1 69:0 [sdcc] 1 mpaths: 0 2097152 multipath 1 queue_if_no_path 0 4 1 service-time 0 1 1 8:32 [sdc] 1 service-time 0 1 1 67:0 [sdaw] 1 service-time 0 1 1 8:160 [sdk] 1 service-time 0 1 1 68:176 [sdbx] 1 appvg-oraapps_vol: 0 2088960 linear 253:27 [dm-27] 2048 appvg-oraapps_vol: 2088960 2088960 linear 253:12 [dm-12] 2048 appvg-oraapps_vol: 4177920 2088960 linear 253:28 [dm-28] 2048 appvg-oraapps_vol: 6266880 2088960 linear 253:23 [dm-23] 2048 appvg-oraapps_vol: 8355840 2088960 linear 253:25 [dm-25] 2048 [...] prodvg1-lvdata1: 2088960 8192 linear 253:5 [dm-5] 10240 prodvg1-lvdata2: 0 2088960 linear 253:7 [dm-7] 2048 prodvg1-lvdata2: 2088960 8192 linear 253:5 [dm-5] 18432 prodvg1-lvdata3: 0 2088960 linear 253:10 [dm-10] 2048 prodvg1-lvdata3: 2088960 8192 linear 253:5 [dm-5] 26624 prodvg1-lvdata4: 0 2088960 linear 253:9 [dm-9] 2048 [...] Show LVM volume information similar to 'lvs' command (--lvs) -------------------------------------------------------------- Users can get LVM volume information similar to the 'lvs' command output by using the '--lvs' option:: crash> dmshow --lvs LV DM-X DEV LV NAME VG NAME OPEN COUNT LV SIZE (MB) PV NAME dm-0 root rhel00 1 51200.00 sda dm-1 swap rhel 0 7808.00 sda dm-2 swap rhel00 2 7808.00 sda dm-29 oraapps_vol appvg 1 9216.00 mpathq (dm-27) dm-29 oraapps_vol appvg 1 9216.00 mpathr (dm-12) dm-29 oraapps_vol appvg 1 9216.00 mpaths (dm-28) dm-29 oraapps_vol appvg 1 9216.00 mpatht (dm-23) dm-29 oraapps_vol appvg 1 9216.00 mpathu (dm-25) dm-29 oraapps_vol appvg 1 9216.00 mpathv (dm-21) dm-29 oraapps_vol appvg 1 9216.00 mpathw (dm-20) dm-29 oraapps_vol appvg 1 9216.00 mpathx (dm-19) dm-29 oraapps_vol appvg 1 9216.00 mpathy (dm-18) dm-29 oraapps_vol appvg 1 9216.00 mpathz (dm-22) dm-30 lvdata0 prodvg1 1 1024.00 mpatha (dm-11) dm-30 lvdata0 prodvg1 1 1024.00 mpathb (dm-5) dm-31 lvdata1 prodvg1 1 1024.00 mpathc (dm-16) dm-31 lvdata1 prodvg1 1 1024.00 mpathb (dm-5) dm-32 lvdata2 prodvg1 1 1024.00 mpathd (dm-7) dm-32 lvdata2 prodvg1 1 1024.00 mpathb (dm-5) dm-33 lvdata3 prodvg1 1 1024.00 mpathe (dm-10) Show LVM volume and volume group's UUID (--lvuuid) ---------------------------------------------------- The output of the '--lvuuid' option is similar to '--lvs', but it also provides the UUID of LVM volumes and volume groups:: crash> dmshow --lvuuid LV DM-X DEV LV NAME VG NAME LV SIZE (MB) LV UUID VG UUID dm-0 root rhel00 51200.00 tnqhh4lxlhQqNuKP4V8WgnE5c6RY46oC YwZPcP2fdN5KlZn52mve8FxANMtkVD0f dm-1 swap rhel 7808.00 5mkMJYaeYUKsNgQpZbhUhEcPIizO5hfW G9ckFgUu9Ta5370Drr3o1HSEC5cKOq8d dm-2 swap rhel00 7808.00 yRYd9VCCakcSQHdxBLrT80RFZP2WPuYV YwZPcP2fdN5KlZn52mve8FxANMtkVD0f dm-29 oraapps_vol appvg 9216.00 Es4gcJ1berhRRjtQcpeUspPl5eT9oHVf eOL5OQSw1ECpqQMOSyBnd5zboTGjnd7Y dm-29 oraapps_vol appvg 9216.00 Es4gcJ1berhRRjtQcpeUspPl5eT9oHVf eOL5OQSw1ECpqQMOSyBnd5zboTGjnd7Y dm-29 oraapps_vol appvg 9216.00 Es4gcJ1berhRRjtQcpeUspPl5eT9oHVf eOL5OQSw1ECpqQMOSyBnd5zboTGjnd7Y dm-29 oraapps_vol appvg 9216.00 Es4gcJ1berhRRjtQcpeUspPl5eT9oHVf eOL5OQSw1ECpqQMOSyBnd5zboTGjnd7Y dm-29 oraapps_vol appvg 9216.00 Es4gcJ1berhRRjtQcpeUspPl5eT9oHVf eOL5OQSw1ECpqQMOSyBnd5zboTGjnd7Y dm-29 oraapps_vol appvg 9216.00 Es4gcJ1berhRRjtQcpeUspPl5eT9oHVf eOL5OQSw1ECpqQMOSyBnd5zboTGjnd7Y dm-29 oraapps_vol appvg 9216.00 Es4gcJ1berhRRjtQcpeUspPl5eT9oHVf eOL5OQSw1ECpqQMOSyBnd5zboTGjnd7Y dm-29 oraapps_vol appvg 9216.00 Es4gcJ1berhRRjtQcpeUspPl5eT9oHVf eOL5OQSw1ECpqQMOSyBnd5zboTGjnd7Y dm-29 oraapps_vol appvg 9216.00 Es4gcJ1berhRRjtQcpeUspPl5eT9oHVf eOL5OQSw1ECpqQMOSyBnd5zboTGjnd7Y dm-29 oraapps_vol appvg 9216.00 Es4gcJ1berhRRjtQcpeUspPl5eT9oHVf eOL5OQSw1ECpqQMOSyBnd5zboTGjnd7Y dm-30 lvdata0 prodvg1 1024.00 NGCX1qOmDeJ7XT77LeKp4WFw1an124Wc 7JS0jiSW2v53RpDzeV2GFJqmspyVS4bY dm-30 lvdata0 prodvg1 1024.00 NGCX1qOmDeJ7XT77LeKp4WFw1an124Wc 7JS0jiSW2v53RpDzeV2GFJqmspyVS4bY dm-31 lvdata1 prodvg1 1024.00 wBcHLGf1c63K5Ki2sWuuu7fVj4OQagT9 7JS0jiSW2v53RpDzeV2GFJqmspyVS4bY dm-31 lvdata1 prodvg1 1024.00 wBcHLGf1c63K5Ki2sWuuu7fVj4OQagT9 7JS0jiSW2v53RpDzeV2GFJqmspyVS4bY dm-32 lvdata2 prodvg1 1024.00 qdctN8EuI1EDVRJZPISc34P0dQBDqXG8 7JS0jiSW2v53RpDzeV2GFJqmspyVS4bY dm-32 lvdata2 prodvg1 1024.00 qdctN8EuI1EDVRJZPISc34P0dQBDqXG8 7JS0jiSW2v53RpDzeV2GFJqmspyVS4bY dm-33 lvdata3 prodvg1 1024.00 G8osg99x1eKGP2OnI29s0S53Hn79gIm3 7JS0jiSW2v53RpDzeV2GFJqmspyVS4bY dm-33 lvdata3 prodvg1 1024.00 G8osg99x1eKGP2OnI29s0S53Hn79gIm3 7JS0jiSW2v53RpDzeV2GFJqmspyVS4bY [...] Show physical volume information similar to 'pvs' command (--pvs) ------------------------------------------------------------------- To view information about LVM physical volumes, use the '--pvs' option:: crash> dmshow --pvs PV NAME PV's MAPPED_DEVICE DEVICE SIZE (MB) VG NAME LV NAME sda [PV not on dm dev, skipping!] 305245.34 rhel00 root sda [PV not on dm dev, skipping!] 305245.34 rhel swap sda [PV not on dm dev, skipping!] 305245.34 rhel00 swap mpathq (dm-27) ffff880211309000 1024.00 appvg oraapps_vol mpathr (dm-12) ffff8800cf28f800 1024.00 appvg oraapps_vol mpaths (dm-28) ffff88021e4eb000 1024.00 appvg oraapps_vol mpatht (dm-23) ffff88021065d000 1024.00 appvg oraapps_vol mpathu (dm-25) ffff88021130a000 1024.00 appvg oraapps_vol mpathv (dm-21) ffff8800cde73000 1024.00 appvg oraapps_vol mpathw (dm-20) ffff8800cde70000 1024.00 appvg oraapps_vol mpathx (dm-19) ffff8800ceaf5800 1024.00 appvg oraapps_vol mpathy (dm-18) ffff8800ceaf1800 1024.00 appvg oraapps_vol mpathz (dm-22) ffff88021065a000 1024.00 appvg oraapps_vol mpatha (dm-11) ffff8800cf28c800 1024.00 prodvg1 lvdata0 mpathb (dm-5) ffff8800cf748800 1024.00 prodvg1 lvdata0 mpathc (dm-16) ffff8802112e4800 1024.00 prodvg1 lvdata1 mpathb (dm-5) ffff8800cf748800 1024.00 prodvg1 lvdata1 mpathd (dm-7) ffff8800cf74e800 1024.00 prodvg1 lvdata2 mpathb (dm-5) ffff8800cf748800 1024.00 prodvg1 lvdata2 [...] Check for common DM issues (--check) -------------------------------------- The '--check' option checks for potential LVM or multipath issues and reports them. For example, the following output shows a system with multipath devices but the 'multipathd' process is not running. This will lead to no path checks and I/O failover will not work:: crash> dmshow --check NUMBER NAME MAPPED_DEVICE FLAGS dm-0 rhel00-root 0xffff88020f221800 flags: 0x40 dm-1 rhel-swap 0xffff88020f227800 flags: 0x40 dm-2 rhel00-swap 0xffff88020f226800 flags: 0x40 dm-3 mpathi 0xffff880210213000 flags: 0x40 dm-4 mpathh 0xffff880210216800 flags: 0x40 [...] dm-36 prodvg2-prdbkplv1 0xffff8800ce86b000 flags: 0x40 dm-37 rhel00-home 0xffff8800cf74c000 flags: 0x40 dm-38 rhel-root 0xffff880210217000 flags: 0x40 Checking for device-mapper issues... Getting a list of processes in UN state... [Done] (Count: 21) Processing the back trace of hung tasks... [Done] ** multipath device(s) are present, but multipathd service is not running. IO failover/failback may not work. Found 21 processes in UN state. Run 'hanginfo' for more information on processes in UN state. ** Execution took 2.40s (real) 2.31s (CPU) crash>