User GuideΒΆ
The PyKdump framework provides quite a few ready-to-use programs to allow users to quickly extract and process various useful information from a vmcore dump.
These built-in programs can be used to quickly fetch details about SCSI adapters, disk devices, multipath maps, LVM volumes, TCP/IP sockets, NFS mounts, processes holding or waiting for mutex locks, and much more. Some of these programs also provide an option to run automated checks and report potential issues.
These programs can be accessed by loading the extension in the crash environment:
crash> extend /test/mpykdump.so
Setting scroll off while initializing PyKdump
/test/mpykdump.so: shared object loaded
Once the extension is loaded, the commands provided by built-in programs can be viewed with the "extend" command as shown below. They will also show up in the list of commands provided by the crash "help" command. Using the '-h' option with these commands will provide more details about what a specific command does and its options:
crash> extend
SHARED OBJECT COMMANDS
/test/mpykdump.so epython xportshow crashinfo taskinfo nfsshow hanginfo
fregs tslog scsishow dmshow pstree modinfo mdadm nvme
keyringshow cifsshow
The epython command allows direct execution of any custom Python programs written using this framework, without a need to rebuild the whole extension.
- Show CIFS mount inforrmation (cifsshow)
- Extracting various useful information from a crash dump (crashinfo)
- Getting a summary of crash/panic
- Print sysctl info (--sysctl)
- Print EXT3 info (--ext3)
- Print Block I/O requests (--blkreq)
- Print Block Devices Info (--blkdevs)
- Print stacks (bt) categorized summary (--stacksummary)
- Print stacks (bt) containing required functions (--findstacks)
- Check stacks of all threads for corruption (--checkstacks)
- Decode Syscalls on the Stack (--decodesyscalls)
- Decode keventd_wq (--keventd_wq)
- Decode kblockd_workqueue (--kblockd_wq)
- Print Locks, Wait queues, and Semaphores (--lws)
- Decode semaphores (--semaphore, --rwsemaphore)
- Print DeviceMapper Tables (--devmapper)
- Print Run Queues (--runq)
- Print Mutex info (--mutex)
- Print Userspace Memory Usage (--umem)
- Emulate 'ls' (--ls)
- Print Workqueues (--workqueues)
- Find and print a radix tree element (--radix_tree_element)
- Print PCI Info (--pci)
- Search for a value and print information about matching locations (detailedsearch)
- Analyzing device-mapper and LVM information (dmshow)
- Show mapped_devices and fields (-d)
- Show multipath devices and fields (-m)
- Show multipath devices similar to 'multipath -ll' output (-ll)
- Show 'dmsetup table' like output (--table)
- Show LVM volume information similar to 'lvs' command (--lvs)
- Show LVM volume and volume group's UUID (--lvuuid)
- Show physical volume information similar to 'pvs' command (--pvs)
- Check for common DM issues (--check)
- Executing custom PyKdump programs using epython
- Finding register contents and arguments at routine entry (fregs)
- Getting hung task details (hanginfo)
- Show information about keyrings (keyringshow)
- Displaying directory and file information from dentry structures (lsdentry)
- Show MD/RAID device information (mdadm)
- Getting module-specific information (modinfo)
- Reviewing NFS server and client details (nfsshow)
- Print all the NFS server/client information (-a)
- Print NFS server specific information (--server)
- Print NFS client specific information (--client)
- Print RPC tasks (--rpctasks)
- Decode RPC task at specified address (--decoderpctask)
- Print info from nfs_client struct at specified address (--nfsclient)
- Maximum number of RPC tasks to print (--maxrpctasks)
- Print NLM locks (--locks)
- Print Deferred Requests (--deferred)
- Find NFS related information for pid (--pid)
- Showing and analyzing nvme information (nvme)
- List nvme namespaces vpd data and capacity (-l)
- Show nvme controller (nvme_ctrl) information (-c)
- Show nvme namespace (nvme_ns) information (-n)
- Show nvme device (nvme_dev, nvme_loop_ctrl, nvme_rdma_ctrl, nvme_fc_ctrl, nvme_tcp_ctrl) information (-d)
- Show nvme queue (nvme_queue, nvme_loop_queue, nvme_rdma_queue, nvme_fc_queue) information (-q)
- Limit output by QID ; for use with -q option (-i)
- Show nvme subsystem (nvme_subsystem) information (-s)
- Check for common nvme issues (-k)
- Graphical representation of parent/child processes (pstree)
- Processing SCSI subsystem details (scsishow)
- Show /proc/scsi/scsi style information (-p)
- Show all devices (-d)
- Show all host bus adapters (optional -s option)
- Show all SCSI targets (-T)
- Show FC remote port details (-f)
- Show iSCSI session details (-i)
- Show in-flight SCSI commands (-c)
- Show commands queued in request queue of device (-q)
- Show requests to SCSI devices (-r)
- Check for common SCSI issues (--check)
- Display time and state information for SCSI commands (--time)
- Analyzing process/task details from vmcore (taskinfo)
- Summary (--summary)
- Show task start times (--times)
- Show the last scheduled task on each CPU (--lastsched)
- Print hung task information (--hang)
- Maximum number of PIDs to print for --hang and --mem (--maxpids)
- Display the details for a given PID (--pidinfo)
- Filter the list of processes by state (--taskfilter)
- Emulate a user-space 'pstree' output (--pstree)
- Use reverse order while sorting and printing task details (-r)
- Run a specific command for each task (--cmd)
- Print a summary of memory usage by tasks (--memory)
- Print info about namespaces (--ns)
- Extracting network and TCP/IP sockets information (xportshow)
- Print all sockets (-a)
- Print routing table (-r)
- Print the sockets used by programs (--program, --pid)
- Print Netfilter Hooks (--netfilter)
- Print Softnet Queues (--softnet)
- Print a summary of network connections (--summary)
- Show TCP, UDP, ICMP statistics (--statistics)
- Print network interface details (-i)
- Decode iph/th/uh (--decode)
- Limit output to the specified port (--port)
- Print LISTEN sockets only (-l)
- Print TCP sockets details (-t)
- List the sockets with specific state (--tcpstate)
- Show only TCP retransmissions (--retransonly)
- Print UDP/RAW/UNIX sockets info (-u, -w, -x)
- Print network sysctl parameters (--sysctl)
- Print dev_pack info (--devpack)
- Print ARP & Neighbouring info (--arp)
- Set net ns address (--netns)
- Run all functions in single command (--everything)
- Packaging and The Contents of mpykdump.so