User Guide¶
The PyKdump framework provides quite a few ready to use programs to allow users to quickly extract and process various useful information from vmcore dump.
These in-built programs can be used to quickly fetch details about SCSI adapters, disk devices, multipath maps, LVM volumes, TCP/IP sockets, NFS mounts, processes holding or waiting for mutex locks and much more. Some of these programs also provide an option to run automated checks and report potential issues.
These programs can be accessed by loading the extension in crash environment:
crash> extend /test/mpykdump.so
Setting scroll off while initializing PyKdump
/test/mpykdump.so: shared object loaded
Once the extension is loaded, the commands provided by built-in programs can be viewed as below. Using ‘-h’ option with these commands will provide more details about what a specific command does and options provided by it:
crash> extend
SHARED OBJECT COMMANDS
/test/mpykdump.so epython xportshow crashinfo taskinfo nfsshow hanginfo
fregs tslog scsi scsishow dmshow pstree modinfo
The epython command in above list allows direct execution of any custom python programs written using this framework, without a need to rebuild the whole extension.
Use below links to view the documentation for individual programs:
- Executing custom PyKdump programs using epython
- Extracting network and TCP/IP sockets information (xportshow)
- Print all sockets (-a)
- Print routing table (-r)
- Print the sockets used by programs (-\-program, -\-pid)
- Print Netfilter Hooks (-\-netfilter)
- Print Softnet Queues (–softnet)
- Print a summary of network connections (-\-summary)
- Show TCP, UDP, ICMP statistics (-\-statistics)
- Print network interface details (-i)
- Decode iph/th/uh (-\-decode)
- Limit output to the specified port (-\-port)
- Print LISTEN sockets only (-l)
- Print TCP sockets details (-t)
- List the sockets with specific state (-\-tcpstate)
- Show only TCP retransmissions (-\-retransonly)
- Print UDP/RAW/UNIX sockets info (-u, -w, -x)
- Print network sysctl parameters (-\-sysctl)
- Print dev_pack info (-\-devpack)
- Print ARP & Neighbouring info (-\-arp)
- Set net ns address (-\-netns)
- Run all functions in single command (-\-everything)
- Extracting various useful information from crash dump (crashinfo)
- Getting a summary of crash/panic
- Print sysctl info (-\-sysctl)
- Print EXT3 info (-\-ext3)
- Print Block I/O requests (-\-blkreq)
- Print Block Devices Info (-\-blkdevs)
- Print stacks (bt) categorized summary (-\-stacksummary)
- Print stacks (bt) containing required functions (-\-findstacks)
- Check stacks of all threads for corruption (-\-checkstacks)
- Decode Syscalls on the Stack (-\-decodesyscalls)
- Decode keventd_wq (-\-keventd_wq)
- Decode kblockd_workqueue (-\-kblockd_wq)
- Print Locks Waitqueues and Semaphores (-\-lws)
- Decode the semaphores (-\-semaphore, -\-rwsemaphore)
- Print DeviceMapper Tables (-\-devmapper)
- Print Runqueus (-\-runq)
- Print Mutex info (-\-mutex)
- Print User-space Memory Usage (-\-umem)
- Emulate ‘ls’ (-\-ls)
- Print Workqueues (-\-workqueues)
- Find and print a radix tree element (-\-radix_tree_element)
- Print PCI Info (-\-pci)
- Analyzing process/task details from vmcore (taskinfo)
- Summary (-\-Summary)
- Print the hung task information (-\-hang)
- Maximum number of PIDs to print for -\-hang and -\-mem (-\-maxpids)
- Display the details for a given PID (-\-pidinfo)
- Filter the list of processes as per their state (-\-taskfilter)
- Emulate a user-space ‘pstree’ output (-\-pstree)
- Use reverse order while sorting and printing task details (-r)
- Run specific command for each task (-\-cmd)
- Print a summary of memory usage by tasks (-\-memory)
- Print info about namespaces (-\-ns)
- Reviewing NFS server and client details (nfsshow)
- Print all the NFS server/client information (-a)
- Print NFS server specific information (-\-server)
- Print NFS client specific information (-\-client)
- Print RPC tasks (-\-rpctasks)
- Decode RPC task at specified address (-\-decoderpctask)
- Print info from nfs_client struct at specified address (-\-nfsclient)
- Maximum number of RPC tasks to print (-\-maxrpctasks)
- Print NLM locks (-\-locks)
- Print Deferred Requests (-\-deferred)
- Find NFS related information for pid (-\-pid)
- Getting hung task details (hanginfo)
- Processing SCSI subsystem details (scsishow)
- Show /proc/scsi/scsi style information (-p)
- Show all devices (-d)
- Show all host bus adapters (optional -s option)
- Show all SCSI targets (-T)
- Show in-flight SCSI commands (-c)
- Show commands queued in request queue of device (-q)
- Show requests to SCSI devices (-r)
- Check for common SCSI issues (-\-check)
- Display time and state information for SCSI commands (-\-time)
- Analysing device-mapper and LVM information (dmshow)
- Show mapped_devices and fields (-d)
- Show multipath devices and fields (-m)
- Show multipath devices similar to ‘multipath -ll’ output (-ll)
- Show ‘dmsetup table’ like output (-\-table)
- Show lvm volume information similar to ‘lvs’ command (-\-lvs)
- Show lvm volume and volume group’s UUID (-\-lvuuid)
- Show physical volume information similar to ‘pvs’ command (-\-pvs)
- Check for common DM issues (-\-check)
- Graphical representation of parent/child processes (pstree)
- Getting module specific information (modinfo)
- Packaging and The Contents of mpykdump.so