Packaging and The Contents of mpykdump.so¶

SYNOPSIS¶

PyKdump crash extension - /usr/lib64/crash/extensions/PyKdump/mpykdump.so

DESCRIPTION¶

The PyKdump framework provides Python bindings to GDB/crash internals. It can be used to create Python-based programs to automate vmcore analysis and quickly process various information from vmcores. The current version of PyKdump is standardized on Python 3.

Automating the vmcore analysis requires programmatic analysis of various data structures, e.g. linked lists, structures, unions, enums, and various global variables. The PyKdump framework provides an extensive set of built-in API calls, which can be used to fetch required structures and symbols, and process them in Python programs.

To allow the kernel data structures to be processed within Python programs, the ‘PyKdump’ framework first maps the C structures used within the Linux kernel to Python objects. For example, PyKdump maps C ‘struct’ and ‘union’ by creating corresponding Python objects with attributes matching the respective field names of the C struct/union. Other C data types are mapped to similar Python types, e.g. C ‘int’ is mapped to Python ‘integer’. Also, most operators in C are mapped to similar Python operators.

The 'mpykdump.so' extension consists of:

o Embedded Python interpreter

o Interface module to crash internals

o Subset of the Python Standard Library

o Standard tools built on top of PyKdump as listed below

The extension file is constructed as a shared library with ZIP-archive appended. This extension includes the Python3 interpreter (embedded) and a comprehensive subset of the Python3 Standard Library (strings, sockets, option parsers, itertools, etc.) but not the whole library (as it is huge). This design makes it completely independent of the Python version installed on the host where you do dump analysis.

There are several practical programs already developed using this framework that can be used immediately, including:

xportshow - Displays information about connections and sockets

crashinfo - Shows general information about the kernel dump

scsishow - Shows SCSI subsystem information from the dump

dmshow - Shows device-mapper, multipath, and LVM information

taskinfo - Prints process/task status information as captured at the time of crash

nfsshow - Prints NFS client/server information from the dump

hanginfo - Summarizes information about hung tasks

The above list is not comprehensive, and new commands are added periodically. For a full list, use the 'extend' command in crash or see the User Guide page in this documentation.

EXAMPLES¶

The mpykdump extension can be loaded in the crash environment as follows:

crash> extend /usr/lib64/crash/extensions/PyKdump/mpykdump.so

To view the ready-to-use programs:

crash> extend
SHARED OBJECT                                    COMMANDS
/usr/lib64/crash/extensions/PyKdump/mpykdump.so  epython xportshow crashinfo taskinfo
                                                 nfsshow hanginfo fregs tslog scsi
                                                 scsishow dmshow pstree modinfo

As soon as the extension is loaded, any of the above programs can be executed as a normal command in the crash environment. For example, running the dmshow program to review LVM information:

crash>  dmshow --lvs
LV DM-X DEV   LV NAME      VG NAME           OPEN COUNT       LV SIZE (MB)     PV NAME
dm-0          lv_root      vg_system                  1           28156.00     sda
dm-1          lv_swap      vg_system                  1            2048.00     sda
dm-2          lv_app2      vg_app2                    1           20476.00     sdd
dm-3          lv_app1      vg_app1                    1          425980.00     sdc
dm-4          lv_swap2     vg_swap2                   1           14332.00     sdb
dm-5          lv_var       vg_system                  1           18432.00     sda
dm-6          lv_tmp       vg_system                  1            2048.00     sda

** Execution took   1.12s (real)   1.11s (CPU)
crash>

Running hanginfo:

crash> hanginfo
*** UNINTERRUPTIBLE threads, classified ***

================== Waiting in io_schedule ==================
... 7 pids. Youngest,oldest: 2757, 1787  Ran ms ago: 80427, 227483
sorted by ran_ago, youngest first
[2757, 1785, 1519, 809, 13462, 17140, 1787]

********  Non-classified UN Threads ********** 5 in total

------- 1 stacks like that: ----------
#0   schedule
#1   start_this_handle
#2   jbd2_journal_start
#3   ext4_journal_start_sb
#4   ext4_dirty_inode
#5   __mark_inode_dirty
#6   file_update_time
#7   __generic_file_aio_write
#8   generic_file_aio_write
#9   ext4_file_write
#10  do_sync_write
#11  vfs_write
#12  sys_write
#13  sysenter_dispatch
#14  ia32_sysenter_target
#15  ia32_sysenter_target
[...]

The '-h' (help) argument with any of the above programs will provide more information about the options supported by the program.

The PyKdump framework also allows execution of newly written Python programs without recompiling the whole extension. If there is any custom python program written under the PyKdump framework, it can be executed directly using the epython command as shown below:

crash> epython  <path-to-PyKdump-python-program>

For example: To run the hello.py PyKdump program from the below location:

$ cat hello.py
# This is a basic PyKdump program
from pykdump.API import*
print("Hello PyKdump")

crash> epython  /usr/lib64/crash/extensions/PyKdump/hello.py
Hello PyKdump

ENVIRONMENT¶

PYKDUMPPATH

The 'PYKDUMPPATH' environment variable is similar to the PATH variable in Linux. It can be used to specify the path for Python programs written under this framework. After setting this variable, users can directly execute the Python program from the crash environment without specifying its full path.

For example, the following directory contains a couple of Python programs:

$ ls /cores/crashext/epython/storage
dm.py  dmshow.py  rqlist.py  scsishow.py

Set the PYKDUMPPATH variable with the above path:

$ export PYKDUMPPATH=/cores/crashext/epython/storage
$ echo $PYKDUMPPATH
/cores/crashext/epython/storage

The epython command provided by mpykdump.so can now directly access the above programs:

crash> extend /usr/lib64/crash/extensions/PyKdump/mpykdump.so
crash> epython -p
3.7.3 (default, Oct  7 2019, 11:22:29)
     [GCC 4.4.7 20120313 (Red Hat 4.4.7-18)]
     ['.', '/cores/crashext/scsishow.so/pylib',
                    '/cores/crashext/epython/storage',
                    '/cores/crashext/scsishow.so',
                    '/cores/crashext/scsishow.so/dist-packages']

crash> ls /cores/crashext/epython/storage
dm.py  dmshow.py  rqlist.py  scsishow.py

crash> epython dmshow.py
NUMBER  NAME                 MAPPED_DEVICE    FLAGS
dm-0    vg00-root       0xffff93d725733800    flags: 0x43      [Device suspended]
dm-1    vg00-swap       0xffff93ee12bac000    flags: 0x43      [Device suspended]
[...]

Changes to the PYKDUMPPATH variable can be made persistent by adding an entry for it in your ~/.bash_profile file:

e.g.
$ cat ~/.bash_profile
export PYKDUMPPATH="$PYKDUMPPATH:/cores/crashext/epython/storage"

To automatically load the crash extensions at the start of a crash session, add the following entry in your .crashrc file:

$ cat ~/.crashrc
extend /usr/lib64/crash/extensions/PyKdump/mpykdump.so

Packaging and The Contents of mpykdump.so¶

SYNOPSIS¶

DESCRIPTION¶

EXAMPLES¶

ENVIRONMENT¶

SEE ALSO¶

Table of Contents

Previous topic

Next topic

This Page