Debugging Aids ============== There are many ready-to-use PyKdump programs included in its distribution. To use them, you do not need to know anything about PyKdump internals; they can be used just like ``crash`` builtins. This section is mainly of interest to developers, both framework developers and user-program developers. Global Options -------------- There are some options parsed by ``pykdump.API`` itself before arguments are passed to programs. They change global behavior and after processing are stripped from the argument list. The most important of them are:: op.add_option("--timeout", dest="timeout", default=120, action="store", type="int", help="set default timeout for crash commands") op.add_option("--maxel", dest="Maxel", default=10000, action="store", type="int", help="set maximum number of list elements to traverse") op.add_option("--usens", dest="usens", action="store", type="int", help="use namespace of the specified PID") op.add_option("--reload", dest="reload", default=0, action="store_true", help="reload already imported modules from Linuxdump") Running Code From Your Source Tree ---------------------------------- By default, all needed framework modules are loaded directly from the *mpykdump.so* file. You can check how modules will be searched for using:: crash64> epython -p 3.8.3 (default, May 21 2020, 13:02:14) [GCC 4.4.7 20120313 (Red Hat 4.4.7-18)] ['.', '/usr/local/lib/mpykdump64.so/pylib', '/usr/local/lib/mpykdump64.so', '/usr/local/lib/mpykdump64.so/dist-packages'] Please note that before searching for modules in *mpykdump.so* we will check the current directory. This makes it possible to quickly write simple ad-hoc programs, creating a file in your current directory (usually where the vmcore resides). But if you are participating in PyKdump development using its GIT repository, you need to search for code in your local repository copy before searching for it in *mpykdump.so*. You rarely need to rebuild *mpykdump.so*. This is only needed if you want to prepare a new binary file for your organization or if you are working on the PyKdump C-module. To change the search path used by PyKdump, set the shell environment variable *PYKDUMPPATH*, e.g.:: $ export PYKDUMPPATH=~alexs/tools/pykdump/progs:~alexs/tools/pykdump/experiments After that, PyKdump will search these locations *before* using the built-in *mpykdump.so*:: crash64> epython -p 3.8.3 (default, May 21 2020, 13:02:14) [GCC 4.4.7 20120313 (Red Hat 4.4.7-18)] ['.', '/usr/local/lib/mpykdump64.so/pylib', '/home/alexs/tools/pykdump/progs', '/home/alexs/tools/pykdump/experiments', '/usr/local/lib/mpykdump64.so', '/usr/local/lib/mpykdump64.so/dist-packages'] Please note one important exception: we still search ``/usr/local/lib/mpykdump64.so/pylib`` *before* our own path. This section of the binary module contains parts of the Python Standard Library. We need them and do not want to override them (except for some very special cases). Controlling Debugging --------------------- A typical program consists of the main program importing several modules. You can add a ``--debug`` option or something similar to your main program, but how to pass this value to modules? We can always pass it as a parameter to subroutines/methods, but this is not very convenient. Assuming that we have modules ``mod1`` and ``mod2``, we can set ``debug`` variable from the main module using the following approach:: import mod1 mod1.debug = debug import mod2 mod2.debug = debug This works, but is rather ugly. There is another problem: importing a module might run some code. So what if we want to display debugging info at this stage? PyKdump initializes the Python machine only once, when you load the extension. Before running each program, it does some cleanup but preserves many things, such as already cached symbolic information. This significantly improves the performance if you execute the same program multiple times (possibly with different options or arguments). This persistence lets us implement per-module debugging controls, somewhat similar to what the Linux kernel does with procfs/sysfs. In your module, you register an attribute in the following way:: registerModuleAttr("debugDLKM", default=0, help="Debug DLKM debuginfo subroutines") This creates in this module a variable ``debugDLKM``,` initializes it to 0, and registers this option in a session cache. Registration is done during import of this module. There is a programmatic interface to change the values of such attributes *externally*. You can do it easily using the 'pyctl' command, modeled after the Linux command ``sysctl``. While the ``pyctl`` command is included in binary *mypkdump.so*, it is not registered as a top-level command (i.e. it is not visible in ``man`` or ``help``). This is done to avoid confusing normal users. To avoid prepending it with ``epython`` every time, you can create an alias and put it into your *.crashrc* file. Examples:: crash64> epython pyctl -a debugDLKM - Debug DLKM debuginfo subroutines currentvalue=0 default=0 debugDeref currentvalue=0 default=0 debugMP_KW - Debug Monkey-Patching of default keywords currentvalue=0 default=0 debugMemoize - Debug Memoization currentvalue=0 default=0 debugReload - Debug reloading Python modules currentvalue=0 default=0 crash64> epython pyctl debugMemoize=2 ** Execution took 0.00s (real) 0.00s (CPU) crash64> epython pyctl -v debugMemoize debugMemoize - Debug Memoization currentvalue=2 default=0 pykdump.Generic func=__func {'pyctlname': None, 'default': 0, 'type': , 'help': 'Debug Memoization'} If you try to assign a value to a non-registered attribute, you will get an error message:: crash64> epython pyctl debugNN=2 Unknown key: , skipping it Reloading Modules ----------------- As we do not reinitialize the Python machine every time we start a program (as long as we do not exit ``crash``), this means that modules imported during the previous command execution stay in memory and are not reimported every time. This is good for performance, but what if we are working on a module and would like to force its reimport, to accommodate for changes we made? To reload our modules, you just add ``--reload`` to your command. To see what is being reloaded, you need to set ``debugReload``. An example: .. code-block:: text crash64> taskinfo --summ Number of Threads That Ran Recently ----------------------------------- last second 114 last 5s 161 last 60s 266 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 896 TASK_NONINTERACTIVE 2 TASK_RUNNING 2 TASK_STOPPED 1 TASK_TRACED 1 TASK_UNINTERRUPTIBLE 161 ** Execution took 0.95s (real) 0.93s (CPU) crash64> epython pyctl debugReload=2 ** Execution took 0.00s (real) 0.01s (CPU) crash64> taskinfo --summ --reload LinuxDump /home/alexs/tools/pykdump/progs/LinuxDump/__init__.py --reloading LinuxDump LinuxDump.percpu /home/alexs/tools/pykdump/progs/LinuxDump/percpu.py --reloading LinuxDump.percpu LinuxDump.inet /home/alexs/tools/pykdump/progs/LinuxDump/inet/__init__.py --reloading LinuxDump.inet LinuxDump.Time /home/alexs/tools/pykdump/progs/LinuxDump/Time.py --reloading LinuxDump.Time LinuxDump.inet.proto /home/alexs/tools/pykdump/progs/LinuxDump/inet/proto.py --reloading LinuxDump.inet.proto LinuxDump.BTstack /home/alexs/tools/pykdump/progs/LinuxDump/BTstack.py --reloading LinuxDump.BTstack LinuxDump.fs /home/alexs/tools/pykdump/progs/LinuxDump/fs/__init__.py --reloading LinuxDump.fs LinuxDump.Tasks /home/alexs/tools/pykdump/progs/LinuxDump/Tasks.py --reloading LinuxDump.Tasks Number of Threads That Ran Recently ----------------------------------- last second 114 last 5s 161 last 60s 266 ----- Total Numbers of Threads per State ------ TASK_INTERRUPTIBLE 896 TASK_NONINTERACTIVE 2 TASK_RUNNING 2 TASK_STOPPED 1 TASK_TRACED 1 TASK_UNINTERRUPTIBLE 161 ** Execution took 0.39s (real) 0.39s (CPU) At present we do not reload the modules of the framework itself - the contents of the ``pykdump`` directory - as this is difficult to implement properly. (Is it OK to reload the module which is responsible for reloading? We are running code from it at this moment!). So this approach works well for developing user programs, but not the framework itself. Monkey-Patching Default Values for Keywords ------------------------------------------- This started as a fun project (to better understand Python internals) but can be really useful in some cases. If you look at the sources of several list-traversal subroutines, e.g. :func:`~pykdump.API.readList`, you will see that we can optionally specify the maximum number of elements to traverse, otherwise we use a default:: def readList(start, offset=0, *, maxel = _MAXEL, inchead = True, warn = True): There are several subroutines of this type in ``pykdump/highlevel.py`` and they all use as a default the ``_MAXEL`` global variable set in the beginning of this file. The idea is to limit the number of elements in lists to traverse, both to prevent infinite iteration or just to warn you about something unexpected. For example, if normally the list size for some kernel table is not greater than 10000, finding more than this number during iteration probably means memory corruption. But in most cases, we do not specify this keyword argument and expect that the default value is good enough. If we reach a limit during traversal, a warning is printed. To demonstrate this, let us set the default to an unreasonably low value:: crash64> xportshow --summ --maxel=100 ... We have reached the limit while reading a list maxel=100 from sk_for_each <- get_AF_UNIX <- TCPIP_Summarize We print a warning and then the sequence of subroutine calls. There are currently 7 subroutines/methods using this, so how can we change the default externally? Originally we used the following approach (this is not a real subroutine, it's just used for illustration purposes):: def a(a1, maxel = None): maxel = maxel if (maxel is not None) else _MAXEL and in pykdump.API we did:: import highlevel ... highlevel._MAXEL = newvalue Now we do it in the following way:: def setListMaxel(newval): patch_default_kw(getCurrentModule(), 'maxel', newval) A new subroutine ``patch_default_kw(mod, kname, newval)`` replaces all default keyword arguments that have name ``kname`` in functions/methods defined in module ``mod`` with the new value ``newval``. Interactive Development ----------------------- The standard Python interpreter can be run as a REPL (Read-Eval-Print Loop), which allows the user to enter Python code and see the result of running it interactively. PyKdump supports this mode of operation as well, with a built in program that can be run using ``epython repl``:: crash> epython repl PyKdump Embedded REPL: Python 3.8.3 (default, Jul 17 2020, 16:58:36) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39.0.3)] Use Ctrl-D to return to crash from pykdump.API import * >>> from LinuxDump.Tasks import TaskTable >>> tt = TaskTable() >>> tt.getByPid(332005) PID=332005 CMD=awk >>> Returning to crash ** Execution took 32.47s (real) 0.00s (CPU) crash> This allows you to import any Python code included by PyKdump and execute it interactively. For the best results, your mpykdump.so should be compiled with a static readline (as is the default, see the installation instructions). This will allow line editing and command history. The REPL will still work without readline, but it is less convenient to use. When you are done using the REPL, use Ctrl-D to exit it. If you re-open the REPL (by running ``epython repl`` again), your variables will be preserved:: crash> epython repl PyKdump Embedded REPL: Python 3.8.3 (default, Jul 17 2020, 16:58:36) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39.0.3)] Use Ctrl-D to return to crash from pykdump.API import * >>> x = 5 >>> Returning to crash ** Execution took 3.14s (real) 0.00s (CPU) crash> epython repl PyKdump Embedded REPL: Python 3.8.3 (default, Jul 17 2020, 16:58:36) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39.0.3)] Use Ctrl-D to return to crash from pykdump.API import * >>> print(x) 5 Profiling --------- The PyKdump extension has Python :mod:`cProfile` included. Please see the Python documentation for usage details. In the simplest form you can do something like this:: crash64> epython -m cProfile -s tottime /home/alexs/tools/pykdump/progs/xportshow.py --every to profile ``/home/alexs/tools/pykdump/progs/xportshow.py`` execution.