Framework - Extension/ directory

This directory contains C source files to implement Python bindings to internal subroutines of crash and GDB.

In addition, its Makefile is used to build the binary distribution of PyKdump - a single self-sufficient file mpykdump.so that can be loaded as an extension in the crash environment.

Directory Structure

configure         - configuration script to generate extra Makefiles
epython.c         - initialization and invoking programs
functions.c       - generic functions and bindings to crash internals
gdbspec.c         - bindings to GDB internals
Makefile          - main Makefile
makestdlib.py     - create and package a subset of Python Standard Library
minpylib-3.N.lst  - list of files from Python 3.N Standard Library to include
pyconf.py         - getting info about Python used for builds
pykdump.h         - main header
pyparsing.py      - a third-party Python module to implement parsers
Setup.local-3.N   - used when you build Python from sources
testmod/          - a test DLKM, used for framework testing
writeREADME.py    - generates README with the contents of mpykdump.so

Dependency on Python and Crash Versions

Usually there is no need to change anything when a new minor version of crash or Python is released. But when there is a new major release of either crash or Python, it is quite possible that the C-module will need some changes.

Dependency on crash Version

There are four types of changes in crash per se that might need changes in the C-module. In addition, we depend on the GDB emebedded in crash.

Generating Makefiles during Configuration

Before running the make command to build the extension, you need to run the configure script to generate Makefiles.

In particular, we need to get the crash version (to be used in reports and checks), GDB version, and target. The crash version used to be specified directly in its Makefile until recently:

VERSION=7.2.8

but after crash project migration to Github, it is not there anymore:

VERSION=

As a result, the pyconf.py script that was used to extract version/target from this Makefile needed to be modified, so now we rely on build_data.c:

char *build_command = "crash";
char *build_data = "Tue Sep  1 08:12:55 EDT 2020 by uid=1000(alexs) on zbook";
char *build_target = "X86_64";
char *build_version = "7.2.8";
char *compiler_version = "gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0";

But it is always possible that in the future it will change again.

Internal subroutines/variables/macros

The C-module depends on several internal crash subroutines, variables, and macros, providing wrappers (usually called Python bindings) for them so that these subroutines can be accessed from Python. Examples of such subroutines/macros:

  • symbol_exists(symbol)

  • MEMBER_SIZE(name, member)

Some constants:

  • KVADDR/UVADDR/PHYSADDR

  • PAGESIZE/PAGE_CACHE_SHIFT

When a new version of crash is released, it is possible that some of these subroutines change or become unavailable (this has happened at least once).

An example: recent versions of crash-7 have a set_error() subroutine that can be used to redirect/suppress internal crash/GDB errors and messages.

This subroutine is used in the PyKdump C-module. As a result, if you try to load mpykdump.so with crash as shipped on RHEL7, you will see:

extend: /usr/local/lib/mpykdump64.so: undefined symbol: set_error

To workaround this, we can use weak symbols (a feature of GCC), like this:

 // Weak symbols as needed for compatibility with older versions of crash
extern FILE * set_error(char *target) __attribute__ ((weak));

<snip>
// Python bindings to crash internal subroutine set_error()
// You provide a single argument - a string with target name, and
// subroutine returns old target name

static PyObject *
py_crash_set_error(PyObject *self, PyObject *pyargs) {
  char *target;
  PyObject *rc;

  // If set_error() is unavailable, do nothing and return None
  if (!set_error) {
    Py_INCREF(Py_None);
    return Py_None;
  }

The logic is as follows: if 'set_error' cannot be resolved when loading the .so, it will be NULL. After that, we check whether it is NULL and if yes, return None without doing anything. In more complex cases, we might use our own subroutine instead, or print a warning.

Signal Handlers and Executing crash Commands

crash has its own signal handlers for several signals, e.g. SIGINT. When we execute Python code, Python has its own signal handlers. To make everything work properly, we need to save/restore signal handlers when executing Python.

PyKdump provides several commands to execute a crash built-in command and return the result as a string. The logic is rather complicated:

  • We need to feed a string as a command line for crash to execute, modify file descriptors to retrieve output, and after command completion do some cleanup.

  • While executing the crash builtin, we need to use its own signal handler, and after that install the Python signal handler again.

Dependency on GDB

crash is built on top of GDB, and to access symbolic information (such as struct/union definitions) we need to execute internal GDB subroutines. New major releases of crash are usually rebased on a newer major GDB version.

As a result, some enumeration definitions (used by GDB) can change (this has happened twice), subroutine signatures can change, and GDB cleanup/error processing can change.

So if PyKdump built on top of a new major crash does not work properly, be ready to look not only in crash sources but GDB sources as well (provided with patches in the crash tarfile).

Dependency on Python Version

When there is a new major release of Python, three things might need changing:

  • The contents of Setup.local used to build Python from sources

  • The list of Python Standard Library subroutines to be included

  • The way to initialize the Python environment and execute Python code

Setup.local and minpylib-3.N.lst

These two files are often updated together. If you decide to include another module from the Python Standard Library, this often (but not always) requires linking statically another C-module included in the Python sources distribution. This has nothing to do with a Python major version change.

But it is not unusual that a new major release of Python rearranges the library, so that you will need to change the contents of minpylib-3.N.lst to make things work. This happened e.g. while migrating from Python-3.6 to Python-3.7.

At this time, the official PyKdump releases are built using Python 3.9. Some code for Python 3.10 enablement has been included, but this is not yet complete.

Python Initialization and Code Execution

The Python environment is initialized only once, while loading the extension. Initialization subroutines - part of the Python C-API - are regularly improved and older ones are sometimes obsoleted. This means that we might need to modify the logic of the _init_python() subroutine (defined in epython.c).

There are two sources of PyKdump Python code to execute:

  • from real files (either user-developed programs or local GIT repo)

  • from the ZIP-file part of the binary mpykdump.so module

Once again, new major releases of Python regularly improve the existing C-API subroutines, while some old ones are being obsoleted.