Analyzing process/task details from vmcore (taskinfo)¶
The taskinfo program available in the PyKdump framework can be used to quickly analyze process/task details. It allows users to list processes by various options (memory usage, last execution time, or start time), run specific crash commands on processes in Running or Uninterruptible state, retrieve process namespace details, etc.
Options provided by 'taskinfo':
crash> taskinfo -h
Usage: taskinfo [options]
Options:
-h, --help show this help message and exit
-v verbose output
--summary Summary
--times Task start times
--lastsched Last scheduled on each CPU
--hang Equivalent to '-r --task=UN' and prints just several newest and several oldest threads
--maxpids=MAXPIDS Maximum number of PIDs to print for --hang and --mem
--pidinfo=PIDINFO Display details for a given PID. You can specify PID or addr of task_struct
--taskfilter=TASKFILTER
A list of 2-letter task states to print, e.g. UN
--pstree Emulate user-space 'pstree' output
-r, --recent Reverse order while sorting by ran_ago
--cmd=CMD For each listed task, display output of specified command, e.g. '--cmd files'
--memory Print a summary of memory usage by tasks
--ns Print info about namespaces
--version Print program version and exit
** Execution took 0.03s (real) 0.03s (CPU)
crash>
Maximum number of PIDs to print for --hang and --mem (--maxpids)
Use reverse order while sorting and printing task details (-r)
Summary (--summary)¶
The '--summary' option prints a brief summary about recently executed tasks, the number of processes in different states, and threads running in their own namespaces:
crash> taskinfo --summary
Number of Threads That Ran Recently
-----------------------------------
last second 27
last 5s 59
last 60s 71
----- Total Numbers of Threads per State ------
TASK_INTERRUPTIBLE 439
TASK_RUNNING 3
TASK_UNINTERRUPTIBLE 21
+++WARNING+++ There are 8 threads running in their own namespaces
Use 'taskinfo --ns' to get more details
******************************************************************************
************************ A Summary Of Problems Found *************************
******************************************************************************
-------------------- A list of all +++WARNING+++ messages --------------------
There are 8 threads running in their own namespaces
Use 'taskinfo --ns' to get more details
------------------------------------------------------------------------------
** Execution took 0.04s (real) 0.03s (CPU)
crash>
Show task start times (--times)¶
To show the start time of each task, use the '--times' option. This will show the date and time stamp using the current TZ setting. Tasks are displayed in oldest to newest order unless '-r' is also used:
crash> taskinfo --times | head -20
=== Tasks in order of start time ===
PID CMD Start Time
------- ------------ -------------------
0 swapper/0 2022-07-30 12:06:35 UID=0
1 systemd 2022-07-30 12:06:35 UID=0
2 kthreadd 2022-07-30 12:06:35 UID=0
3 rcu_gp 2022-07-30 12:06:35 UID=0
4 rcu_par_gp 2022-07-30 12:06:35 UID=0
6 kworker/0:0H 2022-07-30 12:06:35 UID=0
10 mm_percpu_wq 2022-07-30 12:06:35 UID=0
11 ksoftirqd/0 2022-07-30 12:06:35 UID=0
12 rcu_sched 2022-07-30 12:06:35 UID=0
:
Show the last scheduled task on each CPU (--lastsched)¶
To show the last scheduled task on each CPU, use the '--lastsched' option. This will show the most recently scheduled tasks first unless '-r' is also used:
crash> taskinfo --lastsched
PID CMD CPU Ran ms ago STATE
-------- ------------ -- ------------- -----
25046 FlushResourcesH 12 0 RU (tgid=24749) UID=26061
2826 rte-sensor 2 0 IN (tgid=2803) UID=0
> 10741 perfalarm 29 0 RU UID=0
26974 PoolThread 3 0 IN (tgid=24749) UID=26061
> 24806 FlushResourcesT 17 13 RU (tgid=24749) UID=26061
> 25901 PoolThread 31 59752 RU (tgid=24408) UID=26061
> 1 systemd 19 59762 RU UID=0
> 15 rcu_preempt 11 180020 RU UID=0
> 1863 vmtoolsd 25 180257 RU UID=0
:
Print hung task information (--hang)¶
To view a summary of hung tasks, use the '--hang' option:
crash> taskinfo --hang
=== Tasks in reverse order, scheduled recently first (5 tasks skipped) ===
PID CMD CPU Ran ms ago STATE
-------- ------------ -- ------------- -----
11643 dd 0 1 UN UID=0
11644 dd 0 10 UN UID=0
11646 dd 1 28 UN UID=0
308 scsi_eh_0 2 523 UN UID=0
355 scsi_eh_7 2 2598 UN UID=0
7018 jbd2/dm-31-8 2 162648 UN UID=0
7030 jbd2/dm-35-8 1 162649 UN UID=0
7033 jbd2/dm-36-8 3 162650 UN UID=0
<snip>
7314 kworker/u8:1 1 167673 UN UID=0
11459 rm 2 167950 UN UID=0
11449 rm 0 168080 UN UID=0
11428 rm 0 168084 UN UID=0
11450 rm 2 168106 UN UID=0
11451 rm 0 168140 UN UID=0
11446 rm 2 168172 UN UID=0
11452 rm 3 172493 UN UID=0
** Execution took 0.07s (real) 0.06s (CPU)
crash>
Users can get more verbose information about the hung tasks with the '-v' (verbose) flag:
crash> taskinfo --hang -v
=== Tasks in reverse order, scheduled recently first (5 tasks skipped) ===
PID CMD CPU Ran ms ago STATE
-------- ------------ -- ------------- -----
11643 dd 0 1 UN UID=0
PID=11643 CPU=0 CMD=dd
#0 __schedule+0x3dc
#1 schedule+0x29
#2 schedule_timeout+0x239
#3 io_schedule_timeout+0xad
#4 io_schedule+0x18
#5 bit_wait_io+0x11
#6 __wait_on_bit+0x65
#7 wait_on_page_bit+0x81
#8 __filemap_fdatawait_range+0x111
#9 filemap_fdatawait_range+0x14
#10 filemap_write_and_wait_range+0x56
#11 nfs4_file_fsync+0x81
#12 vfs_fsync+0x2b
#13 nfs4_file_flush+0x5e
#14 filp_close+0x34
#15 __close_fd+0x78
#16 sys_close+0x23
#17 system_call_fastpath+0x16, 477 bytes of data
------------------------------------------------------------------------------
11644 dd 0 10 UN UID=0
PID=11644 CPU=0 CMD=dd
#0 __schedule+0x3dc
#1 schedule+0x29
#2 schedule_timeout+0x239
#3 io_schedule_timeout+0xad
#4 io_schedule+0x18
#5 bit_wait_io+0x11
#6 __wait_on_bit+0x65
#7 wait_on_page_bit+0x81
#8 __filemap_fdatawait_range+0x111
#9 filemap_fdatawait_range+0x14
#10 filemap_write_and_wait_range+0x56
#11 nfs4_file_fsync+0x81
#12 vfs_fsync+0x2b
#13 nfs4_file_flush+0x5e
#14 filp_close+0x34
#15 __close_fd+0x78
#16 sys_close+0x23
#17 system_call_fastpath+0x16, 477 bytes of data
------------------------------------------------------------------------------
11646 dd 1 28 UN UID=0
PID=11646 CPU=1 CMD=dd
#0 __schedule+0x3dc
#1 schedule+0x29
#2 schedule_timeout+0x239
#3 io_schedule_timeout+0xad
#4 io_schedule+0x18
#5 bit_wait_io+0x11
#6 __wait_on_bit+0x65
#7 wait_on_page_bit+0x81
#8 __filemap_fdatawait_range+0x111
#9 filemap_fdatawait_range+0x14
#10 filemap_write_and_wait_range+0x56
#11 nfs4_file_fsync+0x81
#12 vfs_fsync+0x2b
#13 nfs4_file_flush+0x5e
#14 filp_close+0x34
#15 __close_fd+0x78
#16 sys_close+0x23
#17 system_call_fastpath+0x16, 477 bytes of data
[...]
Maximum number of PIDs to print for --hang and --mem (--maxpids)¶
The '--hang' option by default prints details for every process, but users can restrict the output to a specific number of processes by using the '--maxpids' option.
For example, using '--maxpids=5' will print the details of the first 5 and last 5 processes sorted by last execution time:
crash> taskinfo --hang --maxpids=5
=== Tasks in reverse order, scheduled recently first (11 tasks skipped) ===
PID CMD CPU Ran ms ago STATE
-------- ------------ -- ------------- -----
11643 dd 0 1 UN UID=0
11644 dd 0 10 UN UID=0
11646 dd 1 28 UN UID=0
308 scsi_eh_0 2 523 UN UID=0
355 scsi_eh_7 2 2598 UN UID=0
<snip>
11428 rm 0 168084 UN UID=0
11450 rm 2 168106 UN UID=0
11451 rm 0 168140 UN UID=0
11446 rm 2 168172 UN UID=0
11452 rm 3 172493 UN UID=0
** Execution took 0.07s (real) 0.07s (CPU)
crash>
Similarly, when the '--maxpids=N' option is used with '--mem', it will restrict the output to the specified number of processes:
crash> taskinfo --mem --maxpids=5
==== First 5 Tasks reverse-sorted by RSS+SHM ====
PID= 6622 CMD=gnome-shell RSS=0.089 Gb
PID= 5770 CMD=firewalld RSS=0.027 Gb
PID= 6581 CMD=X RSS=0.023 Gb
PID= 6685 CMD=gnome-settings- RSS=0.021 Gb
PID= 4624 CMD=multipathd RSS=0.019 Gb
==== First 5 Tasks Reverse-sorted by RSS only ====
PID= 6622 CMD=gnome-shell RSS=0.089 Gb
PID= 5770 CMD=firewalld RSS=0.027 Gb
PID= 6581 CMD=X RSS=0.023 Gb
PID= 6685 CMD=gnome-settings- RSS=0.021 Gb
PID= 4624 CMD=multipathd RSS=0.019 Gb
=== Total Memory in RSS 0.497 Gb
=== Total Memory in SHM 0.000 Gb
** Execution took 0.21s (real) 0.10s (CPU), Child processes: 0.10s
crash>
Display the details for a given PID (--pidinfo)¶
To view more detailed information about a particular process, use '--pidinfo'. It prints the address of the 'task_struct' associated with the given PID, 'uid' and 'gid' credentials with which the process was executed, and resource limits:
crash> taskinfo --pidinfo=355
---- 355(UN) <struct task_struct 0xffff880211d16eb0> scsi_eh_7
cpu 2
-- Parent: 2 kthreadd
-- Credentials
uid=18446612133217246980 gid=18446612133217246984
suid=18446612133217246988 sgid=18446612133217246992
euid=18446612133217246996 egid=18446612133217247000
fsuid=18446612133217247004 fsgid=18446612133217247008
--user_struct <struct user_struct 0xffffffff81a345a0>
processes=401 files=0 sigpending=0
--group_info <struct group_info 0xffffffff81a3dd80>
[]
-- Rlimits:
00 (RLIMIT_CPU) cur=INFINITY max=INFINITY
01 (RLIMIT_FSIZE) cur=INFINITY max=INFINITY
02 (RLIMIT_DATA) cur=INFINITY max=INFINITY
03 (RLIMIT_STACK) cur=8388608 max=INFINITY
04 (RLIMIT_CORE) cur=0 max=INFINITY
05 (RLIMIT_RSS) cur=INFINITY max=INFINITY
06 (RLIMIT_NPROC) cur=30294 max=30294
07 (RLIMIT_NOFILE) cur=1024 max=4096
08 (RLIMIT_MEMLOCK) cur=65536 max=65536
09 (RLIMIT_AS) cur=INFINITY max=INFINITY
10 (RLIMIT_LOCKS) cur=INFINITY max=INFINITY
11 (RLIMIT_SIGPENDING) cur=30294 max=30294
12 (RLIMIT_MSGQUEUE) cur=819200 max=819200
13 (RLIMIT_NICE) cur=0 max=0
14 (RLIMIT_RTPRIO) cur=0 max=0
15 (RLIMIT_RTTIME) cur=INFINITY max=INFINITY
--- thread_info <struct thread_info 0xffff880035200000>
** Execution took 0.12s (real) 0.12s (CPU)
crash>
Filter the list of processes by state (--taskfilter)¶
To get a list of processes filtered by state, use '--taskfilter'.
For example, the below command will list only the processes in running (RU) state:
crash> taskinfo --taskfilter=RU
=== Tasks in PID order, grouped by Thread Group leader ===
PID CMD CPU Ran ms ago STATE
-------- ------------ -- ------------- -----
> 0 swapper/0 0 1172313 RU UID=0
> 6868 bash 1 0 RU UID=0
7598 kworker/1:0 1 340 RU UID=0
** Execution took 0.05s (real) 0.05s (CPU)
crash>
To get a list of processes in Uninterruptible (UN) state:
crash> taskinfo --taskfilter=UN
=== Tasks in PID order, grouped by Thread Group leader ===
PID CMD CPU Ran ms ago STATE
-------- ------------ -- ------------- -----
308 scsi_eh_0 2 523 UN UID=0
355 scsi_eh_7 2 2598 UN UID=0
7015 jbd2/dm-30-8 0 167657 UN UID=0
7018 jbd2/dm-31-8 2 162648 UN UID=0
7021 jbd2/dm-32-8 1 167641 UN UID=0
7024 jbd2/dm-33-8 0 167641 UN UID=0
7027 jbd2/dm-34-8 1 167641 UN UID=0
7030 jbd2/dm-35-8 1 162649 UN UID=0
7033 jbd2/dm-36-8 3 162650 UN UID=0
7314 kworker/u8:1 1 167673 UN UID=0
7317 kworker/u8:4 3 167642 UN UID=0
11428 rm 0 168084 UN UID=0
11446 rm 2 168172 UN UID=0
11449 rm 0 168080 UN UID=0
11450 rm 2 168106 UN UID=0
11451 rm 0 168140 UN UID=0
11452 rm 3 172493 UN UID=0
11459 rm 2 167950 UN UID=0
11643 dd 0 1 UN UID=0
11644 dd 0 10 UN UID=0
11646 dd 1 28 UN UID=0
** Execution took 0.07s (real) 0.07s (CPU)
crash>
Emulate a user-space 'pstree' output (--pstree)¶
The '--pstree' option will print the parent child relationship between the processes.
This output is similar to the Linux 'pstree' command:
crash> taskinfo --pstree
systemd(1)-+-ModemManager(5709)---2*[{ModemManager}]
|-NetworkManager(5788)-+-dhclient(5912)
| `-2*[{NetworkManager}]
|-abrt-watch-log(5718)
|-abrt-watch-log(5720)
|-abrtd(5714)
|-accounts-daemon(5706)---2*[{accounts-daemon}]
|-alsactl(5695)
|-at-spi-bus-laun(6607)-+-dbus-daemon(6612)
| `-3*[{at-spi-bus-laun}]
|-at-spi2-registr(6614)---2*[{at-spi2-registr}]
|-atd(6131)
|-auditd(5670)-+-audispd(5672)-+-sedispatch(5674)
| | `-{audispd}
| `-{auditd}
|-avahi-daemon(5699)---avahi-daemon(5703)
|-chronyd(5781)
|-colord(6713)---2*[{colord}]
|-crond(6130)
|-cupsd(6107)
|-dbus-daemon(5728)
|-dbus-daemon(6604)
|-dbus-launch(6603)
|-dnsmasq(6443)---dnsmasq(6444)
|-firewalld(5770)---{firewalld}
|-gdm(6129)-+-X(6581)---{X}
| |-gdm-session-wor(6593)-+-gnome-session-b(6597)-+-gnome-settings-(6685)---4*[{gnome-settings-}]
| | | |-gnome-shell(6622)-+-ibus-daemon(6666)-+-ibus-dconf(6692)---3*[{ibus-dconf}]
| | | | | |-ibus-engine-sim(6729)---2*[{ibus-engine-sim}]
| | | | | `-2*[{ibus-daemon}]
| | | | `-6*[{gnome-shell}]
| | | `-3*[{gnome-session-b}]
| | `-2*[{gdm-session-wor}]
| `-3*[{gdm}]
|-gssproxy(5738)---5*[{gssproxy}]
|-ibus-x11(6695)---2*[{ibus-x11}]
|-iobkp0.sh(7235)---rm(11450)
|-iobkp1.sh(7236)---rm(11451)
|-iodata0.sh(7237)---rm(11449)
|-iodata1.sh(7238)---rm(11446)
|-iodata2.sh(7239)---rm(11459)
|-iodata3.sh(7240)---rm(11452)
|-iodata4.sh(7241)---rm(11428)
|-ionfs1.sh(7242)---dd(11646)
|-ionfs2.sh(7243)---dd(11643)
|-ionfs3.sh(7244)---dd(11644)
|-irqbalance(5725)
|-ksmtuned(5773)---sleep(11584)
|-libvirtd(6119)---15*[{libvirtd}]
|-lsmd(5698)
|-lvmetad(4614)
|-master(6506)-+-pickup(6507)
| `-qmgr(6508)
|-mcelog(5779)
|-multipathd(4624)---31*[{multipathd}]
|-packagekitd(6678)---2*[{packagekitd}]
|-polkitd(5710)---5*[{polkitd}]
|-pulseaudio(6650)---2*[{pulseaudio}]
|-rhnsd(6204)
|-rhsmcertd(6117)
|-rngd(5702)
|-rsyslogd(5707)---2*[{rsyslogd}]
|-rtkit-daemon(5700)---2*[{rtkit-daemon}]
|-smartd(5696)
|-sshd(6106)-+-sshd(6777)---bash(6792)
| |-sshd(6824)---bash(6830)---journalctl(11478)
| |-sshd(6862)---bash(6868)
| |-sshd(6900)---bash(6906)
| `-sshd(6938)---bash(6952)
|-systemd-journal(4578)
|-systemd-logind(5711)
|-systemd-udevd(4618)
|-tuned(6108)---4*[{tuned}]
|-upowerd(6627)---2*[{upowerd}]
|-wpa_supplicant(6677)
`-xdg-permission-(6670)---2*[{xdg-permission-}]
** Execution took 0.05s (real) 0.04s (CPU)
crash>
Use reverse order while sorting and printing task details (-r)¶
The 'taskinfo' program by default lists the process details sorted as per their PID. The '-r' option allows users to sort process details as per their last execution time. This option can also be used to reverse the sort order for the '--times' and '--lastsched' options:
crash> taskinfo |head -10
=== Tasks in PID order, grouped by Thread Group leader ===
PID CMD CPU Ran ms ago STATE
-------- ------------ -- ------------- -----
> 0 swapper/0 0 1172313 RU UID=0
1 systemd 0 4303 IN UID=0
2 kthreadd 2 176550 IN UID=0
3 ksoftirqd/0 0 113 IN UID=0
5 kworker/0:0H 0 0 IN UID=0
7 migration/0 0 3044 IN UID=0
8 rcu_bh 0 1172282 IN UID=0
crash>
crash> taskinfo -r|head -10
=== Tasks in reverse order, scheduled recently first ===
PID CMD CPU Ran ms ago STATE
-------- ------------ -- ------------- -----
> 6868 bash 1 0 RU UID=0
7119 kworker/3:0 3 0 IN UID=0
4069 kworker/1:1H 1 0 IN UID=0
874 kworker/2:1H 2 0 IN UID=0
5 kworker/0:0H 0 0 IN UID=0
23 ksoftirqd/3 3 0 IN UID=0
5702 rngd 3 0 IN UID=0
crash>
Run a specific command for each task (--cmd)¶
While analyzing a vmcore, it is often useful to run specific crash commands such as 'bt' or 'bt -f' on a list of processes. The '--cmd' option can be used to run a crash command on a list of processes.
For example, using 'taskinfo --cmd bt' will run the 'bt' command on each process in the list:
crash> taskinfo --cmd bt
=== Tasks in PID order, grouped by Thread Group leader ===
PID CMD CPU Ran ms ago STATE
-------- ------------ -- ------------- -----
> 0 swapper/0 0 1172313 RU UID=0
crash> bt 0
PID: 0 TASK: ffffffff81a02480 CPU: 0 COMMAND: "swapper/0"
#0 [ffff88021ea08e48] crash_nmi_callback at ffffffff8104fd11
#1 [ffff88021ea08e58] nmi_handle at ffffffff816b0c57
#2 [ffff88021ea08eb0] do_nmi at ffffffff816b0e8d
#3 [ffff88021ea08ef0] end_repeat_nmi at ffffffff816b00b9
[exception RIP: intel_idle+244]
RIP: ffffffff816adb04 RSP: ffffffff819efe28 RFLAGS: 00000046
RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000001
RDX: 0000000000000000 RSI: ffffffff819effd8 RDI: 0000000000000000
RBP: ffffffff819efe58 R8: 00000000000003e3 R9: 0000000000000018
R10: 00000000000003e2 R11: 0000014a0e39c880 R12: ffffffff819effd8
R13: 0000000000000002 R14: 0000000000000001 R15: ffffffff81ab8a28
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#4 [ffffffff819efe28] intel_idle at ffffffff816adb04
#5 [ffffffff819efe60] cpuidle_enter_state at ffffffff81529e30
#6 [ffffffff819efe98] cpuidle_idle_call at ffffffff81529f88
#7 [ffffffff819efed8] arch_cpu_idle at ffffffff81034eee
#8 [ffffffff819efee8] cpu_startup_entry at ffffffff810e9aba
#9 [ffffffff819eff30] rest_init at ffffffff81694f17
#10 [ffffffff819eff40] start_kernel at ffffffff81b500e1
#11 [ffffffff819eff88] x86_64_start_reservations at ffffffff81b4f66b
#12 [ffffffff819eff98] x86_64_start_kernel at ffffffff81b4f7bc
PID: 0 TASK: ffff88017ce79fa0 CPU: 1 COMMAND: "swapper/1"
#0 [ffff88017ceafe48] __schedule at ffffffff816ab2ac
#1 [ffff88017ceafed0] schedule_preempt_disabled at ffffffff816ac7c9
#2 [ffff88017ceafee0] cpu_startup_entry at ffffffff810e9afa
#3 [ffff88017ceaff28] start_secondary at ffffffff81051b96
PID: 0 TASK: ffff88017ce7af70 CPU: 2 COMMAND: "swapper/2"
#0 [ffff88021eb08e48] crash_nmi_callback at ffffffff8104fd11
#1 [ffff88021eb08e58] nmi_handle at ffffffff816b0c57
#2 [ffff88021eb08eb0] do_nmi at ffffffff816b0e8d
#3 [ffff88021eb08ef0] end_repeat_nmi at ffffffff816b00b9
[exception RIP: intel_idle+244]
RIP: ffffffff816adb04 RSP: ffff88017ceb3e20 RFLAGS: 00000046
RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000001
RDX: 0000000000000000 RSI: ffff88017ceb3fd8 RDI: 0000000000000002
RBP: ffff88017ceb3e50 R8: 00000000000003d4 R9: 0000000000000020
R10: 00000000000007cd R11: 0000014a0e2a8640 R12: ffff88017ceb3fd8
R13: 0000000000000002 R14: 0000000000000001 R15: ffffffff81ab8a28
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#4 [ffff88017ceb3e20] intel_idle at ffffffff816adb04
#5 [ffff88017ceb3e58] cpuidle_enter_state at ffffffff81529e30
#6 [ffff88017ceb3e90] cpuidle_idle_call at ffffffff81529f88
#7 [ffff88017ceb3ed0] arch_cpu_idle at ffffffff81034eee
#8 [ffff88017ceb3ee0] cpu_startup_entry at ffffffff810e9aba
#9 [ffff88017ceb3f28] start_secondary at ffffffff81051b96
PID: 0 TASK: ffff88017ce7bf40 CPU: 3 COMMAND: "swapper/3"
#0 [ffff88021eb88e48] crash_nmi_callback at ffffffff8104fd11
#1 [ffff88021eb88e58] nmi_handle at ffffffff816b0c57
#2 [ffff88021eb88eb0] do_nmi at ffffffff816b0e8d
#3 [ffff88021eb88ef0] end_repeat_nmi at ffffffff816b00b9
[exception RIP: intel_idle+244]
RIP: ffffffff816adb04 RSP: ffff88017ceb7e20 RFLAGS: 00000046
RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000001
RDX: 0000000000000000 RSI: ffff88017ceb7fd8 RDI: 0000000000000003
RBP: ffff88017ceb7e50 R8: 0000000000000137 R9: ffff88021eb97a80
R10: 7fffffffffffffff R11: 7fffffffffffffff R12: ffff88017ceb7fd8
R13: 0000000000000001 R14: 0000000000000000 R15: ffffffff81ab89d0
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#4 [ffff88017ceb7e20] intel_idle at ffffffff816adb04
#5 [ffff88017ceb7e58] cpuidle_enter_state at ffffffff81529e30
#6 [ffff88017ceb7e90] cpuidle_idle_call at ffffffff81529f88
#7 [ffff88017ceb7ed0] arch_cpu_idle at ffffffff81034eee
#8 [ffff88017ceb7ee0] cpu_startup_entry at ffffffff810e9aba
#9 [ffff88017ceb7f28] start_secondary at ffffffff81051b96
------------------------------------------------------------------------------
[...]
This option can also be combinted with '--taskfilter' to run the specific crash command on the list of processes in Running or Uninterruptible state:
crash> taskinfo --cmd "bt -f" --taskfilter=RU
=== Tasks in PID order, grouped by Thread Group leader ===
PID CMD CPU Ran ms ago STATE
-------- ------------ -- ------------- -----
> 0 swapper/0 0 1172313 RU UID=0
crash> bt -f 0
PID: 0 TASK: ffffffff81a02480 CPU: 0 COMMAND: "swapper/0"
#0 [ffff88021ea08e48] crash_nmi_callback at ffffffff8104fd11
ffff88021ea08e50: ffff88021ea08ea8 ffffffff816b0c57
#1 [ffff88021ea08e58] nmi_handle at ffffffff816b0c57
ffff88021ea08e60: 0000000000000000 ffff88021ea08ef8
ffff88021ea08e70: ffffffff81a1a700 a89424830a090e96
ffff88021ea08e80: ffff88021ea08ef8 ffffffff819effd8
ffff88021ea08e90: 0000000000000000 ffffffff819effd8
ffff88021ea08ea0: 00000000ffffffff ffff88021ea08ee8
ffff88021ea08eb0: ffffffff816b0e8d
#2 [ffff88021ea08eb0] do_nmi at ffffffff816b0e8d
ffff88021ea08eb8: 0000000000000000 0000000000000001
ffff88021ea08ec8: 00007f451748c000 0000000000000001
ffff88021ea08ed8: 000000009845a000 ffffffff81ab8a28
ffff88021ea08ee8: ffffffff819efe58 ffffffff816b00b9
#3 [ffff88021ea08ef0] end_repeat_nmi at ffffffff816b00b9
[exception RIP: intel_idle+244]
RIP: ffffffff816adb04 RSP: ffffffff819efe28 RFLAGS: 00000046
RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000001
RDX: 0000000000000000 RSI: ffffffff819effd8 RDI: 0000000000000000
[...]
Print a summary of memory usage by tasks (--memory)¶
To view the memory usage by processes, use '--memory':
crash> taskinfo --memory
==== First 8 Tasks reverse-sorted by RSS+SHM ====
PID= 6622 CMD=gnome-shell RSS=0.089 Gb
PID= 5770 CMD=firewalld RSS=0.027 Gb
PID= 6581 CMD=X RSS=0.023 Gb
PID= 6685 CMD=gnome-settings- RSS=0.021 Gb
PID= 4624 CMD=multipathd RSS=0.019 Gb
PID= 6108 CMD=tuned RSS=0.016 Gb
PID= 5912 CMD=dhclient RSS=0.015 Gb
PID= 6119 CMD=libvirtd RSS=0.014 Gb
==== First 8 Tasks Reverse-sorted by RSS only ====
PID= 6622 CMD=gnome-shell RSS=0.089 Gb
PID= 5770 CMD=firewalld RSS=0.027 Gb
PID= 6581 CMD=X RSS=0.023 Gb
PID= 6685 CMD=gnome-settings- RSS=0.021 Gb
PID= 4624 CMD=multipathd RSS=0.019 Gb
PID= 6108 CMD=tuned RSS=0.016 Gb
PID= 5912 CMD=dhclient RSS=0.015 Gb
PID= 6119 CMD=libvirtd RSS=0.014 Gb
=== Total Memory in RSS 0.497 Gb
=== Total Memory in SHM 0.000 Gb
** Execution took 0.50s (real) 0.29s (CPU), Child processes: 0.09s
crash>
This option can be combined with '--maxpids=2' to restrict the output to a specified number of processes:
crash> taskinfo --memory --maxpids=2
==== First 2 Tasks reverse-sorted by RSS+SHM ====
PID= 6622 CMD=gnome-shell RSS=0.089 Gb
PID= 5770 CMD=firewalld RSS=0.027 Gb
==== First 2 Tasks Reverse-sorted by RSS only ====
PID= 6622 CMD=gnome-shell RSS=0.089 Gb
PID= 5770 CMD=firewalld RSS=0.027 Gb
=== Total Memory in RSS 0.497 Gb
=== Total Memory in SHM 0.000 Gb
** Execution took 0.19s (real) 0.10s (CPU), Child processes: 0.10s
crash>
Print info about namespaces (--ns)¶
To view more details about the namespaces associated with processes, use '--ns':
crash> taskinfo --ns
******************Non-standard Namespaces*******************
~~~~~~~~~~~~~~Namespaces Associated with PID~~~~~~~~~~~~~~
.......<struct nsproxy 0xffff88021ed1f000>........
['mnt_ns']
PID=27 <struct task_struct 0xffff88017c860fd0> CMD=kdevtmpfs
.......<struct nsproxy 0xffff88020fc45000>........
['mnt_ns']
PID=4618 <struct task_struct 0xffff8802126e0fd0> CMD=systemd-udevd
.......<struct nsproxy 0xffff88021508d4b0>........
['mnt_ns', 'net_ns']
PID=5700 <struct task_struct 0xffff880207e91fa0> CMD=rtkit-daemon
.......<struct nsproxy 0xffff88020d2c9000>........
['mnt_ns']
PID=5781 <struct task_struct 0xffff8800c6f2bf40> CMD=chronyd
.......<struct nsproxy 0xffff88020d2c9060>........
['mnt_ns']
PID=5788 <struct task_struct 0xffff8800cac50000> CMD=NetworkManager
PID=5912 <struct task_struct 0xffff880034cf2f70> CMD=dhclient
.......<struct nsproxy 0xffff88020d325450>........
['mnt_ns']
PID=6107 <struct task_struct 0xffff8800cd092f70> CMD=cupsd
.......<struct nsproxy 0xffff88021508d4e0>........
['mnt_ns']
PID=6713 <struct task_struct 0xffff8800cd20eeb0> CMD=colord
** Execution took 0.22s (real) 0.15s (CPU)
crash>