Extracting network and TCP/IP sockets information (xportshow)

The xportshow program available in the PyKdump framework can be used to extract detailed information about network connections, sockets, routing table, TCP retransmissions, various statistics, sysctl network parameters, etc.

Options provided by ‘xportshow’:

crash> xportshow -h
usage: xportshow [-h] [-a] [-v] [-r] [--program PROGRAM] [--pid [PID]] [--netfilter] [--softnet] [--summary] [-s] [-i] [--interface IF1] [--decode DECODE [DECODE ...]]
                 [--port PORT] [-l] [-t] [--tcpstate TCPSTATE] [--retransonly] [-u] [-w] [-x] [--sysctl] [--devpack] [--arp] [--rtcache] [--skbuffhead SKBUFFHEAD]
                 [--netns NETNS] [--version] [--everything]

optional arguments:
  -h, --help            show this help message and exit
  -a                    print all sockets
  -v                    verbose output
  -r                    Print routing table. Adding -v prints all routing tables and policies
  --program PROGRAM     print sockets for cmdname
  --pid [PID]           print sockets for PID
  --netfilter           Print Netfilter Hooks
  --softnet             Print Softnet Queues
  --summary             Print A Summary
  -s, --statistics      Print Statistics
  -i                    Print Interface Info
  --interface IF1       Limit output to the specified interface only
  --decode DECODE [DECODE ...]
                        Decode iph/th/uh
  --port PORT           Limit output to the specified port (src or dst)
  -l, --listening       Print LISTEN sockets only
  -t                    Print TCP Info
  --tcpstate TCPSTATE   Limit display for this state only, e.g. SYN_SENT
  --retransonly         Show only TCP retransmissions
  -u, --udp             Print UDP Info
  -w, --raw             Print RAW Info
  -x, --unix            Print UNIX Info
  --sysctl              Print sysctl info for net.
  --devpack             Print dev_pack info
  --arp                 Print ARP & Neighbouring info
  --rtcache             Print the routing cache
  --skbuffhead SKBUFFHEAD
                        Print sk_buff_head
  --netns NETNS         Set net ns address
  --version             Print program version and exit
  --everything          Run all functions available for regression testing

 ** Execution took   2.35s (real)   2.24s (CPU)
crash>

Show TCP, UDP, ICMP statistics (--statistics)

crash> xportshow --statistics

-------------------- ip_statistics --------------------

                  InReceives               582574
                 InHdrErrors                    0
                InAddrErrors                    0
               ForwDatagrams                    0
             InUnknownProtos                    0
                  InDiscards                    0
                  InDelivers               582572
                 OutRequests              2484936
                 OutDiscards                   84
                 OutNoRoutes                   65
                ReasmTimeout                    0
                  ReasmReqds                    0
                    ReasmOKs                    0
                  ReasmFails                    0
                     FragOKs                    0
                   FragFails                    0
                 FragCreates                    0


-------------------- icmp_statistics --------------------

  not implemented yet

-------------------- tcp_statistics --------------------

                RtoAlgorithm                    1
                      RtoMin                  200
                      RtoMax               120000
                     MaxConn                   -1
                 ActiveOpens                    7
                PassiveOpens                    4
                AttemptFails                    0
                 EstabResets                    0
                   CurrEstab                    6
                      InSegs               582175
                     OutSegs              2483329
                 RetransSegs                 1218
                      InErrs                    0
                     OutRsts                    0
                InCsumErrors                    0


-------------------- udp_statistics --------------------

                 InDatagrams                   42
                     NoPorts                  176
                    InErrors                    0
                OutDatagrams                  208
                RcvbufErrors                    0
                SndbufErrors                    0
                InCsumErrors                    0


-------------------- net_statistics --------------------

              SyncookiesSent                    0
              SyncookiesRecv                    0
            SyncookiesFailed                    0
               EmbryonicRsts                    0
                 PruneCalled                    0
                       [...]
       TCPACKSkippedTimeWait                    0
      TCPACKSkippedChallenge                    0


 ** Execution took   0.03s (real)   0.03s (CPU)
crash>

Decode iph/th/uh (--decode)

The contents of an 'iphdr' structure can be decoded using the '--decode' option as shown below:

crash> xportshow --decode iph 0xffff882fdb99a810
IPv4 <struct iphdr 0xffff882fdb99a810>
tos=0 id=1742 fl=2 frag=0 ttl=1 proto=17 saddr=172.29.23.38 daddr=172.18.101.1

Limit output to the specified port (--port)

TCP socket details printed by xportshow can also be filtered using port numbers. For example, the following output will only show the TCP sockets used by port 2049:

crash> xportshow -a --port 2049
tcp   172.25.0.45:726            172.25.0.43:2049            ESTABLISHED
tcp   172.25.0.45:769            172.25.0.43:2049            ESTABLISHED
[...]

Filtering the TCP sockets used by port 22:

crash> xportshow -a --port 22
tcp6  :::22                      :::*                        LISTEN
tcp   0.0.0.0:22                 0.0.0.0:*                   LISTEN
tcp   172.25.0.45:22             172.25.0.47:58532           ESTABLISHED
tcp   172.25.0.45:22             172.25.0.47:58538           ESTABLISHED
tcp   172.25.0.45:22             172.25.0.47:58530           ESTABLISHED
tcp   172.25.0.45:22             172.25.0.47:58504           ESTABLISHED
udp6  :::908                     :::*                       st=7
udp   0.0.0.0:908                0.0.0.0:*                  st=7
udp6  :::39875                   :::*                       st=7
udp   192.168.122.1:53           0.0.0.0:*                  st=7
udp   0.0.0.0:67                 0.0.0.0:*                  st=7
[...]

List the sockets with specific state (--tcpstate)

The '--tcpstate' option can be used to list the TCP sockets with matching state.

For example, the below command will only list the TCP sockets in ESTABLISHED state:

crash> xportshow --tcpstate ESTABLISHED
tcp   172.25.0.45:22             172.25.0.47:58532           ESTABLISHED
tcp   172.25.0.45:22             172.25.0.47:58538           ESTABLISHED
tcp   172.25.0.45:22             172.25.0.47:58530           ESTABLISHED
tcp   172.25.0.45:726            172.25.0.43:2049            ESTABLISHED
tcp   172.25.0.45:769            172.25.0.43:2049            ESTABLISHED
tcp   172.25.0.45:22             172.25.0.47:58504           ESTABLISHED

 ** Execution took   0.03s (real)   0.02s (CPU)

Similar to other options, users can use '-v' to get more verbose details of the sockets:

crash> xportshow --tcpstate ESTABLISHED -v
------------------------------------------------------------------------------
<struct tcp_sock 0xffff88007a3d4000>                TCP
tcp   172.25.0.45:22             172.25.0.47:58532           ESTABLISHED
    windows: rcv=45696, snd=64128  advmss=1448 rcv_ws=7 snd_ws=7
    nonagle=1 sack_ok=3 tstamp_ok=1
    rmem_alloc=0, wmem_alloc=1
    rx_queue=0, tx_queue=0
    rcvbuf=369280, sndbuf=87040
    rcv_tstamp=7.12 s, lsndtime=7.12 s ago,  RTO=202 ms
------------------------------------------------------------------------------
<struct tcp_sock 0xffff88007a3d47c0>                TCP
tcp   172.25.0.45:22             172.25.0.47:58538           ESTABLISHED
    windows: rcv=42880, snd=64128  advmss=1448 rcv_ws=7 snd_ws=7
    nonagle=1 sack_ok=3 tstamp_ok=1
    rmem_alloc=0, wmem_alloc=1
    rx_queue=0, tx_queue=0
    rcvbuf=369280, sndbuf=87040
    rcv_tstamp=0.00 s, lsndtime=75.70 s ago,  RTO=209 ms
------------------------------------------------------------------------------
<struct tcp_sock 0xffff88007885f640>                TCP
tcp   172.25.0.45:22             172.25.0.47:58530           ESTABLISHED
    windows: rcv=45696, snd=64128  advmss=1448 rcv_ws=7 snd_ws=7
    nonagle=1 sack_ok=3 tstamp_ok=1
    rmem_alloc=0, wmem_alloc=1
    rx_queue=0, tx_queue=0
    rcvbuf=369280, sndbuf=87040
    rcv_tstamp=9.94 s, lsndtime=9.94 s ago,  RTO=201 ms
------------------------------------------------------------------------------
<struct tcp_sock 0xffff88007a3d66c0>                TCP
tcp   172.25.0.45:726            172.25.0.43:2049            ESTABLISHED
    windows: rcv=182272, snd=2323072  advmss=1448 rcv_ws=7 snd_ws=7
    nonagle=1 sack_ok=7 tstamp_ok=1
    rmem_alloc=0, wmem_alloc=1
    rx_queue=0, tx_queue=995976
    rcvbuf=367360, sndbuf=1584128
    rcv_tstamp=17.64 s, lsndtime=0.76 s ago,  RTO=17088 ms
    -- Retransmissions --
       retransmits=6, ca_state=TCP_CA_Loss, 17.36 s since first retransmission
       |user_data| 0xffff8800791be000
[...]

Show only TCP retransmissions (--retransonly)

To check the TCP retransmissions, use '--retransonly':

crash> xportshow --retransonly
tcp   172.25.0.45:726            172.25.0.43:2049            ESTABLISHED
       retransmits=6, ca_state=TCP_CA_Loss, 17.36 s since first retransmission
tcp   172.25.0.45:769            172.25.0.43:2049            ESTABLISHED
       retransmits=6, ca_state=TCP_CA_Loss, 17.43 s since first retransmission

 ** Execution took   0.05s (real)   0.06s (CPU)
crash>

Getting more verbose information with the '-v' option:

crash> xportshow --retransonly -v
------------------------------------------------------------------------------
<struct tcp_sock 0xffff88007a3d66c0>                TCP
tcp   172.25.0.45:726            172.25.0.43:2049            ESTABLISHED
    windows: rcv=182272, snd=2323072  advmss=1448 rcv_ws=7 snd_ws=7
    nonagle=1 sack_ok=7 tstamp_ok=1
    rmem_alloc=0, wmem_alloc=1
    rx_queue=0, tx_queue=995976
    rcvbuf=367360, sndbuf=1584128
    rcv_tstamp=17.64 s, lsndtime=0.76 s ago,  RTO=17088 ms
    -- Retransmissions --
       retransmits=6, ca_state=TCP_CA_Loss, 17.36 s since first retransmission
       |user_data| 0xffff8800791be000
------------------------------------------------------------------------------
<struct tcp_sock 0xffff88007a3d5740>                TCP
tcp   172.25.0.45:769            172.25.0.43:2049            ESTABLISHED
    windows: rcv=182272, snd=1077376  advmss=1448 rcv_ws=7 snd_ws=7
    nonagle=1 sack_ok=7 tstamp_ok=1
    rmem_alloc=0, wmem_alloc=1
    rx_queue=0, tx_queue=815224
    rcvbuf=367360, sndbuf=1296896
    rcv_tstamp=17.65 s, lsndtime=4.88 s ago,  RTO=12928 ms
    -- Retransmissions --
       retransmits=6, ca_state=TCP_CA_Loss, 17.43 s since first retransmission
       |user_data| 0xffff8800356cc800

 ** Execution took   0.04s (real)   0.04s (CPU)
crash>

Set net ns address (--netns)

The netns (network namespace) address can be changed using the '--netns' option as shown below:

crash> xportshow --netns 0xffffffff81ae2d80
 *=*=* Using <struct net 0xffffffff81ae2d80 *=*=*

 ** Execution took   0.01s (real)   0.01s (CPU)
crash>
crash> xportshow --netns 0xffff88007bbc8000
 *=*=* Using <struct net 0xffff88007bbc8000 *=*=*

 ** Execution took   0.01s (real)   0.01s (CPU)
crash>

The xportshow program also verifies whether the mentioned netns address is valid or not. In case of an invalid address, it will just log an error and refuse to change the network namespace:

crash> xportshow --netns 0xffff88007bbc8010
Invalid net ns 0xffff88007bbc8010

 ** Execution took   0.01s (real)   0.01s (CPU)
crash>

Run all functions in single command (--everything)

Users can also run all the above options in a single command by using '--everything':

crash> xportshow --everything | head -100
NPROTO=13, NF_MAX_HOOKS=8
=====PROTO= PF_INET
    NF_IP_PRE_ROUTING
    prio=-400,  hook=ipv4_conntrack_defrag
    prio=-200,  hook=ipv4_conntrack_in
    prio=-150,  hook=iptable_mangle_hook
    prio=-100,  hook=iptable_nat_ipv4_in
    NF_IP_LOCAL_IN
    prio=-150,  hook=iptable_mangle_hook
    prio=0,  hook=iptable_filter_hook
    prio=100,  hook=iptable_nat_ipv4_fn
    prio=300,  hook=ipv4_helper
    prio=2147483647,  hook=ipv4_confirm
    NF_IP_FORWARD
    prio=-225,  hook=selinux_ipv4_forward
    prio=-150,  hook=iptable_mangle_hook
    prio=0,  hook=iptable_filter_hook
    NF_IP_LOCAL_OUT
    prio=-400,  hook=ipv4_conntrack_defrag
    prio=-225,  hook=selinux_ipv4_output
    prio=-200,  hook=ipv4_conntrack_local
    prio=-150,  hook=iptable_mangle_hook
    prio=-100,  hook=iptable_nat_ipv4_local_fn
    prio=0,  hook=iptable_filter_hook
    NF_IP_POST_ROUTING
    prio=-150,  hook=iptable_mangle_hook
    prio=100,  hook=iptable_nat_ipv4_out
    prio=225,  hook=selinux_ipv4_postroute
    prio=300,  hook=ipv4_helper
    prio=2147483647,  hook=ipv4_confirm
=====PROTO= PF_BRIDGE
    NF_IP_LOCAL_IN
    prio=-200,  hook=ebt_in_hook
    NF_IP_FORWARD
    prio=-200,  hook=ebt_in_hook
    NF_IP_LOCAL_OUT
    prio=200,  hook=ebt_out_hook
=====PROTO= PF_INET6
    NF_IP_LOCAL_IN
    prio=0,  hook=ip6table_filter_hook
    NF_IP_FORWARD
    prio=-225,  hook=selinux_ipv6_forward
    prio=0,  hook=ip6table_filter_hook
    NF_IP_LOCAL_OUT
    prio=0,  hook=ip6table_filter_hook
    NF_IP_POST_ROUTING
    prio=225,  hook=selinux_ipv6_postroute
net.core.core.bpf_jit_enable                  0
net.core.core.busy_poll                       0
net.core.core.busy_read                       0
net.core.core.default_qdisc                   (?)
net.core.core.dev_weight                      64
net.core.core.message_burst                   10
net.core.core.message_cost                    5
net.core.core.netdev_budget                   300
net.core.core.netdev_max_backlog              1000
net.core.core.netdev_rss_key                  0
net.core.core.netdev_tstamp_prequeue          1
[...]