Skip to content

Instantly share code, notes, and snippets.

@surki

surki/gist:4b1a3d8d40328712b6586e151a0813c5

Created September 26, 2017 12:19
Show Gist options
  • Save surki/4b1a3d8d40328712b6586e151a0813c5 to your computer and use it in GitHub Desktop.
Save surki/4b1a3d8d40328712b6586e151a0813c5 to your computer and use it in GitHub Desktop.
Download ZIP
Getting a socket's 'struct sock' information
Raw
gistfile1.txt
You can get even more detailed information about a socket from kernel's internal socket struct.
For example, we will try to get nginx listening (on port 81) socket's backlog length
NOTE: You may need to install kernel debug info if not already installed
# yum-config-manager --enable "amzn-main-debuginfo" --enable "amzn-updates-debuginfo"
# yum -y install kernel-debuginfo kernel-devel
Or get socket info for listening socket on port 81
# ss -len | grep :81
tcp LISTEN 0 511 *:81 *:* ino:29842919 sk:55 <->
Get its sk buff address
# grep 29842919 /proc/net/tcp
8: 00000000:0051 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 29842919 1 ffff8800e9d1da00 100 0 0 10 0
Now get the details, for example, the backlog length
# gdb /usr/lib/debug/lib/modules/`uname -r`/vmlinux /proc/kcore
................
................
Reading symbols from /usr/lib/debug/lib/modules/4.4.51-40.69.amzn1.x86_64/vmlinux...done.
[New process 1]
Core was generated by `root=LABEL=/ console=tty1 console=ttyS0 selinux=0 LANG=en_US.UTF-8 KEYTABLE=us'.
................
(gdb) set print pretty on
(gdb) p *(struct sock *)0xffff8800e9d1da00
................
................
sk_ack_backlog = 0,
sk_max_ack_backlog = 511,
................
................
@surki
Copy link
Author

surki commented Sep 27, 2017

Not sure why you are not getting sk buff address. You seem to be using dual stacked connections, not sure if it has anything to do with it.
Here is my output for reference:

$ cat /proc/net/tcp
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode
   0: 00000000:0016 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 1790 1 ffff967299ce5d00 99 0 0 10 0
   1: 0100007F:18EB 00000000:0000 0A 00000000:00000000 00:00000000 00000000   995        0 17539 1 ffff967299cfc5c0 99 0 0 10 0
   2: 0100007F:2BCB 00000000:0000 0A 00000000:00000000 00:00000000 00000000   994        0 18479 1 ffff967299e0ae80 99 0 0 10 0
   3: 7045A8C0:BD9E C51AD9AC:01BB 06 00000000:00000000 03:00000CBE 00000000     0        0 0 3 ffff9671cc74b930
   4: 7045A8C0:97EE 42C53AD8:01BB 01 00000000:00000000 02:00000FA8 00000000  1000        0 7068715 2 ffff9671a7963e00 21 3 1 10 7
   5: 7045A8C0:8258 A21AD9AC:01BB 01 00000000:00000000 02:00000A52 00000000  1000        0 7065382 2 ffff96720a793e00 21 3 27 10 -1
   6: 7045A8C0:83D2 A31AD9AC:01BB 01 00000000:00000199 02:000007A8 00000000  1000        0 7071052 2 ffff96720a791740 20 3 22 10 -1
   7: 7045A8C0:B056 9EE0C717:01BB 01 00000000:00000000 02:00000A52 00000000  1000        0 7067845 2 ffff9670ef374d80 69 3 28 10 -1
   8: 7045A8C0:DE5E A4341517:01BB 08 00000000:00000001 02:00000030 00000000  1000        0 7062147 2 ffff967299cfec80 59 3 28 10 -1
   9: 7045A8C0:9D38 BD447D4A:01BB 01 00000000:00000000 02:00000099 00000000  1000        0 6985911 2 ffff9670ef376c80 24 3 30 10 -1
  10: 7045A8C0:9A56 126C1068:01BB 01 00000000:00000000 02:000009A8 00000000  1000        0 7067611 2 ffff9671cc4dae80 22 3 28 10 -1
  11: 7045A8C0:C804 3997C036:01BB 01 00000000:00000137 02:000006FD 00000000  1000        0 7068281 2 ffff9670ef371740 24 3 12 10 -1
  12: 7045A8C0:ABCA 96EBA423:01BB 01 00000000:00000000 02:000095BF 00000000  1000        0 5213694 2 ffff967233e0dd00 48 3 29 10 -1
  13: 7045A8C0:97F6 BD447D4A:01BB 01 00000000:00000000 02:000008A4 00000000  1000        0 6518248 2 ffff9670ef375540 24 3 30 10 -1
  14: 7045A8C0:DA72 BC447D4A:146C 01 00000000:00000000 02:000002FD 00000000  1000        0 5211109 2 ffff967299cf8f80 26 3 18 10 -1
  15: 7045A8C0:998C 0EA7D536:2563 01 00000000:00000000 02:00000F92 00000000  1000        0 6802944 2 ffff966e77454d80 54 3 30 10 -1
  16: 7045A8C0:DE70 A4341517:01BB 08 00000000:00000001 02:00000030 00000000  1000        0 7070975 2 ffff96720a796c80 61 3 26 10 -1
  17: 7045A8C0:CD30 024900C0:01BB 01 00000000:00000000 02:00000A52 00000000  1000        0 7065266 2 ffff96720a797440 23 3 28 10 -1
  18: 7045A8C0:8368 AE1AD9AC:01BB 06 00000000:00000000 03:0000173C 00000000     0        0 0 3 ffff9671cc74bc18
  19: 7045A8C0:A3E6 45016597:01BB 01 00000000:00000102 02:000006FD 00000000  1000        0 7068280 2 ffff9670ef377440 39 3 12 10 -1
  20: 7045A8C0:E910 CE1AD9AC:01BB 01 00000000:00000000 02:000009A8 00000000  1000        0 7057931 2 ffff9671cc4dbe00 21 3 27 10 7
  21: 7045A8C0:CC84 09021068:01BB 01 00000000:00000000 02:00000A52 00000000  1000        0 7070277 2 ffff967274a44d80 21 3 22 10 -1
  22: 7045A8C0:888C AE1AD9AC:01BB 01 00000000:00000000 02:00000AD3 00000000  1000        0 6979828 2 ffff967274a464c0 22 4 29 10 7
  23: 7045A8C0:D09E 8EC73AD8:01BB 06 00000000:00000000 03:0000173C 00000000     0        0 0 3 ffff9671cc74bb20
  24: 7045A8C0:8920 AE1AD9AC:01BB 01 00000000:00000199 02:000007A8 00000000  1000        0 7071051 2 ffff96720a7926c0 20 3 20 10 7
  25: 7045A8C0:BD3C C31AD9AC:01BB 01 00000000:00000045 02:000007A8 00000000  1000        0 7071053 2 ffff96720a795540 20 3 24 10 -1
  26: 7045A8C0:88F8 AE1AD9AC:01BB 01 00000000:00000000 02:00000A52 00000000  1000        0 7069748 2 ffff967274a445c0 21 3 25 10 7
  27: 7045A8C0:835A AE1AD9AC:01BB 01 00000000:00000000 02:000008E8 00000000  1000        0 6531118 2 ffff967274aaf440 21 3 27 10 7
  28: 7045A8C0:914E 5FC87D4A:01BB 01 00000000:00000000 02:000009A8 00000000  1000        0 7057267 2 ffff967274a40f80 25 3 27 10 -1
  29: 7045A8C0:E924 CE1AD9AC:01BB 01 00000000:00000000 02:00000A52 00000000  1000        0 7062148 2 ffff967299cfa6c0 21 3 21 10 7
  30: 7045A8C0:DE5C A4341517:01BB 08 00000000:00000001 02:00000030 00000000  1000        0 7067846 2 ffff9670ef370f80 49 3 28 10 -1
  31: 7045A8C0:D102 BE447D4A:01BB 01 00000000:00000000 02:00000A52 00000000  1000        0 7044190 2 ffff967299cf8000 29 3 21 10 -1
  32: 7045A8C0:C3D4 45C16597:01BB 01 00000000:00000000 02:00000AFD 00000000  1000        0 7057321 2 ffff967274a41740 24 3 28 10 -1
  33: 7045A8C0:D034 7DFD1EC0:01BB 01 00000000:00000000 02:000006BD 00000000  1000        0 6977121 2 ffff966f4bb00000 81 3 31 10 -1
  34: 7045A8C0:887A 448D0017:01BB 01 00000000:00000000 02:00000A52 00000000  1000        0 7062181 2 ffff967299cfdd00 24 3 28 10 -1
  35: 7045A8C0:9890 5B28E722:01BB 01 00000000:00000000 02:00000A30 00000000  1000        0 7022819 2 ffff966f4bb07440 57 3 24 10 -1
  36: 7045A8C0:D9C8 43C53AD8:01BB 01 00000000:00000000 02:00000A52 00000000  1000        0 7060810 2 ffff96720a794d80 22 3 25 10 7
  37: 7045A8C0:EB86 5E552CA9:01BB 01 00000000:00000000 02:00000BFB 00000000  1000        0 5212900 2 ffff9670efc80f80 48 3 30 2 2
  38: 7045A8C0:B79A 7CFD1EC0:01BB 01 00000000:00000000 02:0000043A 00000000  1000        0 6887741 2 ffff9670ef371f00 50 3 31 10 -1

$ uname -a
Linux suresh-MacBook 4.12.14-1-macbook #1 SMP PREEMPT Mon Sep 25 11:04:24 IST 2017 x86_64 GNU/Linux

If you are interested, here is where it gets printed: IPv4 and IPv6

@lafolle
Copy link

lafolle commented Sep 27, 2017

It seems kafka (and some other processes) are listening on ipv6 addresses (on ports 9092,9093 and 9094) (as you had noted), as shown by

lafolle@yoss:~/Downloads/confluent-3.3.0$ ss -lnt
State       Recv-Q Send-Q                                                        Local Address:Port                                                                       Peer Address:Port
LISTEN      0      128                                                                       *:25672                                                                                 *:*
LISTEN      0      80                                                                127.0.0.1:3306                                                                                  *:*
LISTEN      0      128                                                                       *:5355                                                                                  *:*
LISTEN      0      1024                                                              127.0.0.1:11211                                                                                 *:*
LISTEN      0      128                                                                       *:4369                                                                                  *:*
LISTEN      0      128                                                                       *:22                                                                                    *:*
LISTEN      0      5                                                                 127.0.0.1:631                                                                                   *:*
LISTEN      0      1024                                                                      *:15672                                                                                 *:*
LISTEN      0      100                                                               127.0.0.1:25                                                                                    *:*
LISTEN      0      50                                                                       :::9092                                                                                 :::*
LISTEN      0      50                                                                       :::9093                                                                                 :::*
LISTEN      0      50                                                                       :::2181                                                                                 :::*
LISTEN      0      50                                                                       :::9094                                                                                 :::*
LISTEN      0      50                                                                       :::26471                                                                                :::*
LISTEN      0      50                                                                       :::17639                                                                                :::*
LISTEN      0      128                                                                      :::5672                                                                                 :::*
LISTEN      0      128                                                                      :::5355                                                                                 :::*
LISTEN      0      511                                                                      :::80                                                                                   :::*
LISTEN      0      128                                                                      :::4369                                                                                 :::*
LISTEN      0      50                                                                       :::11027                                                                                :::*
LISTEN      0      128                                                                      :::22                                                                                   :::*
LISTEN      0      5                                                                       ::1:631                                                                                  :::*
LISTEN      0      100                                                                     ::1:25                                                                                   :::*
LISTEN      0      50                                                                       :::12447                                                                                :::*
LISTEN      0      128                                                                      :::12865                                                                                :::*

inode info for sockets on ipv6 are found in /proc/net/tcp6 as opposed to /proc/net/tcp.

lafolle@yoss:~/Downloads/confluent-3.3.0$ cat /proc/net/tcp6
  sl  local_address                         remote_address                        st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode
   0: 00000000000000000000000000000000:2384 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000  1000        0 133817 1 0000000000000000 100 0 0 10 0
   1: 00000000000000000000000000000000:2385 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000  1000        0 134614 1 0000000000000000 100 0 0 10 0
   2: 00000000000000000000000000000000:0885 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000  1000        0 134159 1 0000000000000000 100 0 0 10 0
   3: 00000000000000000000000000000000:2386 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000  1000        0 133980 1 0000000000000000 100 0 0 10 0
   4: 00000000000000000000000000000000:6767 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000  1000        0 133974 1 0000000000000000 100 0 0 10 0
   5: 00000000000000000000000000000000:44E7 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000  1000        0 134489 1 0000000000000000 100 0 0 10 0
   6: 00000000000000000000000000000000:1628 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000   132        0 27118 1 0000000000000000 100 0 0 10 0
   7: 00000000000000000000000000000000:14EB 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000   102        0 26182 1 0000000000000000 100 0 0 10 0
   8: 00000000000000000000000000000000:0050 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 24507 1 0000000000000000 100 0 0 10 0
   9: 00000000000000000000000000000000:1111 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000   132        0 29266 1 0000000000000000 100 0 0 10 0
  10: 00000000000000000000000000000000:2B13 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000  1000        0 134154 1 0000000000000000 100 0 0 10 0
  11: 00000000000000000000000000000000:0016 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 48605 1 0000000000000000 100 0 0 10 0
  12: 00000000000000000000000001000000:0277 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 68754 1 0000000000000000 100 0 0 10 0
  13: 00000000000000000000000001000000:0019 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 27378 1 0000000000000000 100 0 0 10 0
  14: 00000000000000000000000000000000:309F 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000  1000        0 133843 1 0000000000000000 100 0 0 10 0
  15: 00000000000000000000000000000000:3241 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 33243 1 0000000000000000 100 0 0 10 0
  16: 0000000000000000FFFF00000100007F:7158 0000000000000000FFFF00000101007F:2386 08 00000000:00000001 02:00022FB2 00000000  1000        0 133982 2 0000000000000000 20 4 28 10 -1
  17: 0000000000000000FFFF00000100007F:0885 0000000000000000FFFF00000100007F:7790 01 00000000:00000000 00:00000000 00000000  1000        0 135623 1 0000000000000000 20 4 31 10 -1
  18: 0000000000000000FFFF00000100007F:0885 0000000000000000FFFF00000100007F:7788 01 00000000:00000000 00:00000000 00000000  1000        0 130845 1 0000000000000000 20 4 31 10 -1
  19: 0000000000000000FFFF00000100007F:7788 0000000000000000FFFF00000100007F:0885 01 00000000:00000000 00:00000000 00000000  1000        0 132599 1 0000000000000000 20 4 30 10 -1
  20: 0000000000000000FFFF00000100007F:7790 0000000000000000FFFF00000100007F:0885 01 00000000:00000000 00:00000000 00000000  1000        0 130016 1 0000000000000000 20 4 30 10 -1
  21: 0000000000000000FFFF00000100007F:81D6 0000000000000000FFFF00000101007F:2385 08 00000000:00000001 02:00022C8A 00000000  1000        0 132624 2 0000000000000000 20 4 28 10 -1
  22: 0000000000000000FFFF00000100007F:573A 0000000000000000FFFF00000101007F:2384 08 00000000:00000001 02:00022B2E 00000000  1000        0 132603 2 0000000000000000 20 4 28 10 -1
  23: 0000000000000000FFFF00000100007F:0885 0000000000000000FFFF00000100007F:778C 01 00000000:00000000 00:00000000 00000000  1000        0 129993 1 0000000000000000 20 4 31 10 -1
  24: 0000000000000000FFFF00000100007F:778C 0000000000000000FFFF00000100007F:0885 01 00000000:00000000 00:00000000 00000000  1000        0 136261 1 0000000000000000 20 4 30 10 -1

Here the inode of socket in which we're interested in is present, but corresponding address (?) of socket is 0 (and hence socket can't be accessed using gdb.)

So, the problem, I think, boils down to "why socket address (?) is 0 for all active sockets in /proc/net/ipv6?".
But interestingly socket address is 0 also for all sockets in /proc/net/tcp.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment