Add pprof to runc-shim

[henry118]

Adding pprof capability to containerd-runc-shim-v2.

This makes the shim to expose an additional debugging endpoint when running in DEBUG mode. And users can capture pprof data via ctr:

NAME:
   ctr shim pprof - Provide golang pprof outputs for containerd-shim

USAGE:
   ctr shim pprof command [command options] 

COMMANDS:
   block         Goroutine blocking profile
   goroutines    Dump goroutine stack dump
   heap          Dump heap profile
   profile       CPU profile
   threadcreate  Goroutine thread creating profile
   trace         Collect execution trace
   help, h       Shows a list of commands or help for one command

OPTIONS:
   --help, -h  Show help (default: false)

e.g. To capture the heap profile for the shim hosting container "test":

ctr shim --id test pprof heap >heap.prof 

[mxpv]

One minor nit, overall LGTM

[mxpv]

To enable pprof, we'd need to specify both -debug and -debug-socket here, right?
Possibly we can simplify this and enable when -debug-socket != "" ?

[dmcgowan]

How is debug-socket flag getting passed down to the shim call?

[ZhangShuaiyi]

I have analyzed the code.
@mxpv Only the -debug flag is needed to enable pprof.
@dmcgowan The -debug-socket is not necessary. If -debug-socket is not passed, shim calls os.NewFile(4, 'socket') to obtain the fd passed by containerd.
@henry118 Is my understanding correct?

[ZhangShuaiyi]

The unix socket address is generated by sha256.Sum256.

[root@devnode1 containerd]# ctr c ls
CONTAINER    IMAGE                             RUNTIME
mydemo       docker.io/library/nginx:latest    io.containerd.runc.v2
[root@devnode1 containerd]# ls /run/containerd/s/
0b6efc3510d8ebcb7d27d04dfe4444511fcb6a1fb95411e15b50888cdec86dc3
12bede06714e38de587a8a27a83b967a4a5263ff48c6186a4ec274927a8bc692
[root@devnode1 containerd]# echo -n /run/containerd/containerd.sock/default/mydemo | sha256sum
0b6efc3510d8ebcb7d27d04dfe4444511fcb6a1fb95411e15b50888cdec86dc3  -
[root@devnode1 containerd]# echo -n /run/containerd/containerd.sock/default/mydemo/debug | sha256sum
12bede06714e38de587a8a27a83b967a4a5263ff48c6186a4ec274927a8bc692  -

[henry118]

Yes the debug-socket is not used or passed down in regular case, where the shim is spawned by containerd.

I created this cmd option for two reasons:

1. By following the existing pattern in shim where the socket cmd option is optionally passed in and used as the socket for all shim requests. Similarly when shim is spawned by containerd the socket option is not used, and the socket addr is generated in the format of containerd-sock/ns/container-id;
2. IMO both socket and debug-socket would be useful when debugging the shim standalone (manual launch instead of spawned by containerd). In this case a user can specify a well-known socket addr.

[ZhangShuaiyi]

This will create a new debug socket for pprof, which results in shim creating two sockets. Is it prossible to use just one socket instead?

[root@devnode1 containerd]# ls /run/containerd/s/
[root@devnode1 containerd]# ctr run -d docker.io/library/nginx:latest mydemo
[root@devnode1 containerd]# ls /run/containerd/s/
0b6efc3510d8ebcb7d27d04dfe4444511fcb6a1fb95411e15b50888cdec86dc3
12bede06714e38de587a8a27a83b967a4a5263ff48c6186a4ec274927a8bc692
[root@devnode1 containerd]# ctr shim --id mydemo pprof goroutines > goroutines.prof
[root@devnode1 containerd]#

[henry118]

We have to create a separate socket for pprof because

1. The regular endpoint and pprof endpoint do not share protocols (ttrpc vs http)
2. A number of pprof command could potentially block the socket for a long period of time. e.g. the trace command can take a second parameter and trace the program for an arbitrary timeframe.

[ZhangShuaiyi]

Thank you for your explanation, I understand now.

[ZhangShuaiyi]

LGTM

[estesp]

minor nit: "s/pporfCommand/pprofCommand/"

[henry118]

@mxpv @dmcgowan @estesp Mind taking another look? Feel free to let me know anything required to merge this. Thanks :)

[fuweid]

is is required to add debug here? If I understand it correctly, the pprof is using different socket address.

[henry118]

The socket address used by pprof in shim is created in the Start routine by passing in the debug=true flag here.

Both regular service socket and the new debug socket addresses are created by this function, based on the socketPath argument. In both cases, the value of this argument is the containerd's listening socket address - /run/containerd/containerd.sock (Also see the observation here).

So we need to append the debug string at the end to distinguish it from the regular service socket.