How to use strace for a daemon with multiple processes, including children

Sometimes when you are running Linux and have a weird bug, you wish you would know what the process is doing. For example, which files it is opening and such ...

The strace command is very useful.

Use case 1: Simple command as argument

However, its common use cases is when you have a single process running. Usually, you would run strace and your command as an argument, like so:

strace your_program

Use case 2: Tracing a running command

But, there are cases when the above is not possible. For example, you are tracing a running command.

In this case, you need to use the -p command with the process ID as an argument, so the following command is what you use. Note that we use sudo, since most daemons will not be running with the same user as yours.

sudo strace -p process_id

If that process forks other children, and you need to trace those as well, then you use the -f argument:

sudo strace -f -p process_id

Use case 3: Tracing a multi-process daemon, and its children

Sometimes, the daemon is already composed of multiple process, and you don't know which process will receive the request you are interested in. This is the case with web servers and web applications, such as Apache and PHP-FPM.

It is really tedious finding the process IDs of a running daemon, then composing a command with the process IDs as arguments.

So, we use a trick: we use the pidof command to get a list of the process IDs, then we use the tr command to replace spaces by commas, like so:

sudo strace -p `pidof apache2 | tr ' ' ','`

The Ultimate strace: Everything!

But, usually, we want a few more things, for example, a timestamp of every system call, larger amount of data passed to each, and we want the output to go to a file we can examine later.

So, the ultimate strace command line is:

sudo strace -f -tt -o /tmp/php.trace -s1024 -p `pidof php5-fpm | tr ' ' ','`

The options used are:

-f follow children
-tt timestamps, with microseconds
-o output file
-p process IDs

Hope this helps someone ...