strace is great for debugging processes and getting a better idea of what’s happening behind the scenes. The Wikipedia article on strace does a better job explaining:
strace is a debugging utility for Linux and some other Unix-like systems to monitor the system calls used by a program and all the signals it receives, similar to “truss” utility in other Unix systems
It lets you see every system call made by a program. You can try it by running “strace ls”. I’ve stared at a lot of strace output, and part of that staring has helped me identify chunks of system calls as patterns that many processes share. Being able to identify these patterns can be helpful during debugging since it lets you a) skip areas of calls with confidence and b) gives you insight into which stage of the process you’re investigating. There are plenty of obvious ones, but I figured I’d start with one I just realized.
The wall of “close” calls pattern:
I’ve seen strace files with this seemingly-odd chunk of “close” system calls:
[pid 2082] close(7) = 0
[pid 2082] close(8) = -1 EBADF (Bad file descriptor)
[pid 2082] close(9) = -1 EBADF (Bad file descriptor)
…
[pid 2082] close(255) = -1 EBADF (Bad file descriptor)
At first glance, it appears that the process is blindly closing a predetermined number of filehandles. This is exactly what it’s doing, and it’s an indicator that the process is in the process of daemonizing itself. I first saw this in Stevens’ “Advanced Programming in the UNIX Environment”, and the same convention is repeated in the paper, “How to Write a UNIX Daemon (PDF”):
Do not leave stray file descriptors open. More importantly, if any of the file descriptors are terminal devices then they must be closed to allow reset of the terminal state during logout (see below). The typical code sequence is:
for (fd = 0; fd < _NFILE; fd++)
close(fd); /* close all file descriptors */
While this exact treatment is not the most up-to-date (could be using sysconf(_SC_OPEN_MAX) or just hitting /proc/self/fd), it is still widely used and should give you a heads-up when you run into it in your own strace travels.