But I want to point to another very useful tool that got completely shadowed by DTrace: TNF tracing. In short, it is "
printf() debugging" on steroids; read the tracing(3TNF) man page for more details. The TNF logging probes must be manually inserted in the source code at places where printf() would be put, and they produce no output by default. When you run the program under the control of prex(1) program, you may choose to log the otuput of all probes, or select just a subset of them (they are named!). The trace is later analyzed with the tnfdump(1) program. The best thing is that prex works also on running programs; it can attach to them, collect data and detach at later point.
TNF has been an enormous help in finding and correcting complex, time-dependent problems that I'd have much harder time to resolve otherwise.