Prex, DTrace and linux lameness

Adam Leventhal has in his blog exposed the lameness of the SystemTap team (e.g. they claim that they weren't inspired by DTrace at all, but output format of some of their utilities is identical to that of DTrace). He also took an amusing photo showing that users are not very happy with SystemTap.

But I want to point to another very useful tool that got completely shadowed by DTrace: TNF tracing. In short, it is "printf() debugging" on steroids; read the tracing(3TNF) man page for more details. The TNF logging probes must be manually inserted in the source code at places where printf() would be put, and they produce no output by default. When you run the program under the control of prex(1) program, you may choose to log the otuput of all probes, or select just a subset of them (they are named!). The trace is later analyzed with the tnfdump(1) program. The best thing is that prex works also on running programs; it can attach to them, collect data and detach at later point.

TNF has been an enormous help in finding and correcting complex, time-dependent problems that I'd have much harder time to resolve otherwise.