Hints for writing Unix tools

Note: this article has been translated into Japanese

The workaday world of a modern programmer abounds with Unix tools, stitched together in myriad ways. While good tools integrate seamlessly with your own environment, bad ones will constantly frustrate your efforts. Good tools have a seemingly limitless application, constrained only by your own imagination. Bad tools, on the other hand, will often require that you deploy a salvo of brittle hacks to keep them barely working in your own environment.

“One thing well” misses the point: it should be “One thing well AND COMPOSES WELL”

— marius eriksen (@marius) October 10, 2012

I don’t want to attempt to explain what makes for good design; this has been discussed elsewhere. Instead, I want to outline a few established customs that you should take care to follow when writing new tools. While making a truly good tool can be an elusive goal, it isn’t difficult to avoid making a truly bad one. Unix demands good citizenry from its tools: it relies on a set of conventions to make things work, and importantly, to compose, well. Here follows a few key customs, often violated. These aren’t absolute requirements, but you should think long and hard before violating them.

Consume input from stdin, produce output to stdout. Put another way, your program should be a filter. Filters are easily integrated into shell pipelines, arguably the most important utility for Unix tools composition.

Output should be free from headers or other decoration. Superflous output will frustrate users who are trying to parse tool output. Headers and decoration tend to be less regular and more idiosyncratic than the structured data you’re really trying to get at. Don’t do it.

Output should be simple to parse and compose. This usually means representing each record as a single, plain-text formatted line of output whose columns are separated by whitespace. (No JSON, please.) Most venerable Unix tools—grep, sort, and sed among them—assume this. As a simple example, consider the following output from a benchmark suite. It is formatted by starting each record with the benchmark name, followed by a set of key-value pairs associated with the named benchmark. This is a flexible structure to work with as it allows you to add or remove keys at will without violating the output format.

	$ ./runbenchmarks
	Benchmark: fizzbuzz
	Time: 10 ns/op
	Alloc: 32 bytes/op
	Benchmark: fibonnacci
	Time: 13 ns/op
	Alloc: 40 bytes/op
	...
	$

While convenient, it is quite clumsy to work with in Unix. Consider a very common thing we might want to do: look up the timing results for a single benchmark. Here’s how you do it.

	$ ./runbenchmarks | awk '/^Benchmark:/ { bench = $2}  bench=="fizzbuzz"'
	Benchmark: fizzbuzz
	Time: 10 ns/op
	lloc: 32 bytes/op
	$

If instead each line presents exactly one record, where columns are separated by whitespace, this becomes a much simpler task.

	$ ./runbenchmarks 
	fizzbuzz	10	32
	fibonnaci	13	40
	...
	$ ./runbenchmarks | grep '^fizzbuzz'
	fizzbuzz	10	32
	$

The advantage becomes even more evident when reordering or aggregating the input. For example, when the output is record-per-line, sorting the results by time spent is a simple matter of invoking sort:

	$ ./runbenchmarks | sort -n -r -k2,2
	fibonnaci	13	40
	fizzbuzz	10	32
	...
	$

Treat a tool’s output as an API. Your tool will be used in contexts beyond your own imagination. If a tool’s output format is changed, other tools that compose or otherwise build on its output will invariably break—you have broken the API contract.

Place diagnostics output on stderr. Diagnostics output includes anything that is not the primary data output of your tool. Among these are: progress indicators, debugging output, log messages, error messages, and usage information. When diagnostics output is intermingled with data, it is very difficult to parse, and thus compose, the tool’s output. What’s more, stderr makes diagnostics output more useful since, even if stdout is being filtered or redirected, stderr keeps printing to the user’s terminal—the ultimate target of diagnostics output.

Signal failure with an exit status. If your tool fails, exit with a status other than 0. This allows for simple integration shells, and also simpler error handling in scripts. Consider the difference between two tools that build binaries. We’d like to build upon this tool to execute the built binary only if the build succeeds. Badbuild prints the word ‘FAILED’ as the last line when it fails.

	$ ./badbuild binary
	...
	FAILED
	$ echo $?
	0
	$ # Run binary on successful build.
	$ test "$(./badbuild binary | tail -1)" != "FAILED" && ./binary
	$

Goodbuild sets its exit status appropriately.

	$ ./goodbuild
	$ echo $?
	1
	$ # Run binary on successful build.
	$ ./goodbuild binary && ./binary
	$

Make a tool’s output portable. Put another way, a tool’s output should stand on its own, requiring as little context as possible to parse and interpret. For example, you should use absolute paths to represent files, and fully qualified hostnames to name internet hosts. Portable output is directly usable by other tools without further context. A frequent violator of this is build tools. For example, both the GCC and Clang compilers try to be clever by reporting paths that are relative to your working directory. In this example, the source file paths are presented relative to the current working directory when the compiler was invoked.

	$ cc tmp/bad/x.c
	tmp/bad/x.c:1:1: error: unknown type name 'INVALID_C'
	INVALID_C
	^
	tmp/bad/x.c:1:10: error: expected identifier or '('
	INVALID_C
	         ^
	2 errors generated.
	$

This cleverness breaks down quickly. For example if I use make(1) with the -C flag.

	$ cat tmp/bad/Makefile
	all:
		cc x.c
	$ make -C tmp/bad
	cc x.c
	x.c:1:1: error: unknown type name 'INVALID_C'
	INVALID_C
	^
	x.c:1:10: error: expected identifier or '('
	INVALID_C
	         ^
	2 errors generated.
	make: *** [all] Error 1
	$

Now the output is less useful: to which file does “x.c” refer? Other tools that build on this need additional context, the -C argument, in order to interpret the compiler’s output—the output does not stand on its own.

Omit needless diagnostics. Resist the temptation to inform the user of everything that is being done. (But if you must, do it on stderr.) A good tool is quiet when all is well, but produces useful diagnostics output when things go wrong. Excessive diagnostics conditons users to ignore all diagnostics; useful diagnostics output does not require the user to grub around in endless log files to discern what went wrong, and where. There’s nothing wrong with having a verbose mode (typically enabled by a ‘-v’ flag) in order to aid development and debugging, but do not make this the default.

Avoid making interactive programs. Tools should be usable without user interaction beyond what’s provided by the user’s shell. Unix programs are expected to run without user input: it allows programs to be run in non-interactively by cron, or to be easily distributed for execution by a remote machine. Even a single interaction forfeits this very useful capability. Interactivity also makes composition more difficult. Since Unix’s program composition model does not distinguish the output of the various programs involved, it isn’t always clear which program a user is even interacting with. A common use of interactive programs is to ask the user to confirm some dangerous action. This is easily avoided by asking the user instead to supply a flag on the command line to the appropriate tool.

I wrote this because I find myself continually frustrated by attempting to use and compose bad tools—bad tools that waste time and limit their own usefulness. Most of these tools could be made a lot better by following the above advice.

For a more general discussion of Unix tools design, I encourage you to read Kernighan and Pike’s “The Unix Programming Environment .”

Discussion on Hacker News.