Many times, when I've inherited or encountered scientific code written by other people (or occasionally, even my own work), I've noticed that documentation is either sparse or nonexistent. If I'm lucky, I see informative comments. If I'm very lucky, there's even Doxygen comments and a Doxyfile so that I have function interfaces and some formatted HTML to consult. If I'm extremely lucky, there's a PDF manual and examples in addition to the Doxygen and source file comments, and I'm ecstatic, because it makes my life much, much easier.

What information and tools are useful in documenting source code? For that matter, what information and tools are useful to document the data and results that accompany that source code, in the case of scientific software?

I think that documentation for scientific software can be divided into three categories, all of which are necessary for full understanding. The easiest and most common is individual method documentation. There are many systems for this. You mention doxygen, Python has pydoc, and in PETSc we have our own package sowing which generates the following.

However, for any piece of software which goes beyond a simple utility, you need a manual. This provides a high-level view of the purpose of the package, and how its different functionalities integrate to achieve this purpose. It helps a new user structure their code, often through the use of examples. In PETSc, we just use LaTeX for the manual, but the PyClaw package uses the Sphinx framework which I am very impressed with. One thing that we have implemented in the sowing package that I find very useful is the link between example code and function documentation. For example, this example solves the Bratu equation. Notice how you can follow the links for any custom type or function call and get to the low-level documentation, and how those pages link back to examples using them. This is how I learn about new functionality which other people in the project contribute.

A frequently overlooked part of documentation, I think, is developer documentation. It is not uncommon to publish a coding-style document, and instructions for interacting with the repository. However, it is very rare to explain the design decisions made before implementation. These decisions always involve tradeoffs, and the situation with respect to hardware and algorithms will necessarily change over time. Without a discussion of the tradeoffs reviewed and rationale for particular design decisions, later programmers are left to recreate the entire process on their own. I think this is a major impediment to successful maintenance and improvement of old codes when the original developers are no longer in charge.
