Both of them said pretty much the same thing. They said I was too methodical and relied too heavily on documentation. More importantly, I'm sure, was that I didn't yet embrace test-driven development, TDD.
In TDD, you write your tests first, and then you write your code. However, ever since college, I've written my docstrings first, and then I write the code. I generally write the tests after I've written some code, and then I grow the two together. Although, I don't practice TDD, I am test infected, which means I feel uncomfortable about code that I've written unless I've also written tests.
I guess I've always been influenced by Knuth's literate programming. Apparently, someone forgot to tell me that literate programming is no longer cool.
My buddy Dan Aronson introduced me to a great programmer at Pivotal named Kelly Felkins. Pivotal is famous for being one of the most agile shops around. They practice TDD, they peer program pretty much all the time, etc., and they write good software. When I explained to Kelly my style of "docstring-driven development", he said I'd never make it at Pivotal. The other engineers would just delete all my docstrings. I think the idea is that in TDD, comments are to be distrusted, whereas tests are the true way of documenting code.
In real life, comments usually should be distrusted, but that's not the case with my comments. That's because in my style of docstring-driven development, I refactor the comments before I refactor the code. In the Python world, there's even doctest which lets you embed your tests right in your docstrings, which ensures that they are more likely to be current.
Until yesterday, I had never considered there to be a contradiction between literate programming and test-driven development. I just figured most engineers were too lazy to write both tests and documentation. As we all know, many engineers are too lazy to write either tests or documentation ;) Nonetheless, Kelly's arguments for TDD were pretty convincing. For instance, one of his best arguments was that TDD leads to much better test coverage.
It's interesting to note that not everyone likes agile or extreme programming (extreme programming is a specialized form of agile programming). For instance, Donald Knuth, the godfather of computer programming, said:
Let me just say that almost everything I’ve ever heard associated with the term "extreme programming" sounds like exactly the wrong way to go...with one exception. The exception is the idea of working in teams and reading each other’s code. That idea is crucial, and it might even mask out all the terrible aspects of extreme programming that alarm me.Adam Wolff, who worked for several years on OpenLaszlo wrote a particularly insightful anti-agile piece:
Without even looking at the telltale URL, you can see these apps from a mile away. They still have the smell of the original auto-generated scaffolding, but more importantly, it's pretty clear that they were developed piecemeal...If you step back a bit, it's pretty easy to see the whole Web 2.0 thing as giant circlejerk of technology creators making it easier for mediocre developers to ship the next marginally useful application, while they struggle to implement a user interface that has the quality of a typical '80s PowerBuilder app...My second point is potentially more important: piecemeal approaches yield piecemeal solutions...In fact, many of the agile best practices reinforce short-sightedness and lock-in, even as they claim to emphasize nimbleness...The whole point of design is to incorporate the big picture in the little details, but when you're pursuing a "let's try it and see how it goes" approach, there is no big picture; there's just a bunch of stories.If you've ever used an OpenLaszlo app, you know that he has a point. An OpenLaszlo app generally does a better job conveying the big picture. In general, OpenLaszlo apps put Ajax-based Web 2.0 apps to shame.
Perhaps user interfaces are one thing that shouldn't be designed in an agile way. One of the best user experience people I know, Stephanie Hornung, really drove this point home for me. Consider Amazon.com. It leaves me feeling overwhelmed and confused because there are too many piecemeal features vying for my attention. It's like an experiment in agile programming gone wrong. Stephanie argued that good user experience people aren't agile. They have to understand and plan for the big picture. I think this is one thing that Apple is particularly good at.
At the risk of using an overused metaphor, I don't think you can use agile techniques to build Gothic architecture:
However, most of us don't live in Gothic cathedrals. My own house is 100 years old, and has had at least five addons. It's a total mess, but unlike most Gothic cathedrals, it has central heating and air ;)
And just to be fair, even though Pivotal is an agile shop, their tool Pivotal Tracker doesn't feel like a piecemeal solution. It's easy to use, and the big picture is obvious.
The question remains, are agile programming and literate programming at odds with each other? If you're writing a Rails application, documenting the big picture is pointless. There's too much magic (aka convention) going on for in-line documentation to be helpful. The fact of the matter is, if you're building a Rails app, you need to read either a book or a tutorial ahead of time. Once you've done that, there's not much point in writing documentation that covers what you've already read.
However, not everything is a Rails app. For instance, Structure and Interpretation of Computer Programs, SICP, talks about the challenges of writing a memory manager for Lisp. Either you get it right, or you're completely screwed. Imagine trying to write a memory manager from scratch on a new hardware architecture. If you mess it up, your entire object graph is going to be toast. In such cases, I think it does make sense to embrace literate programming. I also think that working it out on paper ahead of time and heavy code review are necessary. Of course, just like Gothic cathedrals, not many people have to write memory managers for Lisp these days. Banging out Rails apps for Web 2.0 is far more the norm.
While I'm on the subject of SICP, perhaps it's telling that M.I.T. has switched from Scheme to Python for its first computer science course that students must face. Gerry Sussman said, "Nowadays, a real engineer is given a big software library, with a 300-page manual that’s full of errors...The engineer must learn to perform scientific experiments to find out how the software and hardware actually work, at least enough to accomplish the job at hand." Sussman pointed out that we may not like it this way (”because we’re old fogies”), but that’s the way it is, and M.I.T. has to take that into account. When you put it that way, TDD makes all the sense in the world.
Futhermore, Kelly pointed out that if you only write as much code as need to make the tests pass, you avoid overengineering a bunch of code that may not match the real world.
Where does that leave me? I'm going to use TDD for my Rails app, but I still maintain that a truly brilliant user interface must be developed like a Gothic cathedral. If you're interested in doing TDD for building a Web app in Python, I suggest some mix of Twill, Nose, and Selenium. If you're using TDD for building a Web app in Rails, Kelly suggested some mix of Cucumber, RSpec, and Selenium.