PaulWoods {dot} [Canada]

Another piece of old research. It’s so interesting, though, I can’t help putting a note up about it. In a piece of research released in 2000, Lutz Prechelt compared C, C++, Java, Perl, Python, Rexx, and Tcl.  (It’s gotten a fair bit of attention before, so it’s not new material)

What’s so good is the fairly rigorous and natural approach – instead of leaning exclusively on local students or one company’s employees performing some unnatural task, the researchers solicited for solutions online for the scripting languages (Perl, Python, Rexx and Tcl) and got 80 different implementations of a simple dictionary task. With these tasks they ran a lot of metrics and generated solid, comparable empirical data. Read it for the details (there is also a prettified version available).

I liked this observation, however:

For all program aspects investigated, the performance variability due to different programmers (as described by the bad/good ratios) is on average about as large or even larger than the variability due to different languages.

It’s quite profound – and even if the study itself is a bit dated and has some minor flaws – this observation probably still holds regardless!

Also, for Perl programmers, this statistic is quite neat:

The variability in execution speed for Perl programs were far lower than for the other languages except Tcl. It might be over-interpreting, but it indicates Perl is a bit more predictable – or Perl programmers are – than most of the other languages. Also, speed-wise is holding up quite good, and even the higher end of the range of execution speed is better than the higher end for all other languages.

The curious thing is that this indicates that the variability between programmers matters less with Perl (and Tcl) than the other languages. Or at least speed-wise, but the low variability also seems to be a tendency in the other metrics in the paper.

What’s so interesting, though, is how this demonstrates in hard numbers the importance of human variables in programming. It’s almost like framework matters less than the people using it.. funny, eh?