Code highlighting: the lowlights

Syntax highlighting is such a ubiquitous feature in program editors that we often give it very little thought. It’s even like an indicator of program code: you can tell something is code if it is in a fixed-width font and some of the words are consistently coloured. It’s clearly a popular feature but is it actually helpful?

The latest paper on this (paywalled, alas) is by Hannebauer et al, which I found via Greg Wilson. They set ~400 participants a variety of comprehension and editing tasks and found zero difference in correctness between having syntax highlighting on and off during the tasks. So it doesn’t look like it helps with programming.

There’s two ways to take this result. One is that since the feature is ineffective, we should stop wasting effort on building it into our IDEs. The other way to view null results is that since it makes no difference either way, we are free to choose based on other considerations. The authors of the paper imply that people seem to like syntax colouring, which may well be for aesthetic reasons. And if it doesn’t get in the way, why not make it look prettier?

The authors end with a suggestion that highlighting syntax keywords may not be most effective use of colour, and propose a few of their own schemes, such as using colouring to do a live git blame display. I’d say the obvious other use of colour is for scope highlighting, where coloured background indicates the extents of code blocks. BlueJ has both syntax highlighting and scope highlighting, which can be a bit busy:

When we made the Stride editor, we left out syntax highlighting and just kept the scope highlighting provided by the frame outlines, which seemed less visually noisy:

We haven’t done a study to look at the effect of this scope highlighting. But David Weintrop and Uri Wilensky did something similar when they looked at multiple choice questions shown in text-based form (with syntax highlighting) versus block-based form (which is effectively scope highlighting), and the non-null effects showed a superiority of blocks over text, although the highlighting is not the only difference:

Their paper is available online (unpaywalled).

So although syntax highlighting of tokens does not seem to make a difference, scope highlighting may aid comprehension. (If anyone wants to study this directly, you can toggle syntax and scope highlighting on and off in BlueJ, so be our guest…)

2 thoughts on “Code highlighting: the lowlights

  1. What about time? Does syntax highlighting allow students to read and understand (and then modify/debug) code faster? Correctness might be the same, but if one group took significantly longer, then there is an advantage.

    1. The authors say “The time needed to solve the tasks was not measured as previous research suggests that code highlighting has an impact on correctness (Baecker and Marcus 1989), whereas there is no statistically significant impact on speed even in experiments that showed correctness improvements through code highlighting (Oman and Cook 1990)”.

      However, since their findings seem to go against the previous work (which had much smaller sample sizes), it’s not clear whether it was wise to rely on the previous results to guide this work… It seems likely that there is a relationship between the two.

Leave a comment