Step 1: The Programming Task
In this demo, we intend to analyze student solutions for a Java programming
task. The task is pretty simple: Write a Java program that decides, whether all
words in a given sentence are palindromes. For input like this the program
should return true:
otto anna lol
On the right we show 48 (correct) example solutions for this task.
Note that those programs apply different underlying strategies. We marked these
strategies in the file name by a three-digit code (AAA to
BBB), where each digit corresponds to one design
decision:
- The student can implement the required methods herself (A)
or rely on some functions from the Java standard library (B),
such as String.reverse().
- The student can process the input sentence using the Java String class
(A) or the primitive single characters
(B).
- The student can process the input sentence by first splitting it into words
and then analyzing the words one by one (A) or by
processing the sentence in a stream-like fashion, analyzing only the current
word (B).
As these design choices are independent, there are 8 possible
programming strategies. Our dataset contains 6 variations for each of these
strategies, where each variation introduces a different form of
'stylistic noise':
- Programs, where the name ends with 1 introduce
comments.
- Programs marked with 2 have differently named
variables.
- In these programs, the order of methods or variables (whole code blocks)
is changed, without changing the order of statements within these code blocks.
- In these programs, the order of single statements in the program is subtly
changed, without changing the function of the program.
- Finally, in the programs marked with 5 we introduce
additional lines of code, for example additional variables storing intermediate
results, without changing the function of the program.
Thus, we model two important observations in our dataset:
- Some changes in program code relate to a different
underlying programming strategy, probably related to a different cognitive
understanding of the programming task and its solution. When providing
automated feedback based on example solutions of other students, one should
try to find a partner solution that implements the same underlying strategy.
Otherwise the feedback might be confusing.
- Other changes in program code are not related to the underlying strategy
at all and might not even change the output in the program. Many stylistic
choices while programming, such as variable naming, commenting and the order
of operations, are just matters of taste.
Therefore, a good similarity or dissimilarity measure for programs should
be sensitive to meaningful differences, while being insensitive to mere
stylistic differences.