Step 1: The Programming Task

In this demo, we intend to analyze student solutions for a Java programming task. The task is pretty simple: Write a Java program that decides, whether all words in a given sentence are palindromes. For input like this the program should return true:

otto anna lol

On the right we show 48 (correct) example solutions for this task. Note that those programs apply different underlying strategies. We marked these strategies in the file name by a three-digit code (AAA to BBB), where each digit corresponds to one design decision:

As these design choices are independent, there are 8 possible programming strategies. Our dataset contains 6 variations for each of these strategies, where each variation introduces a different form of 'stylistic noise':

  1. Programs, where the name ends with 1 introduce comments.
  2. Programs marked with 2 have differently named variables.
  3. In these programs, the order of methods or variables (whole code blocks) is changed, without changing the order of statements within these code blocks.
  4. In these programs, the order of single statements in the program is subtly changed, without changing the function of the program.
  5. Finally, in the programs marked with 5 we introduce additional lines of code, for example additional variables storing intermediate results, without changing the function of the program.

Thus, we model two important observations in our dataset:

Therefore, a good similarity or dissimilarity measure for programs should be sensitive to meaningful differences, while being insensitive to mere stylistic differences.