Could text-based programming be augmented to allow creating programs just by directly manipulating output?

Prior work on Sketch-n-Sketch, a bimodal program editor, has demonstrated the viability of this approach for programs that produce vector graphics. In addition to standard text editing, in Sketch-n-Sketch the programmer may use the mouse to draw, move, align, group, and abstract shapes on the program’s output canvas, allowing some programs to be created entirely by direct manipulation.

Seeking to scale up this approach, our work extends Sketch-n-Sketch with features for constructing larger programs. First, we show intermediate execution products (rather than only the final output) and expose them for direct manipulation. Second, to better control the effects of program transformations (rather than making changes mostly at the top-level), we let the programmer selectively narrow the editing focus to a single function call; interactions are limited to those intermediates and outputs in focus. Finally, we improve and add tools to offer additional meaningful program transformations.

To present our approach, we walk through the step-by-step construction of a program that draws a von Koch curve fractal. By manipulating the visually rendered intermediates, we build the non-trivial recursive program directly and entirely on the output canvas, without resorting to traditional text-editing.