Date Submitted: 3 Feb 2012
Develop the next generation of programming tools for the next generation of computer programmers.
I divide this project into a research component and a prototyping component.
On the research side, I will explore the relationship between how we mentally model the world and how we model programs in computers. I will do this through user research studies and surveys of existing tools. Also on the research front, I hope to engage a critical theory of programming, analyzing the cognitive and sociological aspects of computer programming.
On the prototyping side, I will explore alternative programming paradigms, interfaces, and methodologies. I will do this by creating programming tools and putting them in front of users. At the same time I will try out various target applications, compelling use cases ill-served by existing tools through which I can inject these alternative ideas into mainstream programming discourse.
They say a fish is the last to discover water.
The way we think about computers today is far from the true potential of human-computer collaboration.
To give an example, computers used to be thought of as mainframes--giant calculators. People wrote programs, then fed them to the computer which would chug and spit out an answer. Serious computer programmers worked on artificial intelligence, the vision of a computer which we could feed any problem and it would tell us what to do. But a small group of researchers, inspired by Vannevar Bush's hypothetical memex, instead worked on augmented intelligence, the vision of human and computer working together in real-time, the computer augmenting the human intellect. The Augmentation Research Center, between 1962 and 1970, invented bitmapped screens, the mouse, hypertext, the GUI, and the internet.
The guiding principle behind this project was that computers and the human intellect must co-evolve. The technology we have affects the way we process information (the way we think), which in turn affects the technology we develop.
The computer programming community, like other scientific communities, undergoes paradigm shifts. Previous shifts include structured programming and object-oriented programming. These shifts take decades. They often only occur when a new generation replaces an old one.
I see a new generation of programmers emerging--artists and creatives who program computers to realize their visions, but who do not come from a traditional programming background. This generation provides the potential for another paradigm shift.
Unfortunately, the tools and languages that exist today for the "creative coders" in many ways propagate the old ways of thinking. The programs they encourage us to write are sequential, imperative, and with centralized control. These paradigms can be effective, but in my experience they are often limiting to those who do not think in these terms. They also do a disservice to emerging technologies which are massively parallel (GPUs, FPGAs) and distributed (cloud, mobile devices).
New programming tools must allow creatives to more effectively and faithfully realize their visions, while also promoting an understanding (through doing!) of the mathematical principles at the heart of computing.
Visual Representations of Causality
Visual Representations of Causality is a study of how artists and designers visualize causality and interaction in the projects they are working on. Participants are asked to draw a diagram of how their project works, specifically how the different parts of the project interact with each other.
Status: Done. But may do more surveys.
Survey of Abstraction Patterns in Patch-Based Programming Environments
Patch-based programming environments (e.g. Max/MSP, PD, vvvv, Quartz Composer, TouchDesigner, isabella, Grasshopper) provide a "boxes and wires" graphical environment for specifying dataflow or reactive programs. Because they are parallel, declarative, and highly interactive, they are popular among artists. Yet they are first-order. That is, they lack first-class functions. Further, they lack traditional control structures (conditionals and loops). Each of these environments provides alternative mechanisms to replace these features, sometimes quite clever mechanisms. By interviewing experienced users of these environments, I will catalogue these mechanisms.
Status: Scoped.
I am pursuing four target applications. Each informs the others. As they mature, I will devote my energies to one or two depending on progress, technical feasibility, and effectiveness in user testing.
1. Drawing-based Context Free Art (working title: Fractalville)
By drawing-based I refer to Bret Victor's ambitious Dynamic Pictures manifesto. Context Free Art is a programming language for creating fractal-like drawings (e.g. plants and other organic figures). I plan to take the semantics of Context Free Art and move them entirely into the visual/kinesthetic domain, giving the user a visceral understanding of transformations and recursion. Like designing with video feedback. Such a system would realize Victor's recursion system he outlines in Magic Ink.
Status: Prototypes coming along. 1, 2.
2. Drawing-based GPU Programming (working title: Morphism)
Similar to 1 but replicating the functionality of real-time pixel shaders. A graphical interface (like 1) that would compile to GLSL or Core Image.
Status: Requirements explored and user tested. Interface at the drawing board.
3. Interactive state-machine programming for Physical Computing (no working title)
Through helping ITP students with physical computing projects and user surveys from Visual Representations of Causality, I've determined that there is a major gap between Arduino's programming language and the way artists think about their projects, specifically in the areas of event-based reactivity and concurrency. Urbiscript aims to solve many of these problems for more complex robotics, but it is too big for Arduino.
This fruits of this exploration could make developing Arduino projects much more painless and rewarding, and it would help me explore UI issues around what I call process state.
Status: Technology stack (NodeJS, Firmata, Arduino) working.
4. Multi-user live coding for multi-player games (working title: Totem)
A programming "language" built around the constraint of live coding, the same way Haskell is built around the constraint of lazy evaluation.
Will combine UI paradigms from the previous three. Probably will not be pursued during thesis, but included for completeness as it's a major milestone for the project.
This project was conceived in July 2010 and I intend to continue work on it beyond Thesis Week.
The goal for Thesis Week will be to produce a compelling demo to illustrate these concepts. To do so, I will mature one or two of the above target applications to the point where it can be demoed as alpha software along with showing the creations of novice users. That is, a demo at the maturity of Takeo Igarashi's 1999 SIGGRAPH demo of Teddy.
