Experiment-Oriented Computing

This page collects useful resources regarding Experiment-Oriented Computing (EOC), a concept introduced in my ESEC/FSE 2018 paper The Case for Experiment-Oriented Computing (local free download). Computational experimentation technology can be found in many forms, sometimes explicit and dedicated, but more often intertwined with other concerns. In almost all cases I’m aware of, however, there is no proper understanding of the wide scope that such technology can have. Nevertheless, it is useful to map the technology that does exist, since it can help us track, motivate and imagine the progress of proper EOC tools and systems. I’ve also created a related infographic for dissemination to practitioners.

 

Domenico Remps - Cabinet of Curiosities
Cabinet of Curiosities, by Domenico Remps (17th century). Collecting unique, strange, exotic or inspiring objects used to be popular among enlightened people. Thanks to efficient travel and communication, it is getting harder and harder for us to appreciate the appeal of such a thing.

Tools

A/B Test Libraries, Frameworks and Services

In Software Engineering, experimentation is often confused with mere A/B testing. This is actually a very popular technique, so it would be futile to try to curate all such tools here. Rather, I will focus on those that for some reason are particularly interesting or representative.

Experiment Tracking and Versioning

Experimentation with Users

Other Tools

 

Real-World Applications, Laboratories and Companies

Papers

The nature of experimentation itself:

Experimenter’s workflow:

Industrial applications:

Scientific applications:

Ethics:

In Human-Computer Interaction (HCI), there are tools that help users to experiment with the design of various types of artifacts. These range from very simple approaches (e.g., quick previews) to highly sophisticated ones, based on optimization or learning techniques. Beyond their specific design applications, such tools are, by definition (the ‘Human’ part of HCI), very close to users, and therefore can be rich sources of inspiration for more general experimentation interfaces.

Books

(Computational) Scientific Discovery.

Active Learning. Burr (2012)’s description of the field explains the importance for experimentation very well: “Traditional ‘passive’ learning systems induce a hypothesis to explain whatever training data happens to be available (e.g., a collection of labeled instances). By contrast, the hallmark of an active learning system is that it eagerly develops and tests new hypotheses as part of a continuing,interactive learning process. Another way to think about it is that active learners develop a ‘line ofinquiry,’ much in the way a scientist would design a series of experiments to help him or her drawconclusions as efficiently as possible.“

Software analytics (and related experimental concerns). Although, in principle, software analytics can be entirely passive (and therefore not experimental), in reality software provides an ideal medium for supporting experimentation (i.e., because arbitrary interaction can be implemented). Hence, it is worth understanding the area.

Philosophy of Science. Unsurprisingly, I find the discipline to be quite insightful.

Media coverage