Experiment-Oriented Computing

This page collects useful resources regarding Experiment-Oriented Computing (EOC), a concept introduced in my ESEC/FSE 2018 paper The Case for Experiment-Oriented Computing (local free download). Computational experimentation technology can be found in many forms, sometimes explicit and dedicated, but more often intertwined with other concerns. In almost all cases I’m aware of, however, there is no proper understanding of the wide scope that such technology can have. Nevertheless, it is useful to map the technology that does exist, since it can help us track, motivate and imagine the progress of proper EOC tools and systems.


Domenico Remps - Cabinet of Curiosities
Cabinet of Curiosities, by Domenico Remps (17th century). Collecting unique, strange, exotic or inspiring objects used to be popular among enlightened people. Thanks to efficient travel and communication, it is getting harder and harder for us to appreciate the appeal of such a thing.


A/B Test Libraries, Frameworks and Services

In Software Engineering, experimentation is often confused with mere A/B testing. This is actually a very popular technique, so it would be futile to try to curate all such tools here. Rather, I will focus on those that for some reason are particularly interesting or representative.

Experiment Tracking and Versioning

Experimentation with Users

Other Tools


Real-World Applications, Laboratories and Companies


The nature of experimentation itself:

Experimenter’s workflow:

Industrial applications:

Scientific applications:


In Human-Computer Interaction (HCI), there are tools that help users to experiment with the design of various types of artifacts. These range from very simple approaches (e.g., quick previews) to highly sophisticated ones, based on optimization or learning techniques. Beyond their specific design applications, such tools are, by definition (the ‘Human’ part of HCI), very close to users, and therefore can be rich sources of inspiration for more general experimentation interfaces.


(Computational) Scientific Discovery:

Software analytics (and related experimental concerns). Although, in principle, software analytics can be entirely passive (and therefore not experimental), in reality software provides an ideal medium for supporting experimentation (i.e., because arbitrary interaction can be implemented). Hence, it is worth understanding the area.

Philosophy of Science. Unsurprisingly, I find the discipline to be quite insightful.

Media coverage