Spring 2008

| v.04 n.01|

Intermedia Patterns: Notes for Building a Toolkit

The authors have collaborated over the course of many years to develop an open and generalized approach to describing and creating Intermedia art, particularly with digital media. Their work together includes co-chairing the panels "Erasing Boundaries" at Siggraph 2001 (Los Angeles) and "Intermedia Art in the Digital Age" at ISEA 2002 (Nagoya, Japan). These notes are the product of an ongoing conversation, extended over many years and in many different media, that has lately turned to the possibility of creating a toolkit of "design patterns" for Intermedia art. Although the primary focus of the toolkit is on art production, many examples of Intermedia patterns can be found in scientific and data visualization and human-computer interface design, areas where the toolkit may also prove useful.

Christopher Alexander's influential book "A Pattern Language"(1) is generally credited as the point of origin of design patterns. Alexander's approach to architecture as a domain of design problems over a wide range of scales which could be tackled with a toolkit of modular, interrelated solutions has been adapted with great success for object-oriented software design(2). In general, architectural and software design patterns do not represent a global approach to a problem domain. They emerge from practice, not from theory. They name and classify for the sake of comparison and clarity, not to fit entities into a system. While different design patterns can operate together to create larger scale design patterns, their fundamental philosophy is "mix and match" rather than "categorize and conquer." Therein lies their charm. They provide descriptive clarity without claiming universal authority, and they avoid the essentialism that all too often muddles theories of intermedial or intersensory art, such as Baudelaire's "immense keyboard of universal correspondences"(3) or any number of color-harmony systems. We may not be able to establish universal correspondences, but we can describe structures that work in multiple media, upon multiple senses, and point to a variety of cultural and historical antecedents for their interpretation. Furthermore, we assert that these structures can be of use to the scholar, artist, or researcher in understanding or creating Intermedia art or visualizations, precisely because we look to practice in the field to discover them.

Architectural patterns involve habitable spaces; software patterns involve programming structures--Intermedia patterns involve conceptual mapping, whereby structures in one medium or sensory channel are brought into correspondence with structures in another. Conceptual mapping may operate in both formal and symbolic strata, as a semantically injected metaphor of the sort we are used to in ordinary language, or as a structural mapping, typical of formal languages such as musical composition. In keeping with current metaphor theory, we may consider that the mapping of different domains onto each other creates a new entity, blending the two, negotiating their disparities, and extending their potential for representation. Just as metaphor develops complex fields of cultural application and association that Fauconnier and Turner call "integration networks,"(4), conceptual mappings in Intermedia art can develop rich associations that operate symbolically and emotionally. The following list and descriptions encapsulate some of the authors' initial ideas for a toolbox of Intermedia patterns. It is by no means complete (design pattern toolboxes never are) but we hope that it captures at least some of the most common conceptual mappings used in Intermedia art.

Some Preliminary Definitions

Event stream: A series of spatially or temporally ordered elements. We will use "element" and "event" to some extent interchangeably. Element suggests ordering in space, whereas event suggests ordering in time. Events and elements can contain other events or elements as substructures. Through the fundamental metaphor of space/time equivalence, they become interchangeable terms in a blended domain.

Mapping: The creation of correspondences between one set of elements (the domain) and another set of elements (the range). As the definition suggests, mapping is a formal operation, and may be characterized by a set of rules. Correspondences emerge from an analysis of the parameters that shape events. Parameters from different media are paired up through underlying similarities. Elements correspond because they have similar behavior.

Metaphor: A rhetorical trope that compares different subjects, developing the sense of one by association with the other. A metaphor is a type of conceptual mapping. The term generally suggests a less formal, more subjective set of correspondences than does "mapping." Correspondences emerge from an understanding of the semantic content and cultural context of events. Elements correspond because they have similar meaning.

Selection rule: a method for selecting events based on a set of criteria. Also known as an event filter.

Sensory channel: Any of the various human senses--vision, audition, touch, etc.--or cognitive functions within the senses. For example, vision discriminates brightness and color, and also different kinds of spatial patterns, all of which can be addressed by artworks, computer human interfaces, architecture, etc.

Temporal window: A period of time within which to view events. A window typically represents a slice of a stream in which diachronic form is realized. The "musical present" is a significant temporal window (of about seven seconds duration) based on human cognition. A melodic fragment, a motif, or a harmonic sequence may fit within the musical present.

Single Stream Patterns

Intermedia art may be realized in a single medium or in multiple media, in multiple sensory channels or in a single channel. Within a medium or a channel, it may present single or multiple event-streams. Kurt Schwitters' Ursonate, Apollinaire's Calligrammes, Mallarme's Un coup de D?s, or Thomas Wilfred's lumia are examples of Intermedia art within a single medium--Un Coup de D?s also presents a clear example of multiple streams. We have attempted to distinguish patterns that can be presented in a single stream from those that arise from the relationships of multiple streams.

1. Time equals Space Metaphor

Context: Ordering over time is represented by ordering over space.

Problem: Events occurring over time may be difficult to perceive as a whole, possibly because the temporal window they occupy is very large (or very small) or because they don't draw our attention.

Solution: Presenting time sequences as an image, subjectively available in a single moment of time, provides a means of displaying temporal patterns.

As Fauconnier and Turner point out, the "time = space" metaphor(5) is universal in human culture, and extremely rich in emergent structures. Its applications are so common that we may not immediately think of them as intermedial. In "telling time" we are unlikely to stop to think that clocks are mapping time onto space. The subjective dimension of time to space mapping is everywhere evident in language (time flies or crawls, a second stretches to eternity or the years vanish in an instant). Scientific visualization employs objective mapping of time onto space, since precise measure is critical.

Examples: Computer art pioneer Ben Laposky used analog signal processing equipment to transform time-based signals into two-dimensional images on an oscilloscope. He called the photographs he made of the oscilloscope display Oscillons, which he described as a form of visual music (http://www.atariarchives.org/artist/sec6.php). Jack Ox's paintings developed from Bruckner's Eighth Symphony maps time onto space through a process of musicological analysis and visual coding (http://www.jackox.net/pages/bruckner/bruckner1.html).

2. Gradient

Context: Continuous ranges of values can be distinguished as properties of different media: for example, light to dark, loud to soft, the spectrum of visible color, the range of audible frequencies of sound.

Problem: In working out the correspondences between properties of different media, elements that display a continuous range of variation are good candidates for mappings on each other.

Solution: A range of values in one medium is mapped onto a range of values in another. The mapping may be continuous or consist of discrete steps, but the relation of each successive state is always sequential.

Mappings of color to musical pitch class have a considerable history, in which sequence is not always preserved--e.g., Scriabin's or Rimsky-Korsakov's highly subjective selection of color correspondences do not align pitch classes sequentially to the color spectrum.

Cyclic gradients are a specific subclass of gradients: pitch class is a prime example, as is color spectrum. Thus, the sequence of pitch class in an ascending chromatic scale repeats at intervals of 12 steps, due to the subjective equivalence of octaves. The spectrum of color wraps around in a circle, but does not display the octave-equivalence of pitch classes.

Example: Scriabin's "color-music" correspondences present a stepped gradient color wheel that is, interestingly, not a pure spectrum (the color sequence is chromatic but varies in brightness) and   mapped to the circle of fifths and not to a chromatic scale (see http://prometheus.kai.ru/skriab_e.htm).

Scriabin's color and pitch wheel from Prometheus, as constructed from Bulat Galeyev's description at http://prometheus.kai.ru/skriab_e.htm

3. Container, Boundary, Edge, Inside/Outside

Context: Several related patterns arise from the separation of events into classifications, typically through the action of a selection rule. For example, the orchestra is a container for families of instruments, each of which contains individual instruments, selected by criteria of construction, performance practice, timbre, spectral range, etc. The Container is one of the most broadly used metaphors, although Lakoff and Johnson (6) treat the Container as an image schema, variously called "in-out schemata" or "in-out orientation."

Problem: The domain of an Intermedia mapping can be articulated into two or more distinct sets. We seek a similar articulation in its range.

Solution: By examining the selection rules in the domain we attempt to establish a similar set of rules over the range. Through correspondences in the rules we can create a blended domain. If the container metaphor has been articulated a priori through language or culture, we may need to develop the selection rules through careful analysis its origins and use.

Example: Consider as a classic example Stephane Mallarme's Un Coup de Des, a poem that operates visually and conceptually as a shipwreck, where typography distinguishes separate textual streams strewn across the page. (PDF downloadable from http://www.direz.org/mallarme/index.php?n=Mallarme.CoupDeDes). Each typographic style encloses a sequence of images that can be read separately or concurrently within the fragmentary swirl of the text--even the full title is broken apart. One experiences the synchronic development of the poem as it unfolds on the page and the diachronic structure of its separate streams in different time frames. The synchronic structure emerges in the present in the act of reading, whereas the diachronic structure becomes apparent in retrospect, reassembled in memory.

4. Motion

Context: Change in a spatial or temporal property is mapped (metaphorically or physically) onto motion through space.

Problem: The way in which something changes may be read as unremarkable or even static because we do not readily detect it.

Solution: Change in one or more properties of an element, typically over time, is mapped onto change in spatial position or location in a spatial environment. The human perceptual system readily detects motion. Motion may map continuous change or a sudden transition (i.e., move along a gradient or cross an edge). For interactive digital intermedia, spatial motion can be plotted on a control surface, adding a kinesthetic dimension to the mapping.

In a broader cultural context, motion brings with it a wealth of metaphorical associations. The path traced out by motion (i.e., a cycle, a spiral, a parabolic arc) can reveal hidden qualities of the domain. Motion mapping works particularly well when both the domain and the range are presented together, as in multiple stream patterns.

Examples: John Whitney's experimental animations represent the numerical relationships of the harmonic series as cyclic motion (for example in Arabesque, http://www.siggraph.org/artdesign/profile/whitney/arabesque.html). Natalie Jerimijenko's Ride the Bull Market maps stock market fluctuations leading up to the crash of 1987 onto the bucking of a mechanical bull. Composer Panaiotis's soundtrack for a simulation of kidney function (http://www.hpc.unm.edu/~panaioti/Pages/B_Demos_Neph.html) maps virtual camera motion over different segments of a 3D simulation to musical motifs. The composer's intention in developing the music further is to encode chemical gradient levels into the audio signals.

Multiple Stream Patterns

When two or more streams of events are presented together, new patterns arise from their relationships in time and space. There are a great many possible relationships–a wealth of contrapuntal patterns, different types of collections, sequences, taxonomies, and networks. We shall begin with three patterns–juxtaposition, intersection, and complementarity–that can govern large scale compositional entities. Juxtaposition, the presentation of non-related or independent events, in many respects corresponds to production of events using aleatoric selection rules. Intersection is manifest in partial mappings and loose associations, where streams are composed of both related and non-related events. Complementarity involves tightly coupled events, parallel motion or "synesthetic" selection rules and correspondences. Applied at different points in an intermedia artwork, juxtaposition, intersection, and complementarity can operate as a metalanguage of cross-modal composition, a counterpoint of rupture and cohesion.

5. Juxtaposition

Context: There are no correspondences between stream events, they are simply presented together. Also known as: Independence.

Problem: As artists, we don't always want to obey correspondences, willy-nilly. In such a non-conformist spirit, Satie remarked of Wagner's leit-motifs that he could not see why the orchestra had to sneeze every time the hero came on stage. Sometimes we want to leave media free to display their particular qualities.

Solution: Chance operations and other selection rules that behave as "no rule" allow us to present events arbitrarily. Note that the use of chance operations suggests that juxtaposition isn't necessarily easy to pull off--just telling everyone to do their thing may not suffice.

Examples: At one extreme, we could consider some of the collaborative work of John Cage and Merce Cunningham. Cage developed music from aleatoric processes, Cunningham did the same with dance. They displayed their work together, almost always without intentional synchronization of sound to movement; nevertheless, happening together in the same space, their work evokes a strong sense of correspondence. This may be partly a result of the way human cognition works, seeking pattern even where none is present, but it may also arise from the degree to which randomness operates as a pattern.  Variations V (http://www.medienkunstnetz.de/works/variations-v/video/1/ ), which Cage created in 1965 for the Merce Cunningham Dance Company, uses dancers' movement past photocells to trigger sound, so it is not precisely unsynchronized, but the sound, dance, and projected images are otherwise independent.

6. Intersection

Context: Some events or properties of events correspond, some do not. Correspondences pertain to only some aspects of events/elements, while others are free to vary. The intersection, where correspondences operate, often behaves as a blended domain, where the possibilities for subjective content (symbolism, emotion, cultural reference) expand the potential for meaning.

Problem: Often, mappings between domain and range are incomplete or contradictory.

Solution: Rather than forcing media to work in lockstep, we define a zone of intersection, where properties can effectively correspond, while leaving other properties free to vary, possibly in accordance with traditional practices in the particular medium. This pattern has much in common with Fauconnier and Turner's "double-scope networks" which "blend inputs with different (and often clashing) organizing frames to produce creative emergent frame structure in a blended space" (reference 4., p. xiii, Overview by Chapters for chapter 7)

Examples: A great many intermedia works fall into this category, though not necessarily through conscious application as a compositional strategy. The famous "Luce" part in the score for Scriabin's Prometheus involves correspondences to the harmonies of the musical parts (see Irina Vanechkina's harmonic analysis at http://prometheus.kai.ru/in2_e.htm) and symbolic evocations of the programmatic content of the work. Sometimes a medial structure or database can be used as a point of departure for this pattern. Composer Stephen Dembski and artist Paul Hertz in their collaborative work Fool's Paradise used a database of agreed-upon structural and symbolic material from which they derived both spatial imagery (a virtual world) and music (a cycle of 48 songs). (http://ignotus.com/paradise/foolspar/composition.html, http://ignotus.com/paradise/)

7. Complementarity

Context: Events correspond one-to-one, parametric mappings are direct and exhaustive. Also known as: synesthetic mapping, homeomorphism, isomorphism.

Problem: A tight coupling between simultaneous events might produce an apparent fusion of perceptions, or synesthesia. Like altered perceptions of time (synchronia), and sensory deprivation (anesthesia), synesthesia presents powerful symbolic constellations that may also point to neurological states.

Solution: Acting rather like strict parallel counterpoint, tightly coupled simultaneous variation of corresponding parameters in different sensory channels can prove an effective compositional tool. Whether it elicits the experience of sensory fusion is at best unpredictable. What works as a poetic symbol may not translate well into lived experience; nevertheless, we can create events that evoke the symbolic experience.

Examples: Norman McClaren's Synchromy from 1971 (video: http://www.youtube.com/watch?v=7hl5dbc3yKw, National Film Board of Canada documentation: http://www.nfb.ca/collection/films/fiche/?id=10490) is a pure example of this pattern: the animated image is derived by processing the optical soundtrack printed on the edge of the film with an optical printer. McClaren freely varies the color and spatial placement of columns of horizontal bars derived from images of the soundtrack, but we have no trouble "seeing" the frequency and amplitude of the sound as spatial density and width of the columns. The VJ technique of "transcoding" video and audio using realtime signal processing, as popular in today's club scene as light shows were in hippie times, is another example of tight coupling of different sensory modalities. A great many software applications enable transcoding of one sort or another.

There are of course many other intermedia patterns that can be discovered. We limit our exposition due to constraints of space, but hope that we have offered a glimpse of the potential utility of our approach. The authors' ongoing conversation is partly documented in their blog at: http://ignotus.com/intermedia/.They expect to develop their list of intermedia patterns on this site (see the Toolkit page). Persons interested in commenting on posts or contributing should contact the site administrator, ignotus+intermedia@gmail.com.

Footnotes

1. Christopher Alexander, Sara Ishikawa, Murray Silverstein, A Pattern Language: Towns, Buildings, Construction. Oxford University Press, 1977

2. Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides, Design Patterns: Elements of Reusable Object-Oriented Software, Addison-Wesley, 1994

3. Charles Baudelaire, "L'Exposition Universelle," p. 215 of Curiosit?s Esth?tiques in complete works, 1885 edition, Paris: Calman L?vy

4. Gilles Fauconnier and Mark Turner, The Way We Think: Conceptual Blending and the Mind's Hidden Complexities. New York: Basic Books. 2003.

5. G. Fauconnier and M. Turner, "Rethinking Metaphor," in Ray Gibbs, ed., Cambridge Handbook of Metaphor and Thought. Cambridge University Press, available at http://cogsci.ucsd.edu/~faucon/RethinkingMetaphor19f06.pdf

6. George Lakoff and Mark Johnson, Metaphors we live by. Chicago: University of Chicago Press, 1980.