Digital Sensing: Intersections of Sonification and Visualization in Media Arts

Meredith Hoy

Assistant Professor, Art Department, University of Massachusetts Boston

New media artifacts provide a unique opportunity to represent data sets using multiple interfaces and sensory modalities. For example, the same data set can be, alternatively, visualized or sonified, which will produce two very different affective responses in a human “receiver.”

This presentation addressed the relationship between visualization and sonification, arguing that one of the important questions in media studies coheres around the capacity of digital media for transcoding, or for presenting data through a multiplicity of interfaces. The flexibility of digital media, manifested in the ability of computational processes to morphogenetically alter the shape of data, has often been a topic of exploration in artistic deployments of digital systems. The question that such experimentation addresses is how representing data in multiple modalities will change experience—how the registration of input by different sensory organs might uncover new aesthetic or conceptual dimensions of a given quantity of information. In a world understood to be comprised of data, it becomes possible to conceptualize sound and image as data “maps,” or guides that aid the listener or viewer’s experience of the structural principles of a given data set.

Examples of data visualization and sonification, sometimes derived from the same source, were mobilized to show how each approach reveals distinct aspects of the data at hand, and of the world that data seeks to represent. Both visualizations and sonifications can be understood in cartographic terms, as devices that map information about the world. But whereas the visual cartographic gaze notionally grasps a territory in its synchronic totality, sound maps are durational, partial, calling attention to the impossibility of total knowledge. The experience of sound throws a listener into a more contextual, embodied state than does visualization, provoking a relationship to the sound object through the vibratory, resonant emission of sonic frequencies. Sound calls attention to environmental situatedness, for example the acoustic properties of the space in which the listener is located, or competing ambient sound. Sound provides sensorial details that orient the listener to space, and to their own bodies.

The field of information visualization privileges the sense of sight, the notion that information can be most readily and rapidly absorbed if it is grasped in a visual totality. Sonification, on the other hand, is a process by which data is converted into sound, as in the case of a Geiger counter, which emits sounds that correspond to a measurement of radioactive particles in a given location. Recent experiments in sonification have begun to destabilize the traditionally privileged position of the visual register, suggesting that channeling information through alternate sensory pathways, such as sound, will yield new phenomenological and conceptual encounters. Information, itself, is shown by these projects to be unstable, variable, and shaped in particular ways by the medium used for its transmission.

As a mechanism of revelation, sonification undermines the assumption that data can only be communicated visually. In addition, sonifications show the diversity, and the complexity, of interfaces that can be deployed with digital technology. In Ryoji Ikeda’s “Test Pattern” and Yolande Harris’s “Taking Soundings,” the visual aspect of the pieces operates, like the sonic aspect, not in service of clear, usable information-transmission, but instead, to lend a more intuitive “feel” of the data at hand. Both pieces resist the resolved “image,” instead moving towards a recombinant abstraction, one that traces the shape of data rather than corresponding to a recognizable, figurative territory. The graphical representation, in this way, nudges towards a map of data itself, as the sound creates an environmental emanation of a datascape. In other words, both projects take data derived from the same source and subject it to various transcoding processes that ultimately create an immersive environment. Instead of a “natural” environment, however, we phenomenologically intersect with a territory comprised of data. In this manner, the listener/viewer encounters an ecological view of data and information, instead of the flattened cartographic depiction of data presented through single-channel visualization strategies. These encounters are multi-modal, absorptive, and durational, and ultimately open a pathway to communication between sensory modalities, and differential modes of experience and understanding.

Bio:

Meredith Hoy is Assistant Professor of Contemporary Art in the Art Department at the University of Massachusetts, Boston. She received her Ph.D. from the University of California at Berkeley in 2010. Her current book project, entitled From Point to Pixel: A Genealogy of Digital Aesthetics, traces links between contemporary digital art and modern painting. Drawing on theories of visuality, space and spatial practice, cybernetics and systems theory, phenomenology, and post-structuralism and semiotics, her research focuses on the impact of technology on art and visual culture. She has written on modern and contemporary art and architecture, generative art, information visualization, and the phenomenology of networked space. She teaches courses on modern and contemporary art, visual culture, and media studies.