Augmented Reality: Invention/Reinvention, an Introduction

Renate Ferro

Cornell University, Department of Art, Visiting Associate Professor, Artist

Augmented reality allows artists to critically manipulate the continuum between reality and virtuality testing the abilities to discern fact and fiction. Whether appropriating proprietary software such as Layar or open–source software, artists continue to push the possibilities of augmented reality experiences through 3D modeling, GPS, video, web links, sounds or other digital media via the ubiquity of digital display. Augmented reality introduces both narrative and performativity into a blended continuum between the real and the imagined.

Using a smart phone, tablet or goggles, the user accesses a connection with direct or indirect physical surroundings via coded data. At the crux of this augmented reality continuum, is the moving body that activates information through kinesthetic gestures of the head, the body and the hand through touch, the torque, or tilted movements. Through the combination of human and machine intervention, the real world and that of computer simulation is linked to expand human cognition and perception.

While science fiction writer William Gibson imaginatively constructed virtual possibilities through fiction in the 1980s, and the French philosopher Jean Baudrillard theorized about simulation in 1994, it was fiction writer L. Frank Baum who first conceived of what we know now to be Augmented Reality in his children’s book The Master Key. [1] [2]

On the other hand, continued the Demon, some people with fierce countenances are kindly by nature, and many who appear to be evil are in reality honorable and trustworthy. Therefore, that you may judge all your fellow-creatures truly, and know upon whom to depend, I give you the Character Marker. It consists of this pair of spectacles. While you wear them every one you meet will be marked upon the forehead with a letter indicating his or her character. The good will bear the letter ‘G,’ the evil the letter ‘E.’ The wise will be marked with a ‘W’ and the foolish with an ‘F.’ The kind will show a ‘K’ upon their foreheads and the cruel a letter ‘C.’ Thus you may determine by a single look the true natures of all those you encounter.

While Baum conceived of the Character Marker in 1901, Charlie Gere in his book Digital Culture makes the case that it was the coexistence of a burgeoning capital economy and the post-war military industrial complex that created a nurturing environment for innovative technology including augmented reality in the twentieth century. Gere reminds us that technology alone was not the only promulgator but additionally “techno-scientific discourses about information and systems, avant-garde art practice, counter-cultural utopianism, critical theory and philosophy” all played a part in its innovation.” [4] Whereas, social, cultural, and political structures were just as important as technological innovation, one of the earliest precursors to what we now know as augmented reality was in 1962 by cinematographer Morton Heilig. Helig’s Sensorama was a motorcycle simulator that enabled a viewer to peer through a large immersive visual display enhancing the sights, sounds, smells and movements of a motorcycle ride through 3D feedback. [5] Ivan Sutherland, another innovator, originally trained as a graduate student at MIT in a secretive design lab, built the early platform that converged computer generated graphics within virtuality in a program called Sketchpad. [6] In 1968 Sutherland went on to devise the first six points of freedom structure called The Sword of Damocles. Six points of freedom implants the virtual and the real to become merged in three–dimensional space to coincide with a user’s movements. Thus changes of position can be attained including up and down, forward and backward, left and right within a three-dimensional perpendicular axes. [7]

It was in 1982 that Boeing engineers Tom Caudell and David Mizell coined the terminology augmented reality in their research, design, and construction initiatives at Boeing, one of the most renowned aerospace companies and a leader in manufacturing commercial, defense, and space vehicles.

The enabling technology for this access interface is a heads-up (see-thru) display head set (we call it the “I-IUDset”), combined with head position sensing and workplace registration systems. This technology is used to “augment” the visual field of the user with information necessary in the performance of the current task, and therefore we refer to the technology as “augmented reality” (AR)…A primary difference between what we have begun referring to as “full” virtual reality (VR) and this “augmented reality” concept is in the complexity of the projected graphical objects. In our system, only simple wire frames, template outlines, designators, and text are displayed and animated. An immediate result of this difference is that augmented reality systems can be driven by standard and inexpensive microprocessors, since PC class processors have the computational power to transform and plot simple graphics in real time. [8]

Many of the technological consumer innovations we use today were first innovated and manufactured within the military industrial complex and Gere’s historical compilation fascinatingly recounts that progression especially when business ventures, military strategy, and computing initiatives converged in the latter part of the century. [9] In fact, Caudell and Mizell’s augmented reality glasses, conceived before Google glasses or others we know today, were fabricated by Boeing to practically aid in the assembling of fighter pilot aircraft.

As Gere reflects on the cross-disciplinary cultural pollination that enabled augmented reality and other technological advances, all seven of our panel members link vital cultural issues within their projects that are embedded within augmented reality. The virtual realms of artistic creation reflect the cultural realisms of identity, race, geographic location, gender, sexuality and others. Through critical artistic resistance the artists featured in the panel Augmented Reality: Invention/Reinvention utilize what I refer to as critical spatial practice where the claims of social responsibility remain within the intersections of art, geography, architecture and activism. In the introductory post I wrote for a discussion on the list-serve –empyre– soft–skinned space I wrote in September of 2007, “The alignment of criticality with cyber configurations of space permits especially creative skins of networks, resources, and discussions whose resulting configurations range from texts and performances to buildings and installations.” [10] Augmented reality is exemplary of a critical spatial practice where cultural contexts can be analyzed within a blend of the real and the virtual.

Within the critical spatial practices of augmented reality resides the software that enables augmented or supplemental spaces to be manifested and inhabited. The first open–source augmented reality programming code called ARToolKit, designed by Hirokazu Kato, was introduced in 1999 via the world wide web. The evolution of ARToolKit continues today with its website documenting the history of its growth. [11] Today it is not uncommon for artist’s to appropriate proprietary and commercial software in re-purposing ordinarily intended commercial software for artistic use. One of the advantages of using and incorporating software packages into the practice and teaching of augmented reality is the ease of use and the minimal costs associated with simple programmatic creations. While ubiquity, ease, and affordability are certainly positive aspects, one of the inevitable and adverse effects that I experienced in 2015 was that of obsolescence. After setting up an advanced digital media class with the software Mateo/Junaio, Apple announced that they had purchased the company making it no longer available for use. Many artists now are seeking alternatives for realizing their creative interventions in Augmented Reality, realizing that these software systems come and go through company mergers and acquisitions.

Though the concepts and applications of augmented reality have been around for a long time, in 2016 the possibilities of the technology seem endless. While military applications will remain, the technology is leaching out into the disciplines of medicine, sports, travel, advertising among others. In conceiving the topic Augmented Reality: Invention/Reinvention, the New Media Caucus’ Exhibition and Events Committee agreed that the timing was right to emphasize the more critical and contemplative nature of artistic mediations into augmented reality practices for the College Art Association 2016 panel in Washington, DC.

Artist Jane Prophet shares her work on socially motivated augmented reality projects in overpopulated Hong Kong. Prophet’s essay also accounts for early theoretical influences on the conceptualization of human-machine interfaces. Artist Will Pappenheimer writes about his project, Skywrite, a political augmented reality intervention onto the monuments and skyine of Washington, DC for the Occupy Movement. Tamiko Thiel illuminates upon her augmented reality work at the Venice Biennale (and other sites), Shades of Absence. Together Pappenheimer and Thiel’s collaboration, Biomer Skelter, exposes a virtual space for the participant to deliberate upon our global environmental changes. Also politically motivated, Claudia Pederson and Nicholas Knouf team up to create the augmented reality artist’s book, Art for Spooks. Pederson and Knouf reveal the complex images that act almost as double entendres using what they call a paranoiac-critical method to draw out the multiple, and often times creative, meanings of the images. Lastly, collaborators Nathan Schafer and Patrick Lichty take us through their augmented reality explorations within Alaska’s indigenous territories. All of these artists agree that the qualities of site-specificity and audience engagement and participation make for culturally implicated artistic visions through the technology of augmented reality.


1. Introductory Post by Patrick Lichty on –empyre–soft–skinned–space, “ Introduction: Re-Emergence of the Augment,” April, 2011, (accessed May 1, 2016).

2. – Project – Augmented Reality in “Augmented Reality History, Background and Philosophy” (accessed May 1, 2016).

3. Frank L. Baum, The master key: an electrical fairy tale founded upon the mysteries of electricity and the optimism of its devotees. It was written for boys, but others may read it. (Westport, Conn: Huperion Press, 1974),19.

4. Gere, Charlie, Digital Culture (London: Reaktion Books, 2004), 20.

5. Kipper, Greg and Rampolla, Joseph, Augmented Reality: An Emerging Technologies Guie to AR, 1st edition, (Syngress, 2013), accessed May 1, 2016,

6. Gere, Charlie, Digital Culture, 71.

7. – Project – Augmented Reality “Augmented Reality History, Background and Philosophy” (accessed May 1, 2016).

8. T.P Caudell, D.W. Mizell. Caudell, “Augmented reality: An application of heads-up display technology to manual manufacturing processes.” (Paper presented at the System Sciences for the Twenty-Fifth Hawaii International Conference on System Sciences, 1992).

9. Gere, Charlie, Digital Culture, 62-72.

10. Introductory post by Ferro Renate on –empyre–soft–skinned–space, “Critical Spatial Practice,” (accessed May 1, 2016).

11. ARToolKit website, (accessed May 1, 2016).


Renate Ferro (US) is a conceptual artist working in emerging technology and culture. Most recently her work has been featured at Nanyang Technological Institute (Singapore), The Freud Museum (London), The Dorksy Gallery (NY), The Hemispheric Institute and FOMMA (Mexico), The Janus Pannonius Muzeum (Hungary). Her work has been published in the journals Diacritics, Theatre Journal, and Epoch. She is the managing moderator for the online new media list serv –empyre–soft–skinned space. Ferro is a Visiting Associate Professor in the Department of Art at Cornell University teaching digital media and theory.