New Media Art Takes as its Subject the Processes of the Brain/Mind

Andrea Ackerman
Independent artist, theorist, psychiatrist

Advances in neuroscience are elucidating brain/mind mechanisms with powerful new techniques that can be used to investigate physiological functioning in awake, conscious subjects. These techniques include functional MRI (spatial localization), magnetic encephalography (MEG - temporal localization), recordings from individual neurons and ensembles, microscopic visualizations of individual neuron growth (in situ) and neuro-chemical analyses of processes like memory. Investigators of the brain/mind (scientists, philosophers) view data on mental processes from two positions. The connectionist, materialist, monist, and physicalist position holds that all mental phenomena are the results of physical events in the brain (as it is embedded in the body). The dualist position holds that some mental phenomena are nonphysical (as the soul or spirit). I am a physicalist and this paper is from the physicalist viewpoint.

The most significant shift in art in the last 150 years is that art now primarily takes as its subject the process of experience rather than the content of experience. The process of experience is congruent with the organization and functioning of the brain/mind. Today's most advanced art takes as its subject these newly elucidated brain/mind processes. An underlying reason for this choice is that our culture today is focused on definitively understanding and manipulating the biological mechanisms that make us human or (human/machine) and manipulating our own evolution. New media art is particularly well suited to these subjects because it uses similar structures/objects i.e. those of artificial intelligence, artificial life, etc.

The current evolutionary moment relates to two previous moments of radical evolutionary change in the human mind/brain. The first shift occurred about 40,000 years ago when a specialized area of the brain evolved the capacity for oral language. The second shift occurred about 6,000 years ago and utilized existing brain pathways in new ways, enabling the capacity for read and written language.(1) Humans are now prosthetically extended across technological platforms; our evolution is a human/machine evolution. There is a shift in preferences away from the natural to the synthetic; the synthetic is the new native for the brain/mind. A vivid example of human/machine evolution is the pioneering work by neuro-researcher Miguel Nicolelis in "reading monkey thought."(2) Nicolelis uses electrode arrays implanted into a monkey's brain to record signals of volitional motor neuron ensembles in the cerebral cortex. These recordings are made while the monkey manipulates a hand-held joystick to move a shape in a video game. The distinctive patterns of each movement are identified and then used to enable the monkey to control a cursor purely through thought. This technique is also used to control prosthetic limbs or remote robots through thought alone (image 1). Another researcher, Matt Wilson listens to rats' thoughts and dreams. He listens to the audio output of implanted electrode arrays while awake rats run mazes. He then listens to the audio output when the rats are asleep and can recognize which mazes they are dreaming of and when their dream process begins to create new mazes.(3)

Human brain evolution is one example of the dynamic oscillation between entropy and complexity that occurs in all brain/mind processes. Neuroplasticity in individual brain development, and memory and learning processes are others. During individual epigenetic development, in preprogrammed ways along with some randomness, neurons proliferate in the fetus, migrate into place and make an overabundance of synaptic connections. The greatest number of synapses is present at around 24 months of age-different areas peaking at different times.(4) Used connections are strengthened (selected) and unused neurons/synapses are pruned – neurons that fire together wire together.(5) The neurologist V.S.Ramachandran explains synesthesia (i.e. seeing the number 5 in blue) as the result of aberrant cross-modal connections remaining between the color area and the adjacent number area in the brain, connections which normally would have been pruned. In his view, creativity results from a richness of unpruned cross-modal connections forming a high capacity for metaphor.(6) (Example of cross-modal metaphor/simile "her voice was so sharp it cut through me like a knife" - sharpness is connected in the auditory, visual and touch modalities.)

The newly understood cellular and molecular mechanisms of memory show that memories are unstable during the recollection process. Actively recalling a memory involves removing it from storage in the brain. After recollection, it is reconsolidated back into stable form. Re-storage depends on protein synthesis which if blocked or manipulated means the memory will be lost or changed (thus the ease of implanting false memories.(7, 8) Memory comes in various forms related to how it is organized in the brain some categories include explicit and implicit memory, declarative (language based) and non-declarative (procedural /skill based), episodic memory (personal experiences) and semantic memories (learned facts not personally experienced), spatial memory, etc…(9)

A newly found brain system is the mirror neuron system. Discovered in the early 1990's, mirror neurons are cortical neuron systems that fire both during one's performance of an action or expression of emotion and during observation of another enacting these. Mirror neuron systems seem to form the basis of empathy and the appreciation of art; we understand what others feel by the inner representation or imitation of the observed action or expression. The representation is matched to an existing representation and used to modulate our emotional experience. Thus, empathic individuals exhibit unconscious mimicry of the postures, mannerisms, and facial expressions of others (the chameleon effect) more than non-empathic individuals.(10)

A major division in brain process is conscious vs. unconscious experience (the neurological not Freudian unconscious). The classic example of unconscious experience is blindsight. Blindsight allows a person with damage to the visual cortex to locate a light spot in space with 99% accuracy even as the person protests the total inability to see the spot. The blind-sighted person uses a pathway from the eye to the brainstem, an evolutionarily older part of the brain not able to be conscious to locate the light spot in space, rather than the visual cortex, an evolutionarily younger area of the brain able to reach consciousness.(11)

What in normal experience controls the content of consciousness is the subject of intense study. How conscious experience mediated through different sensory areas is integrated into a serial stream of coherent whole experiences is called the binding problem. A number of prominent theories hypothesize that two brain/mind systems working together control the stream of consciousness, the thalamocortical axis and the prefrontal cortex (PFC) - the latter linked with “working memory”. The thalamus is called “the switchboard of the brain” because virtually all information conveyed from the peripheral sense organs is relayed through the thalamus. The thalamus is the predominant source of internally generated activation of the cortex; the activation is mediated by the thalamic reticular nucleus (TRN). All thalamic relay neurons pass through the TRN, which opens and closes the “gates” to the cortex. These gates can control the simultaneous firing of broad neuronal populations; this leads to neuronal synchrony. Neuronal synchrony both coordinates neuron firing within local areas and across widespread regions, and enhances the activation of neurons in the firing pathways. The TRN has two activation modes “burst firing” mode, and continuous (tonic) firing mode. The tonic firing pattern triggers looping activation in cortical circuits and this looping activity has the capacity to amplify one loop over others. The most compelling theory proposes that this recursive process allows the interacting reentry loops to reinforce each other and compete with each other, so that the dominant loop(s) bring the experience to consciousness. The PFC, the other system controlling the content of consciousness, seems to be required for the experience of an integrated consciousness. The PFC appears to be specialized for maintaining or modifying the focus of attention and updating representations in working memory flexibly and quickly. Some scenarios propose that the large-scale looping competition set up by the thalamus among the cortical regions is biased by top-down projections from the PFC, i.e. it is the PFC that picks the winner from moment to moment.(12, 13, 14, 15) Contemplating this mechanism, it is interesting to imagine that human/machine evolution may one day lead to a mind capable of multiple (rather than unitary) consciousnesses.

It is fascinating that certain neuroscience experiments designed to study the biased competition for reaching consciousness and others which manipulate memory are structurally very similar to the series of works, Walks,(image 2) by the artist Janet Cardiff. In these walks, the viewer/audience is instructed to walk a path during which time he/she is receiving audiovisual input related to the environment in the present, from the past, as if the speaker is there with the viewer, and is in fact not there but somewhere else. There are sounds which are appropriate to the current environment and sounds not. The experience is as if there are streams of sensory stimuli, which loop, magnify, cancel, and conflict, competing with each other for which one will dominate (or won't) from one moment of consciousness to the next. Which one will reconfigure, confuse or implant a memory and how?(16)

If the internet or an intranet (LAN) has a consciousness (as the brain neural net does) what would it be like? This is the subject of Ben Rubin's Listening Post, David Rokeby's n-Chant, and Lynn Hershman Leeson's Dina. Listening Post (image 3) is a curved wall grid of screens displaying text culled from 105 internet chat room users and projecting sound, answering the question “What would 100,000 people chatting on the internet sound and look like [as one]?”. In six looping movements (the first movement starting with chat room text that begins with “I am”), synthesized voice, sound tones and text are mixed to create a compelling, recursively evolving, sensorily accessible surface to an associatively linked sub-region of the internet endowing it with a rudimentary consciousness.(17) In n-Chant (image 4) each of the [n] numbers of computers arranged in an LAN array suspended from the ceiling speaks English sentences in its own voice. In the undisturbed equilibrium resting state they chant with each other. A visitor speaking into a microphone, interrupting and distracting the network, disrupts the coherence of the chanting and the “state of mind” (consciousness) shifts around until the equilibrium loops back again.(18) Dina, (image 5) a literally humanoid looking consciousness is represented as a woman's lip-synched face on a screen. Through voice recognition and artificial intelligence software she engages in social conversation with the viewer using internet searches to help her answer questions.(19)

In the DVD Placebo (image 6) Saskia Olde Wolbers uses an intimate first person stream of consciousness narrative voiceover to describe and evoke individuals existing in a fluid mental space, enclosed worlds where fantasy and reality, real memories, false memories are blurred together. The video imagery is of scale-less interior/exterior landscapes startlingly reminiscent of the neuronal architecture of the brain - networked treelike forms evoking dendrites, straight bundles evoking axonal white matter connections and budding bubbles crossing small gaps evoking synaptic transmissions.(20)

Mirror neurons are the subject of Thomas Struth's museum pictures (image 7). The photographed museum-goers eerily align themselves in shapes reflecting the images they are viewing and their installation/display contexts. Struth's photographs are a representation of his own consciousness as it mirrors the consciousness of the artist who created the displayed work of art (and as it is mediated through the museum goers' consciousnesses). The museum pictures point out the critical role of inner imitation and inner motor representation (enactment), the process externalized in his photographs, that is required for empathy and art appreciation.(21, 22)

Cory Archangel replaces the commonly-experienced-as-unalterable chips in computer games with his own custom made chips (burned with an altered game/graphics code) to explore silicon and neuronal plasticity in memory and learning. The assumed fixed game chip is altered “learning” to be a new game, highlighting the newly appreciated instability/plasticity of brain connections in the processes of memory and learning. Super Mario Brothers is transformed into Super Mario Clouds by deleting everything but the clouds; the shooting game Hogan's Alley is transformed to I Shot Andy Warhol by replacing the generic targets with images of Pope John II, Flavor Flav, and Andy Warhol(23) (image 8).

1. Maryanne Wolf, Proust and the Squid: The Story and Science of the Reading Brain New York: Harper Collins, 2007.

2. Miguel Nicolelis, “Actions from Thoughts”, Nature, 2001, Jan 18:409(6818):403-7
“Actions from Thoughts”, in conference Your Brain and Yourself,

3. Matt Wilson interviewed on Radiolab episode, “Sleep”, (accessed April 21,2008)

4. Joseph LeDoux, Synaptic Self: How Our Brains Become Who We Are. New York: Penguin, 2002.

5. LeDoux

6. V.S. Ramachandran, A Brief Tour of Human Consciousness: From Impostor Poodles to Purple Numbers, New York: PI Press, 2004.

7. Eric Kandel, “The Molecular Biology of Memory Storage: A Dialog between Genes and Synapses”, Nobel Lectures in Physiology or Medicine 1996-2000 edited by Hans Jornvall Apr 2003 (Editor) , World Scientific,

8. LeDoux

9. LeDoux

10. Laurie Carr, Marco Iacoboni, et al, “Neural Mechanisms of Empathy in Humans: A relay from Neural systems for imitation to limbic areas”, PNAS, 2003,

11. Ramachandran

12. Richard H. Granger and Robert A. Hearn “Models of thalamocortical system” Scholarpedia 2(11):1796, 2007,

13. Giulio Tononi and Gerald M. Edelman, “Consciousness and Complexity” Science Vol. 282, December 4 1998,

14. Tiago V. Maia and Axel Cleeremans, “Consciousness: converging insights from connectionist modeling and neuroscience”, TRENDS in Cognitive Sciences Vol.9 No.8 August 2005.

15. Giulio Tononi, “An information integration theory of consciousness”, BMC Neuroscience 2004, 5:42 doi:10.1186/1471-2202-5-42,

16. Janet Cardiff with George Bures Miller website,,

17. Ben Rubin's website, all accessed April 21, 2008.

18. David Rokeby's website,

19. Lynn Hershman Leeson's website,

20. Saskia Olde Wolbers webpage,,

21. Phyllis Tuchman “On Thomas Struth's "Museum Photographs", magazine feature,

22. Barbara Stafford, Echo Objects, The Cognitive Work of Images, Chicago, University of Chicago Press, 2007.

23. Cory Arcangel's website, Copyright Andrea Ackerman, 2008. All rights reserved.