AUDIO-VISUAL RESEARCH 2011-2015
Three creative principles
I have endeavoured to clarify how complex musical musical constructions can be fully experienced in audio-visual performance. This is challenging, because usually vision dominates over audition. My research draws from from artistic practice as well as literature in music, interaction design, audio-visual theory, psychology and neuroscience. The investigations lead to three creative principles for instrument design, composition and performance:
PRINCIPLE OF SONIC COMPLEXITY : to threshold the performer’s control over the instrument and the instrument’s unpredictability, so as to convey sonic complexity and expression.
PRINCIPLE OF VISUAL CONTINUITY : to facilitate the perceptual simplification of the projected moving image, by applying Gestaltist principles to the visual dynamics and dispensing with sudden visual changes, which would automatically attract attention and subordinate audition. There can a be a wealth of visual discontinuities at a level of detail.
PRINCIPLE OF AUDIO-VISUAL FUNGIBILITY : to create an audio-visual relationship that fosters a sense of causation, and at the same time, invites the audience to quit trying to understand the instrument, and focus on the perceptual experience itself. The fungible audio-visual relationship includes synchronised components and seemingly non-related components; it must generate complexity so as to confound the base cause and effect relationships.
Bellow is a short summary of the ideas that ground these three principles, and links to my related articles (see full list HERE) .
Perception, complexity and expression
Perceptual simplification dominates when we focus on a purpose such as discerning
a cause and a meaning, or accomplishing a task. Perception can prioritise
sensory information in different ways when we are not driven by any purpose - when we
focus on the experience itself. The brain can make use of assumptions to
simplify and clarify the perceptual field, and simultaneously, draw upon
their ambivalence. Dealing with non-anticipated sonic events can make a
performer acknowledge and respond to sensory details that she would otherwise
not be consciously aware of. The unexpected event can produce compelling
performative tension. It causes a minimal, yet graspable hesitation, which the
audience senses empathically as suspense; resolving the musical challenge then
causes a sensation of release.
Attention, intensity, continuities and discontinuities
Perception is a process of multi-sensorial
synthesis [Calvert et al. 2004], in which the visual discounts the aural and
the aural discounts the visual. In many aspects
vision dominates over audition, but attention can be manipulated so this does not occur [Sinnet et al. 2007]. I have drawn from perception science to argue that attention dynamics depend greatly on the interplay of continuities and discontinuities. To compare the strength of sound and image I proposed a particular notion of intensity: intensity is the neural imapact of any change in
stimuli causing an increase of neural activity. In this way, it derives from the combined effects of automatic
(exogenous) and deliberate (endogenous) attention. It depends on the event
itself, the stimuli panorama, and perceptual resolution.
Whether musical motion is sonic or audio-visual,
it is the interplay of continuities and discontinuities which fundamentally
directs attention. The intensity of each event depends greatly upon its relation
with other events, as well as upon the current state of the person's attention.
I created a taxonomy to distinguish between a sense of continuity that is
primarily driven from stimuli (exogenous), and a sense of continuity that
depends more on the individual (endogenous). It also distinguishes between
radical discontinuities, which impose disruption, and ambivalent
discontinuities, whose acknowledgement depends on perceptual resolution.
Taxonomy of Continuities and Discontinuities Related with Intensity and Attention
||binds all types of
continuities and discontinuities /
occurs at high hierarchical level in
perceptual organisation /
requires long-term memory, attention and conscious
|no intrinsic motion /
lowest intensity, dispenses with attention
display a similar interval of motion /
gestalt of good continuation
stimulus or sudden interruption of cohesive motion /
|continuous at low
perceptual resolution and discontinuous at high resolution /
on deliberate (endogenous) attention.
How the image can modulate and NOT obfuscate the sonic experience:
|To keep the music on the foreground, one must dispense with radical visual discontinuities, which automatically attract attention. But there can be a wealth of visual discontinuities at detail level, inviting for deliberate attention.
Gestaltist principles can be applied to create
a sense of overall visual continuity, so that attention is invited to focus on ambivalent discontinuities.
The same principles manifest in the visual and the aural domain:
Confounding causation in the audio-visual relationship:
Clearly perceivable cause-effect relationships are as problematic for the music as disruptive visual changes: the surreptitious nuances of the music become insignificant when we form definite conclusions. Perception is a process of multi-sensorial synthesis, and non-fitting sensory information is often distorted [Pick et al. 1969, McGurk and MacDonald 1976, Shams et al. 2002, Calvert et al. 2004, Schutz and Kubovy 2009]. We are driven to form conclusive concepts at the expense of overlooking or skewing any conflicting information. It happens even when we are aware of non-fitting information, as shown in Gestaltist psychology.
The question then is how an audio-visual relationship can foster a sense of causation, and simultaneously confound the cause and effect relationships so that they remain inconclusive. I call this a fungible audio-visual relationship : whilst synchrony produces a sense of causation, complexity and inconsistency confound the actual cause and effect relationships. With a fungible relationship, perception keeps acknowledging conflicting information, embracing convergences and divergences as inconclusive concepts.
Articles about a related study: