Adriana Sa complete work


AUDIO-VISUAL RESEARCH 2011-2015

Research summary

 

Three creative principles

I have endeavoured to clarify how complex musical musical constructions can be fully experienced in audio-visual performance. This is challenging, because usually vision dominates over audition. My research draws from from artistic practice as well as literature in music, interaction design, audio-visual theory, psychology and neuroscience. The investigations lead to three creative principles for instrument design, composition and performance:

PRINCIPLE OF SONIC COMPLEXITY : to threshold the performer’s control over the instrument and the instrument’s unpredictability, so as to convey sonic complexity and expression.
See MORE

PRINCIPLE OF VISUAL CONTINUITY : to facilitate the perceptual simplification of the projected moving image, by applying Gestaltist principles to the visual dynamics and dispensing with sudden visual changes, which would automatically attract attention and subordinate audition. There can a be a wealth of visual discontinuities at a level of detail.
See MORE

PRINCIPLE OF AUDIO-VISUAL FUNGIBILITY : to create an audio-visual relationship that fosters a sense of causation, and at the same time, invites the audience to quit trying to understand the instrument, and focus on the perceptual experience itself. The fungible audio-visual relationship includes synchronised components and seemingly non-related components; it must generate complexity so as to confound the base cause and effect relationships.
See MORE

Bellow is a short summary of the ideas that ground these three principles, and links to my related articles (see full list HERE) .

Research outputs
Theoretical work:
Creative principles
Parametric model
Parametric representation
Practical work:
Practice summary
AG#1
Arpeggio-Detuning
AG#2

 

 

 

 

 

 

 

 

 

 
 

 

Perception, complexity and expression

Perceptual simplification dominates when we focus on a purpose such as discerning a cause and a meaning, or accomplishing a task. Perception can prioritise sensory information in different ways when we are not driven by any purpose - when we focus on the experience itself. The brain can make use of assumptions to simplify and clarify the perceptual field, and simultaneously, draw upon their ambivalence. Dealing with non-anticipated sonic events can make a performer acknowledge and respond to sensory details that she would otherwise not be consciously aware of. The unexpected event can produce compelling performative tension. It causes a minimal, yet graspable hesitation, which the audience senses empathically as suspense; resolving the musical challenge then causes a sensation of release.

Related articles:

 

perception, complexity and expression


Attention, intensity, continuities and discontinuities

Perception is a process of multi-sensorial synthesis [Calvert et al. 2004], in which the visual discounts the aural and the aural discounts the visual. In many aspects vision dominates over audition, but attention can be manipulated so this does not occur [Sinnet et al. 2007]. I have drawn from perception science to argue that attention dynamics depend greatly on the interplay of continuities and discontinuities. To compare the strength of sound and image I proposed a particular notion of intensity: intensity is the neural imapact of any change in stimuli causing an increase of neural activity. In this way, it derives from the combined effects of automatic (exogenous) and deliberate (endogenous) attention. It depends on the event itself, the stimuli panorama, and perceptual resolution.

Whether musical motion is sonic or audio-visual, it is the interplay of continuities and discontinuities which fundamentally directs attention. The intensity of each event depends greatly upon its relation with other events, as well as upon the current state of the person's attention. I created a taxonomy to distinguish between a sense of continuity that is primarily driven from stimuli (exogenous), and a sense of continuity that depends more on the individual (endogenous). It also distinguishes between radical discontinuities, which impose disruption, and ambivalent discontinuities, whose acknowledgement depends on perceptual resolution.

Related article:

 

Taxonomy of Continuities and Discontinuities Related with Intensity and Attention
Exogenous continuity Discontinuity Endogenous continuity
fulfils expectations counteracts expectations binds all types of continuities and discontinuities /
occurs at high hierarchical level in perceptual organisation /
requires long-term memory, attention and conscious awareness.
Steady continuity Progressive continuity Radical discontinuity Ambivalent discontinuity
no intrinsic motion /
lowest intensity, dispenses with attention		 successive events display a similar interval of motion /
gestalt of good continuation		 disruptive;
sudden stimulus or sudden interruption of cohesive motion /
highest intensity, automatic attention		 continuous at low perceptual resolution and discontinuous at high resolution /
intensity depends on deliberate (endogenous) attention.

 

 

How the image can modulate and NOT obfuscate the sonic experience:

To keep the music on the foreground, one must dispense with radical visual discontinuities, which automatically attract attention. But there can be a wealth of visual discontinuities at detail level, inviting for deliberate attention.


 

Gestaltist principles can be applied to create a sense of overall visual continuity, so that attention is invited to focus on ambivalent discontinuities. The same principles manifest in the visual and the aural domain:

gestalts.jpg

 

Confounding causation in the audio-visual relationship:

Clearly perceivable cause-effect relationships are as problematic for the music as disruptive visual changes: the surreptitious nuances of the music become insignificant when we form definite conclusions. Perception is a process of multi-sensorial synthesis, and non-fitting sensory information is often distorted [Pick et al. 1969, McGurk and MacDonald 1976, Shams et al. 2002, Calvert et al. 2004, Schutz and Kubovy 2009]. We are driven to form conclusive concepts at the expense of overlooking or skewing any conflicting information. It happens even when we are aware of non-fitting information, as shown in Gestaltist psychology.

The question then is how an audio-visual relationship can foster a sense of causation, and simultaneously confound the cause and effect relationships so that they remain inconclusive. I call this a fungible audio-visual relationship : whilst synchrony produces a sense of causation, complexity and inconsistency confound the actual cause and effect relationships. With a fungible relationship, perception keeps acknowledging conflicting information, embracing convergences and divergences as inconclusive concepts.

Articles about a related study: