A Perceptual Approach to Audio-Visual Instrument Design, Composition and Performance. Goldsmiths, University of London, 2016
INTER-FACE: International Conference on Live Interfaces 2014 (ICLI 2014), ed. Adriana Sá, Miguel Carvalhais, Alex McLean, pub. Porto University, CECL & CESEM (NOVA University), MITPL (University of Sussex), 2015. ISBN 978-989-746-060-9
Preface discussion “Live Interfaces: Seeds of Debate” by Adriana Sá, Joel Ryan, Edwin van der Heide, Atau Tanaka, Andrew McPherson, Thor Magnusson, Alex McLean, Miguel Carvalhais and Mick Gierson.
The Variables of Spatial Presence: a Parametric Model.
Adriana Sá & Atau Tanaka. In xCoAx 2019 Proceedings.
The term ‘spatial presence’ refers to the feeling of presence in a mediated space. This subjective experience has been discussed in media theory, sound art, film and performance. It depends on multiple variables, or parameters. This paper presents a parametric model that can be used to analyze those variables and their relationships. It exposes methods to assess interaction, characteristics of sound and image, audio-visual relationship and physical setup. It also exposes methods to assess how these variables intertwine in perceptual experience. The model draws from perception science, interaction design, music and audio-visual theory. It is applicable to the broad diversity of aesthetical options and technical platforms, facilitating the analysis of spatial presence in any performance. One can also discard part of the parameters so as to analyze installations, sound art and film.
Designing Musical Expression.
In xCoAx 2017 Proceedings, 2017.
The term New Interface for Musical Expression (NIME) has been applied to a great variety of instruments and systems, since the first NIME conference in 2001. But what is musical expression, and how does an interface intended for idiosyncratic expression differ from ubiquitous interfaces? This paper formulates an understanding where the reciprocal interaction between performer and instrument is important. Drawing from research in perception science , interface design and music, the paper specifies methods that can be used to analyse interaction, attention dynamics and semantics. The methods are applicable to any technical platform and aesthetic approach, facilitating the discussion of creative strategies and the analysis of music experience. The paper uses these methods to describe a NIME that combines an acoustic string instrument and software that operates based on the acoustic sound. The software applies the difference between the detected pitch and the closest tone / half tone to the processing of pre-recorded sounds. The proposed methods help to explain how this NIME enables versatile musical forms, and prevents undesired outcomes.
Between Acoustic and Digital Sound.
In Leonardo Transactions, vol. 48 No. 3 , MIT Press (2015): 280-281.
Mapping digital sound to an acoustic input enables the performer and the software to ‘talk’ simultaneously. Whilst the performer has direct control over the acoustic outcome, the digital can become a means of destabilization because it is mediated through code. Musical expression substantiates as the performer addresses the unexpected resourcefully. This text describes the performative dynamics in terms of perceptual mechanics.
The Fungible Audio-Visual Mapping and its Experience.
Adriana Sá, Baptiste Caramieux and Atau Tanaka. In Journal Of Science And Technology Of The Arts vol. 6 No. 1 (2014): 85-96.
This article draws a perceptual approach to audio-visual mapping. Clearly perceivable cause and effect relationships can be problematic if one desires the audience to experience the music. Indeed perception would bias those sonic qualities that fit previous concepts of causation, subordinating other sonic qualities, which may form the relations between the sounds themselves. The question is, how can an audio-visual mapping produce a sense of causation, and simultaneously confound the actual cause-effect relationships. We call this a fungible audio-visual mapping. Our aim here is to glean its constitution and aspect. We will report a study, which draws upon methods from experimental psychology to inform audio-visual instrument design and composition. The participants are shown several audio-visual mapping prototypes, after which we pose quantitative and qualitative questions regarding their sense of causation, and their sense of understanding the cause-effect relationships. The study shows that a fungible mapping requires both synchronized and seemingly non-related components – sufficient complexity to be confusing. As the specific cause-effect concepts remain inconclusive, the sense of causation embraces the whole.
Repurposing Video Game Software for Musical Expression: a perceptual approach.
In Proceedings of New Interfaces for Musical Expression 2014 (London, 2014): 331-334.
The text exposes a perceptual approach to instrument design and composition, and introduces an instrument that combines acoustic sound, digital sound, and digital image. We explore disparities between human perception and digital analysis as creative material. Because the instrument repurposes software intended to create video games, we establish a distinction between the notion of “flow” in music and gaming, questioning how it may substantiate in interaction design. Furthermore, we describe how the projected image creates a reactive stage scene without deviating attention from the music.
A Study About Confounding Causation in Audio-Visual Mapping.
Adriana Sá, Baptiste Caramieux, Atau Tanaka. In xCoAx 2014 Proceedings (Porto, 2014): 274-288
The text reports a study, which draws upon methods from experimental psychology to inform audio-visual instrument design. The study aims at gleaning how an audio-visual mapping can produce a sense of causation, and simultaneously confound the actual cause and effect relationships. We call this a fungible audio-visual mapping. The participants in this study are shown a few audio-visual mapping prototypes. We collect quantitative and qualitative data on their sense of causation and their sense of understanding the cause-effect relationships. The study shows that a fungible mapping requires both synchronized and seemingly non-related components - sufficient complexity to be confusing. The sense of causation embraces the whole when the specific cause-effect concepts are inconclusive.
How an Audio-Visual
Instrument Can Foster the Sonic Experience.
In Live Visuals, eds. L. Aceti,
S. Gibson, S. M. Arisona, O. Sahin, Leonardo Almanac vol. 19 No. 3, MIT Press (January 2013):
The text formulates an understanding of how an audio-visual instrument can be composed in such a way that the experience is driven through sound organization – modulated, but not obfuscated, by a moving image. This is particularly challenging, as normally the audio-visual relationship is skewed in favor of the visual. The investigation is motivated by insights derived from artistic practice. It outlines psychophysical boundaries with the aid of existing cognition/ attention research, and it describes three principles for the creation audio-visual instruments. As an example, the article describes how they are explored in a specific audio-visual instrument, combining an acoustic zither and modified software from audio processing and video-game technologies. This instrument addresses the three principles while exploring the disparities between an acoustic and a digital output.