A Perceptual Approach to Audio-Visual Instrument Design, Composition and Performance. Goldsmiths, University of London, 2016
INTER-FACE: International Conference on Live Interfaces 2014 (ICLI 2014), ed. Adriana Sá, Miguel Carvalhais, Alex McLean, pub. Porto University, CECL & CESEM (NOVA University), MITPL (University of Sussex), 2015. ISBN 978-989-746-060-9
Adriana Sá, Joel Ryan, Edwin van der Heide, Atau Tanaka, Andrew McPherson, Thor Magnusson, Alex McLean, Miguel Carvalhais, Mick Gierson “Live Interfaces: Seeds of Debate”. Preface discussion, in IINTER-FACE: International Conference on Live Interfaces 2014 (ICLI 2014) .
Designing Musical Expression. In XCoAX 2017 Proceedings
The term New Interface for Musical Expression (NIME) has been applied to a great variety of instruments and systems, since the first NIME conference in 2001. But what is musical expression, and how does an interface intended for idiosyncratic expression differ from ubiquitous interfaces? This paper formulates an understanding where the reciprocal interaction between performer and instrument is important. Drawing from research in perception science , interface design and music, the paper specifies methods that can be used to analyse interaction, attention dynamics and semantics. The methods are applicable to any technical platform and aesthetic approach, facilitating the discussion of creative strategies and the analysis of music experience. The paper uses these methods to describe a NIME that combines an acoustic string instrument and software that operates based on the acoustic sound. The software applies the difference between the detected pitch and the closest tone / half tone to the processing of pre-recorded sounds. The proposed methods help to explain how this NIME enables versatile musical forms, and prevents undesired outcomes.
The Fungible Audio-Visual Mapping and its Experience, Adriana Sá, Baptiste Caramieux and Atau Tanaka. In Journal Of Science And Technology Of The Arts vol. 6 No. 1 (2014): 85-96.
This article draws a perceptual approach to audio-visual mapping. Clearly perceivable cause and effect relationships can be problematic if one desires the audience to experience the music. Indeed perception would bias those sonic qualities that fit previous concepts of causation, subordinating other sonic qualities, which may form the relations between the sounds themselves. The question is, how can an audio-visual mapping produce a sense of causation, and simultaneously confound the actual cause-effect relationships. We call this a fungible audio-visual mapping. Our aim here is to glean its constitution and aspect. We will report a study, which draws upon methods from experimental psychology to inform audio-visual instrument design and composition. The participants are shown several audio-visual mapping prototypes, after which we pose quantitative and qualitative questions regarding their sense of causation, and their sense of understanding the cause-effect relationships. The study shows that a fungible mapping requires both synchronized and seemingly non-related components – sufficient complexity to be confusing. As the specific cause-effect concepts remain inconclusive, the sense of causation embraces the whole.
Repurposing Video Game Software for Musical Expression: a perceptual approach. In Proceedings of New Interfaces for Musical Expression 2014 (London, 2014): 331-334.
The text exposes a perceptual approach to instrument design and composition, and introduces an instrument that combines acoustic sound, digital sound, and digital image. We explore disparities between human perception and digital analysis as creative material. Because the instrument repurposes software intended to create video games, we establish a distinction between the notion of “flow” in music and gaming, questioning how it may substantiate in interaction design. Furthermore, we describe how the projected image creates a reactive stage scene without deviating attention from the music.
A Study About Confounding Causation in Audio-Visual Mapping, Adriana Sá, Baptiste Caramieux, Atau Tanaka. In XCoAX 2014 Proceedings (Porto, 2014): 274-288
The text reports a study, which draws upon methods from experimental psychology to inform audio-visual instrument design. The study aims at gleaning how an audio-visual mapping can produce a sense of causation, and simultaneously confound the actual cause and effect relationships. We call this a fungible audio-visual mapping. The participants in this study are shown a few audio-visual mapping prototypes. We collect quantitative and qualitative data on their sense of causation and their sense of understanding the cause-effect relationships. The study shows that a fungible mapping requires both synchronized and seemingly non-related components - sufficient complexity to be confusing. The sense of causation embraces the whole when the specific cause-effect concepts are inconclusive.
Between Acoustic and Digital Sound. In Leonardo Transactions, vol. 48 No. 3 , MIT Press (2015): 280-281.
Mapping digital sound to an acoustic input enables the performer and the software to ‘talk’ simultaneously. Whilst the performer has direct control over the acoustic outcome, the digital can become a means of destabilization because it is mediated through code. Musical expression substantiates as the performer addresses the unexpected resourcefully. This text describes the performative dynamics in terms of perceptual mechanics.
How an Audio-Visual
Instrument Can Foster the Sonic Experience. In Live Visuals, eds. L. Aceti,
S. Gibson, S. M. Arisona, O. Sahin, Leonardo Almanac vol. 19 No. 3, MIT Press (January 2013):
The text formulates an understanding of how an audio-visual instrument can be composed in such a way that the experience is driven through sound organization – modulated, but not obfuscated, by a moving image. This is particularly challenging, as normally the audio-visual relationship is skewed in favor of the visual. The investigation is motivated by insights derived from artistic practice. It outlines psychophysical boundaries with the aid of existing cognition/ attention research, and it describes three principles for the creation audio-visual instruments. As an example, the article describes how they are explored in a specific audio-visual instrument, combining an acoustic zither and modified software from audio processing and video-game technologies. This instrument addresses the three principles while exploring the disparities between an acoustic and a digital output.
the Audio-Visual to Modulate Rather Than Obfuscate The Sonic Experience. In International Conference on Live Interfaces 2012 (2012).
Conference organisation and related activities
Chair and producer of INTER-FACE, the 2nd ICLI - International conferencfe for Live Interfaces, Lisbon, November 19-23, 2014 - see complete program HERE
Steering committee member of ICLI - International conference on Live Interfaces 2018, University of Porto
Steering committee member of ICLI - International conference on Live Interfaces 2016, University of Sussex
Scientific committee member of XCoAX 2017, Lisbon, Portugal
Scientific committee member of Invisible Places 2017 - Sound, Urbanism and Sense of Place, Açores, Portugal
Submission reviewer for Leonardo Journal, MIT Press, 2017
Scientific committee member of ICMC-International Computer Music Conference 2016 , University of the Arts Utrecht, Netherlands
Scientific committee member of ICMC-International Computer Music Conference 2015, University of North Texas
Scientific committee member of ICLC-International Conference on Live Coding 2015, University of Leeds, UK
Scientific committee member of ICLC-International Conference on Live Coding 2016, University in Hamilton, Canada
Scientific committee member of Invisible Places 2014, Viseu, Portugal
Co-organizer (with Adam Parkinson and Atau Tanaka) of the symposium Real-time Visuals Workshop 2, Goldsmiths College, London 2013
Chair of Master Students Seminar (Term 3), Creative Computing, Goldsmtihs, University of London, June 5th 2014
Reviewer for CHI 2013, Conference on Human Factors in Computing Systems (submissions for installations)
Adriana Sa, “Designing Musical Expression ", at XCoAX 2017 , Lisbon, Portugal (July 6, 2017)
"Music, Interaction Design and Effort" , at Sonic Writing symposium, STEIM, 2016
“A Study About Confounding Causation in Audio-Visual Mapping", at XCoAX 2014 , Porto, Portugal (June 27, 2014).
“A Perceptual Approach to Instrument Design and Composition”, at Live Visuals for
Performance, Gaming, Installation, and Electronic Environments / Closing
Cinema, Newcastle, UK ( January 24th, 2014).
Adriana Sa and John Klima, “Diverging Perspectives
on Music with Video Game Technologies”, at Live Visuals for Performance,
Gaming, Installation, and Electronic Environments / Workshop I, Northumbria
University, UK (February 26th, 2013).
“Architecting the Audio-Visual to Modulate Rather Than Obfuscate The Sonic
Experience”, at International Conference on Live
Interfaces (ICLI), Interdisciplinary Centre for Scientific Research in Music
(ICSRiM), University of Leeds, UK (September 7th, 2012).
“Perception, Attention and Intensity in Performance”, Ciclo de
Conversas, Engineering Faculty, University of Coimbra, Portugal (April 10th,