WINDOWMATTER SERIES

WINDOWMATTER is an audiovisual performance series. Musical instruments are extended with AUDIOGRAPHICS, a software coded by John Klima that Adriana Sá has been modifying to attend her musical perspective. The performative arena comprises a physical space and a 3D world. The instrument playing drops audiovisual crumbs in the 3D world, while determining how the view over that world moves. In turn, the crumbs’ location and the picture plane movements are substantiated through the audio spatialization. The relation between sound and image is not strictly metaphorical, neither is it always transparent. The sound conducts the musical motion – the swing. The moving image potentiates immersion, while modulating the aural with an undulating view over an unchangeable landscape.

Music unfolds throughout floating tonal centers.
It extrapolates the aural with image and context.

full description

Adriana Sa solo

duo with John Klima

duo with Fala Mariam

orchestral version

screenshots & score & audio

funding and supports


DUO WITH JOHN KLIMA


Adriana's zither and John's Portuguese guitar or 'noise-makers' input two different versions of the AUDIOGRAPHICS software - each with its own interaction dynamics, audio architecture and audiovisual mapping. Both versions employ Adriana's world design, resambling an esthetics of painting more than of videogame graphics. Gamepads and foot-switches extend the setups.

The two software versions can be networked. In performance, two moving paintings are beamed side-to-side over performers and setup.

Performances:

SOLO VERSION

The zither inputs a sampler / delay modulator, which inputs AUDIOGRAPHICS.
Music is an immersive choreography of shifting sound textures
and densities driving, diving
us beyond the picture plane

    SYNOPSIS

Performances

  • April 15th '10, ZDB, Lisboa, Portugal

  • October 31st '09, Festival Y, Moagem, Fundão, Portugal

    SCORE


       


DUO WITH FALA MARIAM

Adriana Sa: zither with aged strings and ambiguous tuning + AUDIOGRAPHICS

Fala Mariam: trombone, dry

Full performance recording:
May '10, Teatro S. Luiz, Lisboa, Portugal

       

          ORCHESTRAL VERSION:


A number of musicians is invited to extend their instrumentation through AUDIOGRAPHICS, with custom code and configurations. A performance score is developed within pictorial installations: several screens are projected onto semi-transparent surfaces, creating a scenographic architecture made of light.

Performances      

 studies
toshio's instruments
     The in-door performance:

john klima and david maranha
manuel mota
adriana sa, fala mariam and and andre gonçalves
toshio kajiwara
0
toshio kajiwara and sei miguel
         The outdoor rehearsal:
john klima
adriana sa
margarida mendes
cesar buraga
sei miguel, john klima and toshio kajiwara
adriana sa and fala mariam
toshio kajiwara and margarida mendes
fala mariam, cesar buraga, manuel mota and david maranha
john klima, david maranha, toshi kajiwara and sei miguel
fala mariam and manuel mota
photos by O-blaat (that's why she's not in the pictures!...)

WINDOWMATTER project series

ORIGINS

In 2008 John Klima created code for frequency and amplitude analysis in 3D graphics. The original software version was ironically named Portuguese Guitar Hero, after the popular Guitar Hero videogame. John's first approach focused on game-play, but the possibility of further modifications was also embedded. Subsequently modified by Adriana Sa, the software became part of an instrument for transdisciplinary music performance. It evolved in accordance with the idiosyncrasies of a particular performer and her non-traditional modes of playing a specific zither. Changed were the audiovisual reactions to the audio input, the audio architecture, the 3D-world design and the parameter configurations. Adriana extends her zither with technology imported from the entertainment industries, but these are freed from their typical connotations.

Audiographics entails the challenge of bending videogame technologies, which embed specific assumptions, to art music, which addresses another sphere of concerns. Rather than focusing beyond the screen, the performer focuses on the musical instrument - the “interface” - and the surrounding audio room. Live-control must not depend on her vision. In fact, the moving image is projected over her and the setup; they stand in the window to the 3D world. Rather than confining to a commensurable intention - the goal of a game – the performer shapes the action while permeating the circumstances. She handles the situation with intentionality. Rather than aiming at transparency, the relation between her gesture, the zither sound and the outcome of its codified terms seek ambivalence and performative tension.

ARCHITECTING ART MUSIC IN AUDIOVISUAL TERMS

Musical motion operates through a continuous oscillation between tension and release, binding diverse levels of hierarchical organization, and conducting the perceptual navigation in-between detail and structure. Both sound and moving image are capable of such motion. However, emphasizing correspondences does not put them at equitable level.

Exploring art music with audiovisual means is a challenge: consciousness tends to focus on the visual, subordinating the aural – not only because most people are visually driven, but also because directing the eyes to the screen automatically improves the perceptual resolution of visual information. Since the ears are not directed, achieving equitable aural resolution requires modulating the sensitivity of corresponding neural circuits. Furthermore, people may find it difficult to discuss the impact of image over sound and of sound over image, because attending to one or the other sense has an influence on sensory discrimination.

Adriana has been exploring how experience can be driven by sound organization, while modulated, but not obfuscated, by moving image. In the WINDOWMATTER series, attention works similarly to how it does in a musical drone: complex changeability sprouts in bold continuity. A drone drives attention to focus the subtle variations in the continuous mass of sound. The current audiovisual architecture drives it to focus the aural changeability through the visual continuity.

TECHNICAL DESCRIPTION

Adriana's version of AUDIOGRAPHICS has been developing for the input of a zither with aged strings and ambiguous tuning.

AUDIOGRAPHICS detects audio frequencies from the zither, triggering digital audio samples that intertwine with its amplified sound. The audio samples are organized in banks mapped to gamepad buttons, and there is a button mapped to sample muting. The performer has full control over which sample banks are active. Nevertheless, she does not fully predict which frequencies will be detected upon her playing, i.e. which audio samples will be triggered. A single zither sound may inclusively trigger more than one. Moreover, since the software reacts above certain sound amplitude, she may shift between triggering samples (while playing mezzo forte or forte) and not triggering samples (while playing piano) without touching the gamepad.

Dropped in a 3D-world as audiovisual ‘crumbs’, the digital audio samples are simultaneously visualized as lights. Crumbs can accumulate, forming temporary visual paths. Their visualization occurs whether the corresponding audio is muted or not. Furthermore, the audio input is mapped to virtual camera movements. These produce a dynamic view over the 3D world - an undulating deserted landscape, whose appearance recalls painting more than videogame graphics. The audio samples’ spatialization derives from the relation between this changing camera view and the crumbs’ position in the 3D world.

The disparities between the analogue and the digital are architected to potentiate complexity. On one hand, the resilience of the performer’s physical gesture exceeds its codified terms; on the other, the sensitivity of the instrument exceeds human perception. The field of possible relations between sound and image is set previous to performance, through audio-to-audio and audio-to-visual mappings. The fact that the outcome is determined but not fully determinable attends the performer’s musical manner - violated expectations require her to respond resourcefully. Moreover, the mechanical relations between input and output also become less transparent to the audience, challenging the tendency of constraining perception to cognitive purposes. Thresholding unpredictability in instrument configurations is a core task.

AUDIOGRAPHICS software in action

Adriana Sá, solo performance:

    SCORE

AUDIO

SCREENSHOTS

  

 

 

FUNDING AND SUPPORTS:


Direcção Geral das Artes / Cultural Ministry Portugal
Quarta Parede
Festival Numero
MusicBox