DESCRIPTION
Gathering audio-processing technologies (FMOD) and a 3D game engine (Torque), my custom <audiographics> software version has been developing for the input of a zither with aged strings and ambiguous tuning.
It escapes both functionality and esthetics usually associated with 3D-graphics - specifically in game design. Dealing with the tension between game-culture and transdisciplinary music, my changes transform mechanics and appearance: there is a musical articulation of sound, dynamic painting and performance.
The <windowmatter> project series includes the development of this multimodal instrumentation system.
It pursues my understanding of music as an entity that, far from pure abstraction, is ontologically connected to tangible factors beyond aural concerns – including cross-sensorial experience, media invention and the shifting transformation of criteria originated in other disciplinary fields.
Investigations address:
* Emergence
System complexity and external input wish to exceed predictable outcomes but maintain certain characteristics. What are the strategies to threshold unpredictability?
* Time-experience
How does the relationship between control over what is happening and acceptance-integration of the unexpected unfold in performance?
* Input-output correspondence
More than demonstrating mechanics, the artist aims at a poetic field of shifting experiences. What factors does the digital input-output bridging point at?
* Sound-image negotiation
How does the performative presence of image become musical?
<Audiographics> analyses the real-time sound-spectrum produced by analogue instrumentation. Accordingly, it triggers spatialized audio samples, which are also visualized as fading light-spots placed in a 3D-world; simultaneously, the analyzed sound-spectrum determines the virtual camera movements in that world.
The interface is extended with a gamepad allowing the cumulative choice of sample banks and the interruption of sample triggering.
My 3D-world design wishes to be perceived as a moving painting; the image is a dynamic picture plane view over an imaginary, deserted landscape. |
My software changes concern:
* Dynamic interaction – establishing parameters for the software’s reaction to the zither input; this is made in script (Typless C), since there is no graphical interface for changing the software mechanics.
* 3D world design and image – creating a wireframe representation of a three-dimensional world, images applying to its surface, images applying to a ‘skybox’, the 3Dworld is contained in a box), images for semi-transparent fluids and gases creating visual depth, intrinsic fluid and gas motions.
* Audio-to-audio mapping – establishing what detected frequencies from the zither’s acoustic output trigger what digital audio samples; rather than a transparent logic for easy apprehension, my strategies blur the perceptual distinction between cause and effect.
* Audio-to-visual mapping – establishing what detected frequencies trigger what visual events; my strategies negotiate a oscilation between perceptual distinction and ambiguity.
Digital configurations set thresholds of performative unpredictability. Performance experience extrapolates instrumental features, but these are an indicator of artistic concerns. <Windowmatter> investigations include:
* To minimize combining visual elements & maximize their dynamic variation possibilities, reducing processing effort & temporal density of attentional demands (cognitive load) – is Gestalt’s ‘Invariance’ property applicable in the relationship between dynamic painting and music?
* To merge all sensorial sources into a single experiential timeline, addressing dramatization, stimuli competition & provoked shift in attention
* To Set fields of unpredictability in software response:
- Input volume for amplitude-detection & sample triggering is merely intuited in live performance
- Multi-frequency detection upon a single string - discrete analysis becomes unadjusted to perception. Is this due to perceptual inhibition? Is the time needed to experience the present longer than the sampling interval? Is there a synthesis of two stimuli into a single perception?
* To maximize mapping-diversity with:
- Variable audio sample ‘attacks’ and lengths
- Variable and often dissonant tonal relations between input frequency and mapped sample
- Abstract-figurative range of samples
* To negotiate aural and visual criteria; since the sound is placed in the 3D world, visual depth implies decrease in audio volume. Parameter settings for 3D-camera dislocation seek to negotiate visual diversity and audio volume control. |
