TAPESTREA: Techniques and Paradigms for Expressive Synthesis, Transformation, and Re-composition of Environmental Audio (thesis)

Abstract:

TAPESTREA is a sound design and composition framework that facilitates the creation of new sound from existing digital audio recordings, through interactive analysis, transformation and re-synthesis. During analysis, sinusoidal modeling and transient detection techniques are used to parametrically extract desired sound templates of different types:
sinusoidal events, transient events, and stochastic background. Each extracted template is transformed and synthesized independently using an appropriate technique, such as sinusoidal re-synthesis or wavelet tree learning. This allows specialized transformations on each template based on its type; sinusoidal templates undergo real-time, large-scale time and frequency transformations, while background is generated parametrically from extracted samples. The user interacts with TAPESTREA via a set of graphical interfaces. Synthesis is further controlled through ChucK scripts, which allow simultaneous, precise manipulation of many parameters. They also allow control via external input devices and user-defined GUI elements.

These combined techniques form a workbench for completely transforming a sound scene, dynamically generating soundscapes, or creating musical tapestries by weaving together transformed elements from different recordings. Thus, TAPESTREA introduces a new paradigm for composition, sound design, and other sonic sculpting tasks. Work on further improving the system includes user studies to compare alternative algorithms for generating stochastic background noise.