Artificial
Saraswati
Interdisciplinary Ph.D. Research
Project Proposal
For Degree: Intelligent Music and Media
Technology
Departments: Computer Science, Mechanical
Engineering
Electrical and Computer Engineering,
Music & Psychology
by,
Introduction/Overview:
In
the Hindu religion, Saraswati is known as the Goddess of music and knowledge.
In this proposal I will describe a Ph.D. project proposal to create a knowledgeable
system for artificial intelligent music performance. The goal of Artificial
Saraswati, is to have a musical robot, perform on stage, reacting with a
human musician. This project will draw on knowledge from many disciplines:
music, computer science, electrical engineering, philosophy, neurological
psychology and genetic algorithmic studies. In this proposal I will describe
the key areas of research and development which must be accomplished to make
this project a success. Below is a list
of research and development that will be discussed:
- Artificial
Saraswati Software
- New
Instruments for Musical Expression
- Media
Software Design
Artificial
Saraswati hardware:
Artificial
Saraswati is the name
of the musical robot which will be constructed during this project. She will be
a stringed instrument which can acoustically create sound using motors and
robotic technology.
Proposed
Research\Experiments:
- Real time Systems
- Digitial Signal Processing
- Robotics
- LEMUR Guitarbot[1]
Design
Considerations:
- System will have plectrums
attached to a motor, which spins according to rhythmic impulses.
- System will have one string which
to create acoustic melodies. A moveable motor device will roll on the
string, being able to change the pitch of the device.
- System will have a motor to keep
string in tune. It will have a pitch detection algorithm which keeps
checking the tuning for the string and adjusting only when it goes out of
tune.
- System must be able to receive
music data protocol (
Artificial Saraswati software:
This
software will act like a musician brain as it reacts to what it
"hears" (transmitted sensor data) a human performing with digital
instruments such as the Electronic Sitar (ESitar), Electronic Tabla
(ETabla), and Radio Drum, and sends control signals to the robot
to create acoustic sound.
Proposed
Research\Experiments:
- Philosophy
of human logic in musical performance
- Psychology
of human brain and behavior during musical performance
- Genetic
algorithms
- Neural
Networks
- Artificial
Intelligence
- Pattern
Recognition
- Algorithmic
Composition [3]
- Intelligent
Music [4]
[5]
- Neural
Network Digital Signal Processing
- Indian
Classical Raga (melodic) and Theka (rhythmic) systems
System
Design Considerations:
- System
must have intelligence of rhythm. Will have database of different types of
Indian rhythmic cycles known as thekas. Data structure for rhythm
must be able to accommodate for varying number of beats per cycle, as well
as 64 divisions of time per beat.
- System
must have intelligence of pitch. Will have database of different types of
Indian ragas with different rules for performance. Data structure
for pitch must be able to accommodate pitch bending. (notes must be able
to pass through every frequency between 2 desired pitches).
- System
should have intelligence to listen. It "hears" by capturing
controller data being sent from the ESitar, ETabla, Radio Drum, and
other controllers. Must be able to do beat detection to find the
tempo of the human performer. Must be able to derive pitch of human
performer.
- System must be able to compose
and interact in real time.
New
Instruments for Musical Expression:[6]
New
instruments for musical expression will capture human gestural information
enabling a performing musician to create a new genre of music and media with
the aid of software, as well as enable Artificial Saraswati to capture
data about the human musician's playing in real time.
Proposed
research to create new controllers:
- Research
sensors for data capturing: piezo, FSRs, accelerometers, sliders, knobs,
TacTex Controllers, Inc. touch pad, etc.
- Research
microprocessors to translate sensor data for software: Parallel Basic
Stamp or the Atmel Chip[7].
- Research
Protocol to communicate with software. Open Sound Control or
- Research
Wireless technology to enable instrument to have freedom from wires:
Bluetooth
- Research
the art of traditional Indian Classical Sitar performance and instrument
making.
- Research
the art of traditional Indian Classical Tabla performance and instrument
making.
The Electronic Sitar
One
controller will be an Electronic Sitar (ESitar), which will
enhance the performance of a real sitar (a traditional 19-string instrument of
The Electronic Tabla
Another
controller which will be designed is the Electronic Tabla (ETabla). The
Tabla is a traditional set of drums of
As
of now, the ETabla uses force sensing resistors to trigger sounds and
graphics. The new goal would be to upgrade the ETabla to use TacTex
Controls, Inc., pressure-sensitive touch pad which obtain sensor data in the x,
y, and z axis, to get a better gestural response. [8]
For
more information on the ETabla visit:
http://www.cs.princeton.edu/sound/research/controllers/etabla/
The Radio Drum [9]
I
will also help in the development of the Radio Drum developed by
Professor Andy Schloss and others. The Radio Drum consists of two parts:
a rectangular surface ("drum") with embedded antennae, and two
transmitters embedded in conventional sticks ("mallets").
Proposed
research and development:
·
Help
redesign the Radio Drum to obtain the response characteristics of a fine
acoustic instrument, by sampling and processing the analog gesture signals.
·
Help
add wireless sticks, which broaden its possible use in popular music and other
applications, including as a conducting device for karaoke. [10]
Media Software Design:
Real
Time software must be designed to take control signals from the ESitar,
ETabla, and Radio Drum as well as from Artificial Saraswati to
produce a multimedia based entertainment experience. Control signals will
trigger both sound and performance based visual graphics.
Sound Control:
Research/Development:
- Become
expert using MAX/MSP[11]
and Pd ("Pure Data")[12]
- Become
expert at ChucK and Marsyas
- Physical
Modeling techniques: STK Toolkit[13]
- New
methods of controlling
(gestural inputs) sounds based on physical models.[14]
Artificial Saraswati ChucK or Marsyas:
A
ChucK or Marsyas will be written to control sound processing of the performance
of the ESitar, ETabla, and Radio Baton. Features should include:
- Selection
of what Controller is being used.
- Map\graphical
interface showing all data captured by sensors for a particular
Controller.
- Ability
to trigger/control rhythmic beats with different sounds (Tabla beats,
electronic techno beats, western beats, etc.)
- Ability
to trigger/control melodies, modes with different tambres (Sitar, Sarood,
flute, guitar, etc.)
- Ability
to change DSP effects of real sound Sitar is producing (comb filters,
flangers, Equilization, Panning, Volume, etc.)
- Ability
to play a drone modeling a Tanpura (A four string Indian string
instrument).
- Ability
to physically model sounds in real time that simulate Indian instruments
such as sitar and Tabla.[15]
Visual
Control:
Research/Development:
- Research existing software
packages: JIT, Onadime, Metasynth, NATO
- Research controlling video sequencing
using parameters of H.263 or MPEG compressed video[16]
- Collaborate with Philip Davidson
to use veldt (real time software for visual performance)
PROJECT PLAN:
2003 July –
December: Build
Electronic Sitar Controller
2004 January -
April: Audio Feature
Extraction
Introduction
to Music Information Retrieval
Digital
Audio Effects
VICON
Gestural Capturing at
Digital
Signal Processing
2004 May – August: Affective Computing
2004 September –
December: Introduction to
Machine Learning
Acoustics
of Musical Instruments
Introduction
to Marsyas
2005 January -
April: Music
Information Retrieval
Begin
building Intelligent Music Software - Intellitrance
2005 May – August: Background
Research on Musical Robotics
Begin
Working on Bayan Robot
Introduction
to Advanced Recording Techniques
Build Wearable Sensors
- KIOM
2005 September – December: Computer
Music Seminar
Brain
Signal Processing using MEG
2006 January – April: Computer Music
Seminar
Complete
Bayan Robot
Begin
Redesign of ESitar
2006 May - December: IntelliTrance for
Robotic Improvisation Composition
2006 – 2008: Research/Complete
Dissertation