Artificial Saraswati




Interdisciplinary Ph.D. Research Project Proposal

University of Victoria




For Degree: Intelligent Music and Media Technology


Departments: Computer Science, Mechanical Engineering

Electrical and Computer Engineering, Music  & Psychology








In the Hindu religion, Saraswati is known as the Goddess of music and knowledge. In this proposal I will describe a Ph.D. project proposal to create a knowledgeable system for artificial intelligent music performance. The goal of Artificial Saraswati, is to have a musical robot, perform on stage, reacting with a human musician. This project will draw on knowledge from many disciplines: music, computer science, electrical engineering, philosophy, neurological psychology and genetic algorithmic studies. In this proposal I will describe the key areas of research and development which must be accomplished to make this project a success.  Below is a list of research and development that will be discussed:


  • Artificial Saraswati  Software
  • New Instruments for Musical Expression
  • Media Software Design



Artificial Saraswati hardware:

Artificial Saraswati is the name of the musical robot which will be constructed during this project. She will be a stringed instrument which can acoustically create sound using motors and robotic technology. 


Proposed Research\Experiments:

  • Real time Systems
  • Digitial Signal Processing
  • Robotics
  • LEMUR Guitarbot[1]


Design Considerations:

  • System will have plectrums attached to a motor, which spins according to rhythmic impulses.
  • System will have one string which to create acoustic melodies. A moveable motor device will roll on the string, being able to change the pitch of the device.
  • System will have a motor to keep string in tune. It will have a pitch detection algorithm which keeps checking the tuning for the string and adjusting only when it goes out of tune.
  • System must be able to receive music data protocol (MIDI or OSC[2]) and turn motors to create sound in real time, with little or no delay. 



Artificial Saraswati software:


This software will act like a musician brain as it reacts to what it "hears" (transmitted sensor data) a human performing with digital instruments such as the Electronic Sitar (ESitar), Electronic Tabla (ETabla), and Radio Drum, and sends control signals to the robot to create acoustic sound.


Proposed Research\Experiments:

  • Philosophy of human logic in musical performance
  • Psychology of human brain and behavior during musical performance
  • Genetic algorithms
  • Neural Networks
  • Artificial Intelligence
  • Pattern Recognition
  • Algorithmic Composition [3]
  • Intelligent Music [4] [5]
  • Neural Network Digital Signal Processing
  • Indian Classical Raga (melodic) and Theka (rhythmic) systems


System Design Considerations:

  • System must have intelligence of rhythm. Will have database of different types of Indian rhythmic cycles known as thekas. Data structure for rhythm must be able to accommodate for varying number of beats per cycle, as well as 64 divisions of time per beat.
  • System must have intelligence of pitch. Will have database of different types of Indian ragas with different rules for performance. Data structure for pitch must be able to accommodate pitch bending. (notes must be able to pass through every frequency between 2 desired pitches).
  • System should have intelligence to listen. It "hears" by capturing controller data being sent from the ESitar, ETabla, Radio Drum, and other controllers. Must be able to do beat detection to find the tempo of the human performer. Must be able to derive pitch of human performer.
  • System must be able to compose and interact in real time.





New Instruments for Musical Expression:[6]

New instruments for musical expression will capture human gestural information enabling a performing musician to create a new genre of music and media with the aid of software, as well as enable Artificial Saraswati to capture data about the human musician's playing in real time.


Proposed research to create new controllers:


  • Research sensors for data capturing: piezo, FSRs, accelerometers, sliders, knobs, TacTex Controllers, Inc. touch pad, etc.
  • Research microprocessors to translate sensor data for software: Parallel Basic Stamp or the Atmel Chip[7].
  • Research Protocol to communicate with software. Open Sound Control  or MIDI.
  • Research Wireless technology to enable instrument to have freedom from wires: Bluetooth
  • Research the art of traditional Indian Classical Sitar performance and instrument making.
  • Research the art of traditional Indian Classical Tabla performance and instrument making.


The Electronic Sitar

One controller will be an Electronic Sitar (ESitar), which will enhance the performance of a real sitar (a traditional 19-string instrument of India) with electronic sensors on the body.


The Electronic Tabla

Another controller which will be designed is the Electronic Tabla (ETabla). The Tabla is a traditional set of drums of Northern India. The digitized ETabla is a controller which triggers both sound and graphics simultaneously. It allows for a variety of traditional Tabla strokes and new performance techniques. Graphical feedback allows for artistical display and pedagogical feedback.


As of now, the ETabla uses force sensing resistors to trigger sounds and graphics. The new goal would be to upgrade the ETabla to use TacTex Controls, Inc., pressure-sensitive touch pad which obtain sensor data in the x, y, and z axis, to get a better gestural response. [8]


For more information on the ETabla visit:


The Radio Drum [9]

I will also help in the development of the Radio Drum developed by Professor Andy Schloss and others. The Radio Drum consists of two parts: a rectangular surface ("drum") with embedded antennae, and two transmitters embedded in conventional sticks ("mallets"). 


Proposed research and development:

·        Help redesign the Radio Drum to obtain the response characteristics of a fine acoustic instrument, by sampling and processing the analog gesture signals.

·        Help add wireless sticks, which broaden its possible use in popular music and other applications, including as a conducting device for karaoke. [10]





Media Software Design:

Real Time software must be designed to take control signals from the ESitar, ETabla, and Radio Drum as well as from Artificial Saraswati to produce a multimedia based entertainment experience. Control signals will trigger both sound and performance based visual graphics.


Sound Control:


  • Become expert using MAX/MSP[11] and Pd ("Pure Data")[12]
  • Become expert at ChucK and Marsyas
  • Physical Modeling techniques: STK Toolkit[13]
  • New methods of controlling (gestural inputs) sounds based on physical models.[14]


Artificial Saraswati ChucK or Marsyas:

A ChucK or Marsyas will be written to control sound processing of the performance of the ESitar, ETabla, and Radio Baton. Features should include:


  • Selection of what Controller is being used.
  • Map\graphical interface showing all data captured by sensors for a particular Controller.
  • Ability to trigger/control rhythmic beats with different sounds (Tabla beats, electronic techno beats, western beats, etc.)
  • Ability to trigger/control melodies, modes with different tambres (Sitar, Sarood, flute, guitar, etc.)
  • Ability to change DSP effects of real sound Sitar is producing (comb filters, flangers, Equilization, Panning, Volume, etc.)
  • Ability to play a drone modeling a Tanpura (A four string Indian string instrument).
  • Ability to physically model sounds in real time that simulate Indian instruments such as sitar and Tabla.[15]


Visual Control:


  • Research existing software packages: JIT, Onadime, Metasynth, NATO
  • Research controlling video sequencing using parameters of H.263 or MPEG compressed video[16]
  • Collaborate with Philip Davidson to use veldt (real time software for visual performance)












2003 July – December:                           Build Electronic Sitar Controller


2004 January - April:                              Audio Feature Extraction   

                                                                  Introduction to Music Information Retrieval

                                                                  Digital Audio Effects

                                                                  VICON Gestural Capturing at  Queen Alexander Childrens Hospital

                                                                  Digital Signal Processing


2004 May – August:                              Affective Computing  


2004 September – December:               Introduction to Machine Learning

                                                                  Acoustics of Musical Instruments

                                                                  Introduction to Marsyas


2005 January - April:                              Music Information Retrieval  

                                                                  Begin building Intelligent Music Software - Intellitrance


2005 May – August:                               Background Research on Musical Robotics

                                                                  Begin Working on Bayan Robot

                                                                  Introduction to Advanced Recording Techniques

                                                                  Build Wearable Sensors - KIOM


2005 September – December:               Computer Music Seminar

                                                                  Brain Signal Processing using MEG


2006 January – April:                             Computer Music Seminar

                                                                  Complete Bayan Robot

                                                                  Begin Redesign of ESitar


2006 May - December:                           IntelliTrance for Robotic Improvisation Composition


2006 – 2008:                                             Research/Complete Dissertation

[1]  Singer, E., Larke, K., Bianciardi, D., “LEMUR GuitarBot: MIDI Robotic String Instrument”, In Proceedings of New Interfaces for Musical Expression (NIME 03). Montreal, Canada, 2003.

[2] Wright, M. and A. Freed. “Open SoundControl: A New Protocol for Communicating with Sound Synthesizers,”  Proceeding of the International Computer Music Conference. 1997.

[3]  Cope, D. The Algorithmic Composer (Computer Music and Digital Audio Series, V. 16), A-R Editions; Book and CD-ROM edition (June 2000).

[4] Cope, D., Experiments in Musical Intelligence (The Computer Music and Digital Audio, Vol 12), A-R Editions; Book and CD-ROM edition (July 1996).

[5] Rowe, R., Machine Musicianship, MIT Press; (March 5, 2001).

[6] Cook, P. R. "Principles for Designing Computer Music Controllers," ACM CHI Workshop in New Interfaces for Muiscal Expression (NIME), Seattle, April 2001.

[7] Wilson, S.,  M. Gurevich, B. Verplank, and P. Stang  “Microcontrollers in Music HCI Instruction - Reflections on our Switch to the Atmel AVR Platform,” Proceedings of the International Conference on New Instruments for Musical Expression (NIME), Montreal, Canada, May 2003.

[8] Kapur, A., G. Essl, P. Davidson, and P. R. Cook. "The Electronic Tabla Controller," Proceedings of the International Conference on New Instruments for Musical Expression (NIME), Dublin, Ireland, May 2002.

[9] Mathews, M and W.A. Schloss   The Radio Drum  as a  Synthesizer Controller.   ICMC Ohio State proceedings, 1989.

[10] Driessen, P.  and A. Scholoss. Grant Proposal to NSERC.

[11] Puckette, M. and Apel, T. 1998. "Real-time audio analysis tools for Pd and MSP". Proceedings, International Computer Music Conference. San Francisco: International Computer Music Association, pp. 109-112.

[12] Puckette, M. 1996. "Pure Data: another integrated computer music environment." Proceedings, Second Intercollege Computer Music Concerts, Tachikawa, Japan, pp. 37-41.

[13] Cook, P. R., and G. Scavone. "The Synthesis ToolKit (STK)," International Computer Music Conference, Beijing, October, 1999.

[14] P.F. Driessen, A. Schloss, "New algorithms and technology for analyzing gestural data", IEEE Pacific Rim Conf, Victoria, August 2001.

[15] Essl, Georg. “Physical Wave Propagation Modeling for Real-Time Synthesis of Natural Sounds,” Ph. D. Thesis, Princeton University, 2002.

[16] A. Dumitras, B. Haskell, “An encoder-only texture replacement method or effective compression of entertainment movie sequences”, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP02), Orlando, FL, May 13-17, 2002