General Information
	Still Image Viewer Generating High-Resolution Images Making Slides using PowerPoint Showing Single Images Showing Slides
	Multi-channel Sound Server Introduction The NDS sound file format The NDSoundServerStub client interface The NDSoundServer The Sound Server FAQ
	OpenGL Server Running an OpenGL program Compiling a program An OpenGL example program
	User Interface Devices Wand Gyro Mouse Hand
	Virtual Display Driver
	WallThumbs (thumbnail image generator)
	Contacts

Multi-channel audio reproduction beyond stereo is an area of active research. We wrote a multi-channel sound server software which allows to control the playback of multi-channel sounds on large scale multi-channel systems over a simple network interface.

Introduction

The NDSoundServer is TCP/IP server-software running under Windows95 that provides flexible control over multi-speaker arrays.

---

---

In the current implementation we use a SoundBlaster-compatible sound card and two 8-channel Darla boards by Echo yielding a total of 18 mono-channels. The two SoundBlaster-channels are typically used to drive two sub-woofers, because of the lack of synchronization between the SoundBlaster- and Darla-drivers. The Darla-drivers synchronize all open channels if wanted.

The NDS sound file format

The NDS sound file used by the server follows the format of the standard Sun/NeXT snd file format, except, that all the data is stored using little endian instead of big endian. This spares the server from byte-swapping buffers for streaming sound files. Sun/NeXT snd-files can be converted to nds-files using the snd2nds-tool. There is a MS-Dos based version available on the server machine and the Unix source is available on request.

The NDSoundServerStub client interface

The purpose of the NDSoundServerStub-class is to provide an easy to use RPC-like client interface to the sound server which spares sound server users from implementing the communication protocol. These are the methods provided by the class:

 * NDSoundServer()
 *     Default constructor. Opens the connection to the default NDSoundServer(). Later
 *     versions of this class will provide a constructor which allows to specify the
 *     hostname and port number.
 *     [Possibly this should be changed to having a separate method to initialize connection
 *     to allow returning errors and the like. For now NDSoundServer() will exit on errors.]
 *     The current default server is "malleus.cs.princeton.edu".
 *
 * ~NDSoundServer()
 *     Default destructor. Closes the connection to the default NDSoundServer if open.
 *
 * int connectServer()
 *     Opens the connection to the default NDSoundServer. If a connection is already open
 *     it should be closed first with closeServer().
 *
 * int closeServer()
 *     Closes the connection to the default NDSoundServer if open.
 *
 * int load(char* lname, int& tag, int& srcs)     
 *     Loads a file from the sound servers file-directory into the pool of activated
 *     sound sources. Active sound sources can be addressed using the "tag" which is
 *     return by this function (similar to a file-descriptor). "srcs" is the number
 *     of channels which are provided by the opened sound source.
 *        
 * int free(int tag)
 *     Frees a previously opened active source "tag". NOTE: Every load()'d "tag" has to
 *     be free()'d by its opener!
 *
 * int map(int tag, int src, int speaker)
 *     Maps (i.e. virtually connects) a mono sound source stream "src" from the active
 *     source "tag" to the speaker with number "speaker". "src" must be less than the
 *     number of sources returned by load().
 *     
 * int play(int tag)
 *     Starts the playback of all mono sound sources provided by the active source "tag".
 *     If paused it will resume play. Newly loaded sources are paused by default.
 *     If already playing, it is ignored. If it stopped at the end of a source stream
 *     it will resume playing at the beginning.
 *
 * int pause(int tag)
 *     Pauses the playback of all mono sound sources provided by the active source "tag".
 *     If already paused, nothing happens. Newly loaded sources are paused by default.
 *
 * int reset(int tag)
 *     Resets the position of the playback of all mono sound sources provided by the
 *     active source "tag" to the beginning if possible for the given source.
 *
 * int loop(int tag, int flag)
 *     All mono sound sources of the active source "tag" will be looped if "flag" is non-zero.
 *     Otherwise it won't be looped. Some sources may not support this feature. The default
 *     is no looping.
 *
 * int atEnd(int tag, int& flag)
 *     Sets "flag" to non-zero value if playback reached end. Otherwise it sets "flag" to zero.
 *     Please keep in mind, that repeatingly calling this function might put heavy load in the
 *     server.
 *
 * int gain_speaker(int speaker, float gain)
 *     Sets the gain of a speaker to "gain". "gain" should be between 2.0 and 0.0.
 *     (very loud to muted). 1.0 is the default gain for all speakers.
 *
 * int gain_channel(int tag, float gain)
 *     Sets the gain of the active source "tag" to "gain". "gain" should be between 2.0
 *     and 0.0. (very loud to muted). 1.0 is the default gain for all channels.
 *
 * int gain_all_speakers(float gain)
 *     Sets the gain of all speakers to "gain". "gain" should be between 2.0 and 0.0.
 *     (very loud to muted). 1.0 is the default gain for all speakers.
 *
 * int fade_speaker(int speaker, float time, float start_gain, float end_gain)
 *     Fades the gain of "speaker" uniformly from "start_gain" to "end_gain" in "time" seconds.
 *     To be implemented.
 *
 * int fade_all_speakers(float time, float start_gain, float end_gain)
 *     Fades the gain of all speakers uniformly from "start_gain" to "end_gain" in "time" seconds.
 *     To be implemented.
 *
 * int fade_channel(int tag, float time, float start_gain, float end_gain)
 *     Fades the gain of sound source "tag" uniformly from "start_gain" to "end_gain" in "time" seconds.
 *     To be implemented.
 *
 * int getNrChannels(int tag, int& srcs)
 *     Returns the number of mono sound sources provided by the active source "tag". This
 *     should always be the same numbers as initially returned by load().
 *
 * int getNrSpeakers(int& srcs)
 *     Returns the number of speakers handled by the NDSoundServer. This number should be constant
 *     during one session with the server.

The NDSoundServer

This is the structure of the sound server:

---

---

The Sound Core is the interface of the server to the hardware drivers. It handles both the playback of samples as well as the collection of data via input channels like microphones. The Mixer combines the mono sound streams to create the sound streams for the individual speakers and controls the gain of the speakers. The Mapper is responsible for separating multiple channel streams (e.g. stereo) from a source into mono sound streams and is also responsible to keep the multi-channel source synchronized with respect to the mono streams it provides. The Sound Source Handler manages open sound sources and assigns them universal tags which are used throughout the system to identify open sources and their channels. Each Sound Source like the File Sound Source or the Microphone Source provides the interface to embed the characteristics of the particular source stream with the system. The Socket Server provides the TCP/IP network interface. It implements the protocol and converts protocol commands into the respective interface calls of the other modules.

Sound Core

The sound core contains the interface between the NDSoundServer and the sound drivers of the used sound cards. It contains functions to start and stop the playback to speakers and the recording from microphones or other inputs. In general, all used sound devices are opened and playback and recording is constantly running. Whether or not actual sound will be heard, i.e. the playback buffers are not zero will be handled at higher level. The link to the higher levels in the server architecture are provided in the callbacks of the sound devices when a sound buffer has been played. At this point the sound core will request the next data block for a speaker channel from the mixer. Also the callback will initiate synchronization of sources through the mapper.
The MicSoundSource is closely linked to the callback of the recording device. The buffers used in the sound core for this purpose are directly used by the MicSoundSource to avoid unnecessary copying. Not, that there is no synchronization guarantee between recording and playback.

Mixer

The mixer handles three tasks. It handles links of mono sound sources to the speakers. Any number of mono sound sources can be connected with any number of speakers allowing any possible mapping of sources into the virtual room. It provides gain control for the individual speakers, which is mainly important for master gain control. Finally it mixes the mono sound streams to a speaker channel on request from the sound core. The mixing and gain control is implemented using long integers for performance reasons and to allow future porting to MMX.

Mapper

The mapper maintains the logical connection from virtual mono sound sources to physical sound sources which may provide multiple channels. This means, that given a virtual mono sound source, the mapper is able to locate the appropriate sound source and request the correct channel buffer from that source. As the mapper is also aware of all active physical sources, it is responsible to advance the sources input buffers on a synchronization event triggered by the sound core.

Sound Source Handler (SoundSourcePool)

The sound source handler assigns newly opened physical sound sources a virtual tag, which is used throughout the server to uniquely identify this sound source. The sound source handler can identify if a provided tag is valid and if so, it can provide the physical sound source for this tag.
Conceptually effects are treated like a sound source by the handler and active effects are also addressed by a unique tag.

Generic Sound Source

Every sound source has to provide an implementation of a specified set of operations. This includes providing a specific current mono block in the source stream and preparation of the next blocks of data in the stream. The outgoing mono sources can be amplified by a gain which is valid for the physical sources (i.e. for all channels the same gain). The playback can be paused, started and if applicable for the stream type, it can be checked if the end of the stream has been reached. If the sound source permits, a toggle for loops is provided. The start position and the end position for playback/loops in the sound stream can be specified.

File Sound Source

The file sound source implements all generic operations as intuitively expected. The end of the stream can only be checked if the file sound source is not set to loop.

Microphone Sound Source

The microphone sound source, due to its inherent properties, does not provide some of the generic features of a sound source. First of all, loops are not possible and the position cannot be reset to the beginning of the stream. The effect of pause and play are somewhat different from the file sound source, in that the input from the mic is muted, but, as is to be expected, resumes playback at the current moment in time and not at the moment it was paused.

Socket Server

Incoming TCP/IP requests on port 5180 are handled by the socket server. The socket server has to maintain a link between an open socket connection and active sound sources which were loaded by this connection. In case that the connection is closed by the client and not all active sources have been closed, the sound server has to close all those "zombie" sources.
The ND-Sound Protocol (NDSP) implemented by both the Socket Server and the NDSoundServerStub is a simple text-based protocol. The server only responses to request from the client. A client request consist of one text line of specified protocol commands and the server responds either with "NO" if the request failed, or with either "OK" or a specific response depending on the request.
Valid NDSP requests with responses are:

---

Request	Response	Remarks
LOAD <sourcename>	<tag> <nrchannels>	If sourcename is __MIC, __DI1 or __DI1, the microphone, DarlaIn1 or DarlaIn2 channel will be opened if possible, otherwise the server will be looking for an NDS sound file in its directory. <tag> is the virtual tag for the opened sound source. <nrchannels> is the number of mono channels provided by the sound source.
GETCH <tag>	<nrchannels>	Returns the number of mono channels of a sound source.
GETSP	<nrspeakers>	Returns the number of speakers handled by the sound server.
GAINCH <tag> <gain>	OK	Sets the gain of a sound source to the float value <gain>. Should be between 2.0 and 0.0.
GAINSP <speaker> <gain>	OK	Sets the gain of a speaker to the float value <gain>. Should be between 2.0 and 0.0.
PLAY <tag>	OK
PAUSE <tag>	OK
RESET <tag>	OK
LOOP <tag> <loop>	OK	If <loop> is 0, the sound source will not loop otherwise it will.
START <tag> <pos>	OK	Set start of playback/loops to sample-position <pos>. Default is 0.
END <tag> <pos>	OK	Set end of playback/loops to sample-position <pos>. Default is the end of the stream.
ATEND <tag>	OK/NO
FREE <tag>	OK
MAP <tag> <channel> <speaker>	OK	Maps the mono source channel <channel> of source <tag> to <speaker>.
MAPALL <tag> <channel>	OK	Maps to all speakers.
UNMAP <tag> <channel> <speaker>	OK
UNMAPALL <tag> <channel>	OK

---

If you have further questions, feel free to contact us.

	Still Image Viewer: yuqun@cs neilshah@cs zjn@cs
	Sound Server: prc@cs gessl@cs
	Interface: yuqun@cs emilp@cs zjn@cs
	OpenGL server: rudro@cs