Page  00000378 Extended Applications of the Wireless Sensor Array (WISEAR) David Topper Virginia Center for Computer Music University of Virginia 112 Old Cabell Hall Charlottesville, VA 22903 tope: rvirginia.edu ABSTRACT WISEAR (Wireless Sensor Array)8, provides a robust and scalable platform for virtually limitless types of data input to software synthesis engines. It is essentially a Linux based SBC (Single Board Computer) with 802.11a/b/g wireless capability. The device, with batteries, only weighs a few pounds and can be worn by a dancer or other live performer. Past work has focused on connecting "conventional" sensors (eg., bend sensors, accelerometers, FSRs, etc...) to the board and using it as a data relay, sending the data as real time control messages to synthesis engines like Max/MSP and RTcmix1. Current research has extended the abilities of the device to take real-time audio and video data from USB cameras and audio devices, as well as running synthesis engines on board the device itself. Given its generic network ability (eg., being an 802.11a/b/g device) there is theoretically no limit to the number of WISEAR boxes that can be used simultaneously in a performance, facilitating multi-performer compositions. This paper will present the basic design philosophy behind WISEAR, explain some of the basic concepts and methods, as well as provide a live demonstration of the running device. 1. INTRODUCTION Previous work on the WISEAR box has been an attempt to build upon conventional sensor applications using BASIC stamps and PIC chips. While both versatile and relatively inexpensive, they are also are prone to forcing projects into a unique one-of-a-kind category. Typically projects end up being very custom made and difficult to reproduce. Additionally, the inherent limitations of the BASIC language are often found to be a nuisance. A primary goal of the WISEAR project was to put forth a general purpose, generic, robust and scalable platform for sensor / data / wireless transmission that contemporary controller research often seeks. The first generation WISEAR is essentially a TS-7200 Single Board Computer (SBC) made by Technologic Systems2' It is part of an emerging trend, using SBCs for wireless data communication5: Paradiso's Sensemble6 and Flety's WiSe Box7 are contemporary examples. The beta version of WISEAR used an x86 based chip which was much larger and power hungry than the current version, which uses a 200 MHz ARM9 processor with power as low as 1/2 Watt. The device is essentially a complete Linux based computer that weighs less than 1lb and measures approximately 4" x 6". Hard drive space is provided via a flash memory card. A standard configuration includes two USB ports, a serial port, RJ11 (ethernet), twenty channels of digital IO and eight channels of analog input. Daughter boards are available to increase the total number of inputs. The only thing that needs to change from application to application when using WISEAR is some basic soldering to the inputs. Software loaded onto the board automatically scans Troy Rodgers Virginia Center for Computer Music University of Virginia 112 Old Cabell Hall Charlottesville, VA 22903 tsr9f@virginia.edu each input device for a signal, and if received sends data out via wireless TCP packets to any number of software synthesis engines such as Max/MSP and RTcmix. Data is transmitted in generic pairs (controller number / data value) and hence is routed on the receiving end. Under Max/MSP, data can be received easily using calls to the mxj tcp.recieve object. Using 802.11 a/b/g instead of FM radio overcomes a variety of issues. Range is increased. Tuning the respective transmitter and receiver is no longer an issue. The frequency range for 802.11 devices is also far less prone to interference from other devices, since it is such a popular standard for wireless internet access. Bandwidth is in the 5 megabyte per second range (eg., for 802.1 ib). Being a general purpose device, it becomes very simple to add off-the-shelf (OTS) components such as USB audio and video input. There is no need to design a special interface to read audio or video data. Similarly, 802.11a/b/g hubs and routers are readily available. Since the unit is also a fully functional personal computer running the Linux operating system, it is capable of running software synthesis engines on board. Speed is obviously an issue, being a 200mhz processor, but simple synthesis and effects processing algorithms are easily realized. WISEAR helps resolve several data transmission issues inherent in MIDI. The analog to digital converters built into the board offer twelve bit resolution. Data speed is only limited by the sensor latency and the processor speed of the device, particularly for digital i/o based sensors. Current realapplication measured latency and speed measurements are currently in progress, but initial results suggest orders of magnitude improvement over the 31.2kbs data rate. Even at helf full bandwidth of 54 mbs (the theoretical 802.1 g upper limit) the data transmission bottleneck is clearly no longer a problem. Instead, using low latency circuitry (eg., accelerometers with 1ms latency vs 10ms) can be a primary focus. Since the device is a complete Linux PC, program variables can be of type float or long integer. 2. AUDIO INPUT / OUTPUT WISEAR does not come standard with an audio input device. It does, however, have two built in USB ports. As a result, using the Linux ALSA driver, it is possible to connect most generic USB audio devices. Our research has used the USBgear USB microphone and speaker dongle. The device is both lightweight and small, which allows the WISEAR device to remain both portable and wearable. Audio data is read using standard ALSA read/write calls from RTcmix running on board the device. It is then sent out in a manner different from other WISEAR sensor data. Sensors are numbered 0-27. The first eight are analog inputs. When written via TCP write calls, data is sent using two sixteen bit integers. The first is the controller number. The second is the data itself. Audio data, while being read at a higher rate, is 378

Page  00000379 various other utilities isn't really feasible. Thankfully, Technologic Systems provides a development kit which facilitates compiling binaries on your Linux PC and copy them over to your Linux ARM-based SBC. The kit involves installing a custom kernel and using special compilers which will build binaries for ARM processors under an x86 architecture. But development is still relatively seamless. Code that runs on a Linux PC, even USB video code, can be recompiled and copied. Getting RTcmix running on the WISEAR was simply a matter of recompiling the basic program using the development kit. Speed remains somewhat of an issue for complex processing. Figure 1. WISEAR using TS-7200 SBC with standard ports. written out via TCP in regular interleaved audio frame format. Each channel is sent sequentially, first left then right. A special purpose RTcmix instrument has been created to perform the audio reading and writing called TCPMIX. WISEAR supports full duplex audio via the USB audio device. So data can also be received via the TCPMIX instrument and sent out through the USB audio device, then routed to a pair of wearable speakers or headphones. The generic nature of the device supports virtually limitless applications. 3. VIDEO INPUT Video input is also achieved using OTS hardware. Our research has used Logitech Quickcam for Notebooks Pro. Other webcams can also be used, but the Logitec device is already designed to be portable and lightweight, so fits this application. Video data is read using the Video4Linux3 API. Pixels are sent as RGB data (uncompressed) in a manner somewhat different from other audio and sensor data. There is no real need for controller/value pairs as the data type qualifier is only necessary at the outset: video. Currently, video can only be sent to the GAIA4 (graphical audio interface application) program. But work is being done to facilitate sending network video data to Max/MSP using Jitter. Image information is sent line by line via TCP writes. So, for example, once six hundred and forty rgb triplets have been sent, the next line of the image is written, another four hundred and eighty times. This is not the most optimal method. But using USB video with WISEAR is still very much in the development stage. Work needs to be done on compressing the video image. Currently, we are averaging approximately five frames per second of video. 4. EMBEDDING REAL TIME AUDIO APPLICATIONS INTO WISEAR Persistent memory space comes at a high premium on the TS7200 and most SBCs. So putting a complete Linux distribution on the chip, complete with libraries, header files, compilers and Figure 2. Dancer wearing WISEAR using TS-7200 SBC with standard ports, bend sensors, FSRs and accelerometers. 5. FUTURE DIRECTIONS Despite limitations of speed and built in memory, WISEAR remains an effective platform for gathering sensor, audio and video data. It's small form factor and wireless data transmission method make it an ideal candidate for dancers and performers. More work needs to be done optimizing video data compression. Benchmarks for latency on digital and analog sensors also needs strict testing. Additionally, more work is needed using multiple WISEAR boxes simultaneously. Current testing has only looked at two. There is theoretically no limit, other than wireless bandwidth, on how many can run simultaneously in a performance space. There are many implications for using audio and video feedback, using the work described in this paper. Similarly, special sensors such as sonar can be used in novel ways by synchronizing transmit / receive times. One major critique of the initial WISEAR platform has been size. Dancers have pointed to the support waist belt as being too warm to wear for extended periods of time. Musical performers have expressed a desire for a device small enough to be mounted on an instrument. It has been a contention of the original authors8 that SBC technology would continue to shrink in size while simultaneously becoming faster, following the general trend in computer technology. As of the writing of this paper, that time has come. 379

Page  00000380 Gumstix, Inc.9 has developed a line of "waysmall" computers which are, quite literally, the size of a stick of gum. Figure 3 shows the original WISEAR alongside a newer version that uses the Gumstix technology. The paradigm behind the Gumstix is a bit different from previous generations and perhaps even better suited to the needs of the electronic music / musical interface community. Devices are ordered in a modular fashion, normally with a main motherboard having external expansion boards that plug in via different connectors built into the motherboard. The device shown in figures 3 and 4 consists of a 400 mhz ARM "Connex" motherboard, an 80211ig "wifistix" expansion board and a special purpose "robaudiostix" board which allows for a combination analog-to-digital conversion, digital input and output, and audio input and output all on one chip. The expansion boards are stacked vertically on top of or below the motherboard, depending on the connection type. The configuration shown is less than 0.5 inches tall and weights approximately eight ounces. Several other expansion boards are also offered: GPS, USB, etc... Figure 4. Next generation WISEAR Gumstix platform. 6. REFERENCES [1] (n.d.) Retrieved 4/31/07 [2] Technologic Systems, (n.d.) Retrieved 4/31/07 [3] htt://ýywyex ioitsorv41/ (n.d.) Retrieved 4/31/07 [4] ] Topper, D. "GAIA: Graphical Audio Application", Proceedings of ICMC2004, (November 1-6, 2004, Miami, FL) [5] "A Wearable Wireless Sensor Platform for Interactive Art Performance", Proceedings. of PerCom 2006, (March 13-17, 2006. Pisa, Italy), pp. 52-59. [6] Aylward, R. and Paradiso, J.A., Sensemble: A Wireless, Compact, Multi-User Sensor System for Interactive Dance, Proceedings of NIMEO6 (June 4 -8, 2006, Paris, France), pp. 134-139. [7] Flety, E. The WiSe Box: a Multi-performer Wireless Sensor Interface using WiFi and OSC. Proceedings of NIMEO5, (Vancouver Canada, May 26-28, 2005), pp. 266-267. [8] Swendsen, P and Topper, D. Wireless Dance Con trol: PAIR_ a WISEAR. Proceedings of NIMEO5 (Vancouver Canada, May 26-28, 2005), pp. 76-80. Figure 3. First and second generation WISEAR side by side. Work using the Gumstix platform is still very much in the early stage of development. But it seems clear that this will be the future of WISEAR research. The advantages of size, speed and modularity are considerable. Much like the ARM based TS7200 board, a development kit must be installed on the host computer to compile executable code. But there is a growing community and large body of documentation to help facilitate this research. As of the writing of this paper, it is possible to develop code under Mac OS X, using the Gumstix "Buildroot" development kit and compile code which can be exported to a 400 mhz Connex motherboard via serial connection. It seems likely that bluetooth and tcp based file sharing is also possible. Another current area of future research using WISEAR and Gumstix is musical robotics. The Gumstix platform is already being used by the robotics community. The general purpose 1O (GPIO) and digital 1O (DIO) connections on the robaudiostix (shown as part of figures 3 and 4) are designed in large part for this purpose. So the device will serve not only as a means of sending sensor data, sending and receiving audio data, and sending video data. It will serve as a platform for controlling robotic musical agents. 380