Page  00000317 DIPS for Linux and Mac OS X Shu Matsuda, Chikashi Miyama, Daichi Ando, Takayuki Rai Sonology Department, Kunitachi College of Music 5-5-1, Kashiwa-cho, Tachikawa-shi, Tokyo, Japan. { shu, chikashi, daichi, rai } Abstract In this paper, we would like to present the new Linux and Mac OS X versions of DIPS (Digital Image Processing with Sound); the set of Max objects that handle real-time visual image processing events and OpenGL functions in the jMax environment. We will describe the significant modifications and improvement of DIPS for Linux and Mac OS X Furthermore we include that DIPS for Mac OS X can be utilized as an internet tool by adapting the Apple QuickTime technology. 1. Introduction The DIPS (Digital Image Processing with Sound), developed by Shu Matsuda (Digital Art Creation) and presented in the ICMC 2000 in Berlin, is an ensemble of Max objects that handle real-time visual image events in the jMax environment. This set of objects enables the interaction between audio events and visual events in a Max patch, and strongly supports the realization of realtime interactive multimedia art. There are three important features of DIPS. (1) It enables programming OpenGL functions in the Max GUI programming environment. (2) No need to compile. So we can work heuristically in real-time. (3) Enables interaction between audio events and visual events by normal Max patch programming. However, running DIPS on an SGI machine has some disadvantages. First of all, the CPU's calculation speed of SGI machines is not evolving dramatically since a few years ago. DIPS requires a great deal of processing power for 3D-model rendering and video processing. And SGI machines are relatively expensive compared to recent powerful PC and Macintosh computers. Thus we decided to port DIPS to the Linux environment and also to the Mac OS X environment. Fig 1. The DIPS programming in the jMax patch 2. Porting the core part of the DIPS The core part of DIPS for SGI has been developed with the Digital Media Library of SGI machines. Because the Digital Media Library is the specialized library for SGI machine, the core part of the DIPS; the window handling and frame buffer management, had to be newly developed. For the window handling OpenGL extensions are employed for both Linux and Mac OS X environments. On the other hand, most of the DIPS objects executing OpenGL functions can be compiled for both environments without any difficulties. Fig 2. DIPS for Linux 317

Page  00000318 3. DIPS for Linux Here are some remarkable modifications and improvement realized in DIPS for Linux. A) Flexible window handling In DIPS for SGI the rendering windows didn't have any flexibility. The DIPS for SGI has two windows for rendering, that can not be resized or closed during the session. While DIPS for SGI implemented the window management functions as a part of the DIPS core, DIPS for Linux allows users to handle multiple rendering windows using the new DIPS object "DIPSWindow". With "DIPSWindow" object users can define properties of each windows just like other DIPS objects. Therefore the window handling possibility of the DIPS for Linux can be extended. B) Graphical devices On the SGI machines we can not update the graphic chip with the most recent powerful one. But on the PC/Linux machine we can replace the graphic chip with the most recent powerful one at any time without any difficulty. The current version of DIPS for Linux is tuned for nVidia's GeForce 3 graphic chip and we are planning to replace it with the GeForce 4 TI or any faster graphic chip that appears soon. Since most of OpenGL calculations are executed on the graphic chip it is critical to use the faster graphic chip for more smooth rendering. C) Applying Quicktime for Linux Quicktime for Linux, provided by the Heroine Virtual, is a very convenient library for reading and writing uncompressed quicktime movies. DIPS for Linux utilizes it for loading the sequence of pictures in the supported format to the processor memory. Quicktime for Linux can handle the following formats; OpenDivx, DV, MJPA, JPEG Photo, PNG, Uncompressed RGB, Component Video, YUV 4:2:0 Planar, Component Y'CbCr 8-bit 4:4:4, Component Y'CbCrA 8-bit 4:4:4:4, and Component Y'CbCr 10 -bit 4:4:4. D) Applying Linux video capture technology The video capture fuctions in the DIPS for Linux are realized by the library "Video for Linux". In most Linux distributions it is included as one of its default libraries and supports various video capture chips. The video capture objects in DIPS for Linux are written in C language with this "Video for Linux" library. Currently our PC/Linux machines are equipped with the video capture chip "BTTV8*8 series". I. U I Fig 3. Architecture of DIPS for Linux 4. The DIPS for Mac OS X Porting jMax to Mac OS X has been making progress, so that we could start porting the DIPS to Mac OS X environment. Mac OS X utilizes Apple QuickTime, which is the API for handling video/animation/graphics, text, as well as music/sound. We decided to introduce the full video features of the Apple QuickTime to the DIPS project. DIPS for Mac OS X is becoming a more powerful image processing extension of jMax for Mac OS X. Here are some of its features; A) Supported media types DIPS for Mac OS X is able to import and export more than thirty formats into the "DIPStable" object. It can handle any video, image and text including PDF format that QuickTime 5 deals with. Therefore users can introduce any digital visual materials to their own works without any preparation. In addition, DIPS for Mac OS X employs the real-time direct QuickTime renderer for video rendering from any sources to the frame buffers, the texture buffers, files, and network that will be discussed later. As the result the physical memory limitation of "DIPStable" in DIPS for SGI is solved. The import file formats include; AVI, BMP, DV, FlashPix, FLC, GIF, JPEG, JFIF, MacPaint, Macromedia Flash 4, MPEG-1, PDF, Photoshop, PICS, PICT, PNG, QuickTime Image File, QuickTime Movie, SGI, TARGA, Text, TIFF, AVI, BMP, DV Stream, FLC, Image Sequence movie exporters, JPEG, JFIF, MacPaint, Photoshop, PICT, PNG, QuickTime Image, QuickTime Movie, SGI, TARGA, Text, TIFF, and MPEG-4 will follow soon. And the supported compression formats are; Animation, Apple BM, Apple Video, Cinepak, Component video, DV NTSC and PAL, Graphics, H.261, H.263, Motion JPEG A and B, None Compression, Photo JPEG, Planar RGB, PNG, Sorenson Video, Sorenson Video 3, TGA, and TIFF (extensible by installation of Components (the thirdparty compressor components)). 318

Page  00000319 B) Video rendering Apple QuickTime is optimized for the Macintosh computer equipped with several video acceleration hardwares and works fast enough for video representation. It also has a flexible file/user interface. In order not to spoil these features, DIPS for Mac OS X wraps the QuickTime functions as follows; C) Sequence Grabber objects Sequence Grabber components of the Apple QuickTime allow DIPS for Mac OS X to obtain digitized data from external sources. By using video part of Sequence Grabber, DIPS for Mac OS X realizes real-time video capture from any QuickTime sources compatible with video capture devices. D) Supporting real-time QuickTime visual effects Apple QuickTime supports some real-time visual effects hardware (various third-party hardware accelerator cards). The DIPS realizes the following real-time visual effects built in the QuickTime 5; Alpha gain, Blur, Color balance, Color style, Color tint, Edge detection, Emboss, Film noise, General convolution, HSL balance, Lens flare, RGB balance, Sharpen, SMPTE effects, Travelling matte, and Zoom. The CPU-based visual effects and video signal analysis realized by the "DIPStable" object in the DIPS for SGI are still available. URL. Therefore all of fast-start movie data put on the internet is accessible directly from the DIPS objects. In addition, users can point URLs of the Real-Time Streaming Protocol (RTP/RTSP) as video sources by importing the reference movie or the SDP(Session Description Protocol) file. This means that users can employ streaming video data on any site of the internet to their own works in real-time. Furthermore, DIPS for Mac OS X can render video data to network as we mentioned before. The "DIPSstreaming" object is designed for making the DIPS into a video streaming server. The "DIPSstreaming" can take media files, sequence grabbers, frame buffers and audio signal lines of jMax as source of video and sound data. It realizes the streaming broadcast as uni-cast or multi-cast. Therefore, DIPS for Mac OS X can become a QuickTime Streaming Broadcaster. DIPS for Mac OS X is a unified programming environment of multimedia in this broadband internet era. 6. Implementing DIPS for Linux into art creation Several composers and students at the Sonology Department, Kunitachi College of Music have already realized interactive multimedia works using DIPS for Linux. Most of those works were written for instrument(s) and a live computer electronic system. For the signal processing of instrumental sound most composers employ Max/MSP on Macintosh computers, while the visual part is realized with DIPS on PC/Linux machines. The synchronisation between two machines, or between sound events and visual events, is realized via the Ethernet using the UDP/IP Protocol. Via the scan converter, the rendering window on the PC/Linux processed with the DIPS objects is projected to the screen behind the stage. Here we would like to mention two examples; A) "Aqua for a MIDI piano and a multimedia interactive system" was created by Chikashi Miyama in 2002. In this work, the composer mainly utilized various particle effects with DIPS. With the Pentium 4/1.4GHz PC/Linux machine equipped with the GeForce3 chip more than a hundred of texturemapped simple particle objects can be rendered smoothly. Fig 4. Architecture of DIPS for Linux. 5.Accessibility of network video data in DIPS The "DIPStable" video objects can be instantiated not only from media files and Sequence Grabber, but also from the fast-start/streaming movies. DIPS for Mac OS X handles file sources as a description of 319

Page  00000320 Fig5. Aqua for a MIDI pft and a multimedia interactive system. B) "Fusai Sankei (Three Landscapes) for percussion and a multimedia interactive system" created by Daichi Ando in 2002. In this work movement of a percussion player on the stage is sent to the computer via CCD camera. This live video image is processed and mixed with the 3D rendered objects as well as analyzed. In some parts the analyzed image data, thus the movement of the player, controls the sound events. hardware. development we still have to wait for faster hardware to become available. For instance, the video capture process realized on the PC/Linux machine is much slower than on SGI machines. But, we are sure the circumstance of DIPS will be improved and that DIPS for both Linux and Mac OS X will stimulate composers and artists to create more sophisticated interactive art. Reference [1] Dechelle, F., De Cecco, M., Maggi, E., Schnell, N. "jMax: An Environment for RealTime Musical Applications", in Computer Music Journal, 1999, Beijing, China. [2] Dechelle, F., De Cecco, M., Maggi, E., Schnell, N. "jMax recent developments", in Proceedings of the International Computer Music Conference 1999, Beijing, China. [3] Dechelle, F., De Cecco, M., Maggi, E., Schnell, N. Rovan, B., Borghesi, R. "jMax: A new JAVA-based editing and control system for realtime musical application", in Proceeding of the International Computer Music Conference 1998, Ann Arbor,Michigan,, USA. [4] Dechelle, F., De Cecco, M., Maggi, E., Schnell, N. Rovan, B., Borghesi, R. "Latest evolutions of jMax realtime engine", in Proceedings of the International Computer Music Conference 1998, Ann Arbor, Michigan, USA. [5] Maggi, E., Dechelle, F. "The evolutions of the graphic editing environment for the IRCAM Musical Workstation.", in Proceeding of the International Computer Music Conference 1996, HK. [6] Lindmann, E.,Dechelle, F., Starkier, M., Smith, B., "The Architecture of the IRCAM Musical Workstation.", in Computer Music Journal, 15(3):41-50,1991. [7] Danks,M., "The Graphic Environment for Max", in Proceedings of the International Computer Music Conference 1996. [8] Takashiro, O.,Rai, T., "The Interactive Multimedia Computer System using SGI and NeXT/ISPW computers", in Proceedings of the International Computer Music Conference 1996. [9] Matsuda, S.,Rai, T., "A visual-to-sound interactive computer performance system "Edge", in Proceddings of the International Computer Music Conference 1995. [10] Matsuda, S.,Rai, T., "DIPS: the real-time digital image processing objects for Max environment", in Proceddings of the International Computer Music Conference 2000. [II] Heroine Virtual "QuickTime for Linux [] [12] Apple Computer, Inc., Inside QuickTime: API Reference for QuickTime [] [13] Apple Computer, Inc., What's New in QuickTime 5 [14] Apple Computer, Inc., Preliminary Documentation: Broadcasting with QuickTime5 Fig 6. Fusai Sankei for percussion and a multimedia interactive system 7. Conclusion As we have been describing, by being ported to the PC/Linux and Mac OS X environments, now DIPS becomes a broader powerful tool for the creation of realtime interactive sound-visual art. We also presented the possibility that DIPS could be utilized in the internet field by adapting QuickTime technology. However we must mention that it still has to be improved. First of all we haven't been able to assemble the handy DIPS objects library. Programming in the DIPS requires considerable knowledge of OpenGL programming technique. The DIPS objects library will lighten the programming burden imposed on composers. In the point of 320