Page  00000224 REALITV Peter Castine ABSTRACT RealiTV is an audio/visual art project developed out of that which has de facto become the most ubiquitous of today's technical communications media: digital satellite television. By simultaneously processing multiple incoming broadcast streams in real-time using a rich vocabulary of processing algorithms -ranging from the deterministic to highly indeterministic models -continuously decontextualizing the images and sounds a creative and highly asthetic collage stream is created. The realiTV project has a number of distinct realizations: it can be configured as a performed work or as either an interactive or passive installation. All configurations share a common asthetic and technical basis described here. As well as documenting the asthetic premises of the work, this paper discusses the technical means used to achieve the work's ends. 1. INTRODUCTION Television is an X-ray. Not a pictorial, not a visualform. It is X-ray. - Nam June Paik The medium is the message. - Marshal McLuhan McLuhan's [in-]famous equality has been widely debated, disparaged, and deliberately misunderstood. Perhaps the most famous critique is Eco's attempt to distinguish alternately between form, code, and channel of "the medium" [3]. It is precisely the obstacles and impediments inherent in the manifest question, "What is the medium?" that encourages the authors to tackle this issue in the context of that most ubiquitous of contemporary media, satellite television. The realiTV project encompasses several different realization structures, ranging from performance configurations to a variety of interactive and passive installation versions. These share a common a~sthetic and technical fundament. In the following, the installation format will be the focus, as one exemplary form of the project as a whole. RealiTV is realized as a machine, producing streams of images and audio material, torn out of the increasingly ubiquitous digital satellite television program by means of stochastic processes. These serve as the raw materials of an a~sthetic collage. In the process, the viewer's evaluation of individual scenes and perceptions moves to a decontextualized metaobservation of entire programmatic sequences; the John Dekron viewer is forced to create her own individual transmission logic given the sounds and images provided by realiTV. Imagine a room. More spacious than your typical living room or TV den; nevertheless, the association with the situation of watching television at home is in evidence. In the background, almost accidentally, cables connect television, satellite receivers, and antennx, thereby making a signal chain from satellite dish to display evident. Lighting is discrete. In this almost cozy environment, the viewer's focus is on a television screen, continually flooding the eye with images, the ear with sound, all beyond control of the observer. Through coincidental juxtaposition of picture, music, and spoken word, all of it torn out of the intended context, the broadcast material is processed in ways that explicate the "medial" nature of television. We cannot remind ourselves often enough that the word "media" is derived from the Latin plural neuter of medius- the center or focus, coming through centuries of modern usage connoting intermediary, carrier, mediator, conveyor of someone or something. Thus, the medium television mediates something by means of transmission. Not only are we interested in the technical coding and decoding of transmitted material, we are also dealing with a visual encoding, one which the experienced television viewer may be able to decipher or "read". The technical aspects of the coding/decoding process should, ideally, be transparent, even invisible, to the viewer. This norm of the viewing process is deliberately contradicted by realiTV, bringing the nature of the medium television to the fore. The television experience is no longer that to which we have become accustomed. The subconscious television "decoding" process continues to run, regardless of realiTV's transformations and deformations, reading the "message". The new content is a collage constructed by the machine realiTV, fabricated from constituent parts in ways never conceived by their original producers. Viewers are confronted by the impossible task of attempting to make sense out of it all. The reaction may be any of anger, fear, sadness, disgust, surprise, curiosity, or joy [4] -or any combination. Acceptance is difficult, despite the ultimate inevitability of this final state. 2. DEVELOPMENT AND HISTORY The realiTV project began in the German underground Club scene, in an attempt to escape the prevalent (and 224

Page  00000225 Figure 1 A scene from realiTV. increasingly tedious) norm of looping and mixing preastheticized materials as the primary, if not exclusive, processing technique used by VJs. After several years of development and refinement of realiTV as a purely video art form used to accompany club performances that had, at best, a coincidental musical relation to the asthetics pursued by the video, the authors came together to join their skills in pursuing a gesamtkunstwerk, coinvolving sound and moving image as equal components with common artistic goals. The first public viewing of this cooperation was displayed in the 2005 edition of the DESIGNMAI festival in Berlin. Subsequent installations have taken place at the "Long Night of the Sciences" festival, jointly organized by all Berlin institutes of higher education and research, and at the Transmediale festival in 2006. 3. IMPLEMENTATION OVERVIEW The technical implementation of realiTV is a machine, consisting of at least two satellite dishes, the same number of digital satellite receivers, two computer programs (which may either be run on a single computer, or on two networked computers), and a large TV display. This last component can be implemented by a beamer, although we prefer to use one or more widescreen TV displays (34" or greater). Depending on the installation space and TV display two or more loudspeakers may be installed to enhance the audio output. The machine automatically chooses two (or possibly more) television channels from the available transmission range, changing channels algorithmically. Editing and effect processes are controlled algorithmically by a supervising computer program. The video supervisor program is, in turn, sensitive to the rhythmic output generated by the audio program. The audio program generates sound output by processing the television input signal, generating a broad continuum of rhythmicized and arrhythmic patterns, with both subtle distortions and less subtle, to the point of Figure 2 Another scene from realiTV: eight seconds after Figure 1. unrecognizability. The audio program is also sensitized to the processing performed by the video supervisor and can react intelligently to changes made to the video signal. 4. THE SOFTWARE 4.1. The Video Processor RealiTV's video engine is realized using Max/MSP with Jitter. The video raw material is typically drawn from two digitizers (possibly more, depending upon machine configuration). The digitizers receive signals from satellite receivers and transmit them to the computer for processing. The video process rends the input material asunder, generating a multitude of clips that are joined together, forming one or more new collage-like streams. The individual streams are treated by a variety of effect processes and united in accordance with custom algorithms. These video processes are controlled by a specially developed sequencer, serving as a metaphor for the "cut rhythm" used in preparing typical television broadcast material. The patterns used are, however, continually generated anew. The video effects used are related to realiTV's media-theoretical context. For instance, the effect named Polke rasterizes a television image to a coarser dot grid of offset points, determining the diameter of the dot to be rendered at each offset point based on the mean brightness of the original image across the respective grid area. The offset points themselves reflect the digital nature of the signal transmitting the original image. The ensuing distortion underscores the circumstance that the image is, in fact, an optical illusion. Or, as Paik put it, an X-ray. Another group of video effects thematizes the forms used for representing digital images. Typically, image files are defined with the first word in the data stream representing the pixel at the top left, working left-toright, top-to-bottom. Above everything else, the image 225

Page  00000226 Figure 3 Roundtable effect: the assumption of rectangular representation is contested. is assumed to be rectangular. Fig. 3 applies a different interpretation to a stream of image data. A third group of effects deal with mixing. Multiple video sources are juxtaposed simultaneously and thereby contextualized (Fig. 2 gives a simple example). No single television program can be viewed as an autonomous entity. The statement made by the medium is determined neither by a single broadcast program, nor by a single channel, nor even by a single monolithic "media empire". Paraphrasing McLuhan: TV is the message. 4.2. The Audio Processor Analogously to the video processor, the audio processor has been implemented using Max/MSP, making extensive use of external objects by Tristan Jehan (analyser- and shifter-), Miller Puckette's inescapable fiddle-, as well as the Litter Power Package, particularly those objects performing real-time spectral and timedomain mutation [2]. The satellite receivers serve not only as video sources, but also as audio sources. In the current implementation the audio signal from each of two receivers is reduced from stereo to mono and routed to either the left of right channel of the computer's audio input. The first process applied to the input is an artificial rhythmicization of the input signals, which in many cases will let the signals through unchanged. This parallels the cut-rhythm of the video impulse, but at a less macroscopic time level. This is followed by the main processing step, which chooses from a vocabulary of effects including variable delay; time-domain and spectral mutators ([5] and [2]); several convolution processes; analysis/resynthesis processes inspired by Barlow's Im Januar am Nil [1] (resulting in something that, unlike Barlow's orchestral score, sounds remarkably like recognizable speech); pitch shift processes; and a variety of cross-synthesis processes. All of these processes allows for a number of variants, typically choosing between either of the two input signals or a mix of both (in the latter case echoing the juxtaposition techniques used for the video signals). Like the video modules, each of the audio processing modules has additional parameters; a detailed documentation of these is beyond the scope of this paper. Due to the enormous variation in input levels, differences in the attributes of the signal processing modules, and variance in signal compression used by the broadcasters, an automatic gain control system is implemented. Without gain control, dynamics between processes can vary in excess of 120dB, resulting either in long periods of inaudible output or burnt-out loudspeakers. As impressive as the latter can be, the potential damage to the audience's collective breathing capacity, never mind the expense, are not to be disregarded. Finally, given Murphy's Law, a speaker is bound to burn out precisely when those responsible for installation maintenance are off-site for an extended period. Realizing that the entire realiTV project inherited a certain aesthetic stance from both Paik as well as Cage's Imaginary Landscapes compositions, use of the I Ching as the principle to inform the musical decision-making process in realiTV was felt to be dharma. We have developed techniques for examining I Ching oracles in great detail, going far beyond the notion of using yarrow sticks as a glorified pair of dice. Thus, the results of the individual trigrams, digrams, and even individual lines will be drawn upon for making certain decisions. Attention is also paid to those lines deemed to be "changing" (old yin and old yang) and the resultant "future" hexagram, complementing the primary hexagram chosen by the yarrow sticks (cf. Wilhelm [6]). The result is an intricate complex of interrelated decision-making processes at each step of the work, affecting the parameters used by the currently active processing module. 4.3. Communication between software components The video program provides hints about its activity to the audio program and vice versa. The video program can decide it has gotten bored with what the audio program is doing and trigger a new process (complete with new sets of time variant processes). And vice versa. We needed a simple, fast, and reliable underlying protocol allowing these two units to communicate. The number of computers used (which varies from configuration to configuration) must also be transparent to the protocol. It proved quite straightforward to build such a system on top of IP multicasting, using Java code supplied with the [mxj net.multi.*] objects. All that was needed was to determine an appropriate vocabulary of message identifiers and associated parameter values. Parameters can be calculated at arbitrary places inside the main Max/MSP patches and sent to dedicated network sender/receiver subpatches for multicasting. 226

Page  00000227 4.4. Communication with hardware components The goal of controlling the satellite receivers programmatically from Max/MSP proved considerably more challenging. When realizing the realiTV project as a performance instrument, there is an obvious and straightforward interface: the standard remote control used to switch programs. Although not the subtlest of performance interfaces known to mankind, these provide a simple and effective way to interact with the machine that is realiTV. With some effort we have found we can introduce a certain amount of sensitivity and even a nascent virtuosity while using these devices. In an installation environment, this is not an option. We have performed initial experiments in circuitbending conventional remote controls as an interface allowing an audience to interact with the realiTV system. At this date the results have not been sufficiently robust, nor are we convinced that this approach is truly compelling for the project as a whole. In a passive installation environment-which we find aesthetically most compelling--we need a way for one or both of the computers to control the satellite receivers, switching program automatically. As a rule, neither consumer nor professional satellite receivers provide this kind of functionality. We have experimented with customizing consumer receivers to allow us to control program selection via a card connected to a serial port. These experiments have proved less robust than we require. Attempts to solve the problem with commercially available video cards for desktop computers have also been unsuccessful to date, largely due to idiosyncrasies in the ways these cards interact with the host hardware -it is near impossible to even use two different cards in the same machine, and allowing both to simultaneously display their respective signals onscreen is more an exercise in interesting new ways to crash the OS than anything else. (We have been intrigued by a recent versions of eyeTV produced by Elgato Systems, but are now pursuing a different track, as described below.) Mid-2005 we began experimenting with the Dreambox, a Linux-powered DVB satellite receiver. The receiver comes with a stripped-down Unix shell and customized HTTP server and Ethernet interface. The latter two features are what we use to integrate the receivers into the realiTV system. In principle, this can be achieved with some simple messages to a jit.uldl object to send messages using HTTP. In practice, the programming is relatively arcane, requiring parsing of the undocumented data format used by the Dreambox to store user bouquets (the mapping of channel numbers to station frequencies and other broadcast parameters). It has also been less robust than we would desire, particularly because certain HTTP messages to the Dreambox generate no response. Regardless of whether this is compliant with the HTTP standard, it is the way the device works. Unfortunately, jit.uldl is easily confused by this state of affairs. Despite these irritations, the system in this form is flexible and seems to be sufficiently reliable. 5. FUTURE DEVELOPMENTS We are planning to develop a custom Max object to handle communications with the Dreambox, pending further evaluation of the dependability of our current approach. Additional modules for both audio and video processing are constantly under development. Modules for "freezing" and time-stretching moments of sound (analogous to video still-frame), stutter effects (deliberately reversing the conventional priority of audio over video continuity), as well as several granular resynthesis approaches are at the top of the to-do list. A more sophisticated look-ahead model for handling sudden changes in audio levels is also on the drawing board. 6. ACKNOWLEDGEMENTS The authors thank Dream Multimedia for their generous support. We are also grateful for support from realities:united GmbH and the Seminar for Media Studies at the Humboldt University, Berlin, in particular Prof. Wolfgang Ernst and Martin Carle. Verena Zell and Sebastian D6ring worked enthusiastically to place realiTV in the public eye. We are grateful for comments and suggestions from the paper readers. Ulrike Brtickner graciously provided professional graphic designs for print materials. She, together with Luzie, Wendy, and Danny, also put up with the noise. Thanks. 7. REFERENCES [1] Barlow, C. "Im Januar am Nil for Ensemble," Feedback Papers, Cologne, 1982. [2] Castine, P. "Litter Mutators: Moving Mutation inside Time." Proceedings of the International Computer Music Conference, Copenhagen, 2007. [3] Eco, U. Il costume di casa, Bompiani, Milan, 1973. (Engl. trans. Travels in Hyperreality. Harcourt, New York, 1986.) [4] Plutchnik, R. Emotion: A Psychoevolutionary Synthesis. Harper Collins, New York, 1980. [5] Polansky, L. and Erbe, T. "Spectral Mutation in SoundHack: A Brief Description," Proceedings of the International Computer Music Conference, Banff, 1995. [6] Wilhelm, R. (trans.) I Ging: Text und Materialien. Diederichs, Jena, 1924. (Engl. trans. I Ching: Or, Book of Changes. Princeton, Princeton New Jersey, 1950) 227