Page  00000327 Visualizing Sound Environment during Orchestra Performance based on Time Frequency Analysis Satoru Morita and Sayaka Tokunou Faculty of Engineering, Yamaguchi University smorita@yamaguchi-u.ac.jp Abstract We simulate the sound environment based on the beam tracing in the virtual hall generated using computer graphics. We visualize the sound environment on orchestra performance by coloring the major frequency observed in each positions of the virtual hall. In the case of defining the orchestra formation or instrument positions, it is available that it can be visualized sound environment considered player's positions. In this paper, we visualize the simulated sound environment while "Kanon" composed by Pachelbel is played by the string quartet in the virtual multipurpose hole at our university. 1 Introduction Various concert is being held in big and small various hall. Research has been done in the architectural field since the old days to build the hall of the good sound environment. A model experiment and a numerical simulation are repeated at the stage of the design, and a form is examined. Moreover, it is evaluated by actually measuring sound at the existent hall, and the clue of the next design is taken. Sound in the hall is simulated by computer. The trial to visualize the sound propagation in the virtual environment is being done (N. Tsingos and Carlbom 2001). It doesn't aim at visualizing the musical interval of music here. There is a simulation system (M. Monks 2000) to find the most suitable hall form by visualizing a sound environment. The various icons which evaluate a sound environment are expressed visually, and the hall form which it aims at by operating this icon is found out with this system. A sound environment is visualized for the purpose of an architect's examining design, and it doesn't aim at the visualization with a position of the player who uses a hall. A sound environment during the performance is visualized to support for not only the persons concerned in building but also a player. The state of the distribution of the sound in the hall is expressed by the classification by color of the frequency. A sound source is arranged together to a position of an instrument of the player on the stage, and the performance of every part is used as each sound source. The state that a sound environment changes can be seen in accordance with the performance, and a sound environment to change by changing player's arrangement can be seen, too. Before visualizing a sound environment, the following condition can be given to it. * 1. The sound can be visualized at all the positions inside the hall. * 2. The sound environment can be visualized in accordance with music. * 3. The sound environment can be simulated at the existent hall. * 4. The sound environment can be simulated at the virtual hall. * 5. Hall structure and a skin material can be changed. * 6. A sound source can be arranged together to the actual player's position. * 7. The simulation can be done to change player's arrangement and a tune. * 8. A hall form can be found in the specific purpose from the visualized result. The sound environment of the hall during the performance are visualized here the simulation which it satisfies a condition from 1. to 7. A condition except 2.6.7. is satisfied with the simulation system introduced first. 1. is different from the visualization of this research, and visualizes a change in the hall form and the evaluation symbol due to the change in the material mainly. But, player's arrangement isn't being taken into consideration in the sound simulation of the building field, and it can think about organization with the player a hall and a tune and instrument arrangement that a sound environment during the performance in consideration of the player's arrangement can be visualized. The process to compose performance in each position of the hall is expressed by using the sound beam tracing in section 2, and the performance is analyzed, and how to color main frequency in 327

Page  00000328 section 3. In section 4, we shows the effectiveness by visualizing a sound environment during the performance for the hall. 2 Sound Beam Tracing Performance in each position of the hall is composed to examine a sound environment at the hall. A reflection propagation route from the sound source to the receiving point of the sound is found by the sound beam tracing. The sound beam tracing, (Beranek 1996) which many sound beams start in the place solid corner interval from the sound source point and which is the way of chasing a reflection propagation route. A sound beam starts from the receiving point, and propaga rnm) (a) receiving point sound source ra(b) (b) Figure 2: Sound source and sound beam. that propagation distance d'(m) for each sound beam which reached a sound source from the receiving point where the weight w is calculated from the base distance D(m) to get the radius of the sound source and the propagation distance d'(m). D2 (2) Figure 1: Sound beam tracing.. From propagation route of each sound beam, a decline by the time when the arrival of the sound takes it, and the distance is computed and acoustic absorptivity coefficient to the wall side is found due to the reflection. When sound speed is made 343.5(m/s), the time when the arrival of the sound takes it is d second. As the spherical surface area of the radius d(m) is 47d2(m2), and the total energy W(watt) to pass for every unit time is in inverse proportion to square of the distance(Beranek 1996), the strength I(W/m2) of the distance d(m) between the sound source point and the receiver point is calculated as tion route to the sound source is found to compose performance in each position of the hall of this research. The receiving point of the sound by the sound beam tracing and a reflection sound beam figure to the sound source are shown in the figure 1. When propagation route is extended with the time, as for the sound beam, the probability that the mutual interval of the sound beam hits an extent sound source lowers. Because of that, you must find that strength by the number of sound beams which reach the inside of the fixed territory. A sound source is made the ball which has a fixed radius, and it thinks about the sound beam which reaches that territory by this research. As shown in the figure 2 (a), the radius r(m) of the ball is decided by using the distance D(m) which it is based on to the solid angle 0~ which a sound beam is shown from as W I=d2 47xd2 (3). Here, from the sound source, the decrement Dd(O < Dd < 1) of the distance d'(m) between the sound dource and receiver point is calculated as Dd ( (5 + 1)2 (4) r =D tan 0. So that it may be shown in the figure 2 (b). When it reaches a sound source, a distance from the accurate receiving point to the sound source is d(m). But, when d is big enough, it can think with dud'. The weight of the strengthen is defined by. Moreover, the wall side which reflects it from propagation route is specified. The rate of decrease of the energy by acoustic absorptivity of each sound beam is calculated by the acoustic absorptivity coefficient based on the skin material of the wall side. Performance in the receiving point is composed by computing these about each sound beam which reached a sound source from the receiving point separately and adding 328

Page  00000329 the performance of all the sound beam and putting it together. The wavefile is used as the sound source by this research. When the sampling frequency of wavefile is F(Hz), the data number n for the time s(sec) which begins the arrival of the sound is calculated as. Analyzed frequency is expressed by the change of the escalating color from blue to red. Green g(f) is defined to express a change in the frequency as a change in the color. The green g(f)(0<g(f) < 766) is calculated for the frequency f(Hz) as =Fx s (5) log2 +1 (f) fbas- x 512 log2 f + I fba~ (8). s(sec) is late by delaying sound data in a position of a sound source in the n individual, and the sound which arrives can be made. The time when the arrival of the sound takes it by propagation distance d'(m) of each sound beam is calculated. When data on the sound which delayed data on that time duration xi(T) and the number of sound beams which reached a sound source is made m, the data X(7) of the sound composed of the receiving point is calculated as m X(T) = xZi() x Wi x Ddi x (1- ai) (6) i=1 where wi is the weight calculated from the propagation distance d'(m) and the base distance D(m), Ddi is the decrement of the sound by the distance and alphai is the acoustic absorptivity of the wall. 3 Time Frequency Analysis and Coloring the frequency It is colored it to the main frequency of sound to hear in each position of the hall to visualize a sound environment at the hall. At this time, you must make a color to assign to each frequency as much as possible easy to understand. It passes through yellow-green from blue, and the frequency of the sound is made to cope with a change in an optical spectrum to change to red by this research. It is the visualization which is intuitively easy for to make it cope with an optical spectrum to know. It turns red in the low frequency in blue, the frequency in the middle in yellow-green, the high frequency. A color to use for the gradation creates a change in the color which the color whose degree of chroma was as high as possible was chosen than and which got clear. Performance in each position of the hall is composed by the method expressed with a preceding paragraph. Frequency in each time is analyzed from the composed performance. Fourier transform is used for the frequency analysis. The frequency which has the biggest amplitude by Fourier change is examined. The frequency which has the biggest amplitude is considered the frequency which comes to take the greater part. When a sampling frequency is F(Hz), when Fourier transform is calculated by the number of data of the N individual, where fmax = 3000(Hz) and fbace = 55(Hz). 3000(Hz) made the upper limit to color it it catches the range that it becomes the tonic of the sound easily. When 440(Hz) is the standard sound of A, 55(Hz) is the sound of A under three octaves. It is colored it by using green g(f) with a 24bit color to the frequency f(Hz). RedR(g(f))(0 < R(g(f)) < 256), green G(g(f))(0 < G(g(f)) < 256) and blue B(g(f))(0 < B(g(f)) < 256) and are calculated as R(g(f)) G(g(f)) B(g(f)) 0, g(f) < 255 g(f) - 255, 255<g(f) < 510 255, 510<g(f) g(f), g(f) < 255 255, 255<g(f) < 510 (9) 765-g(f), 510 < g(f) 255, g(f) < 255 510 - g(f), 255<g(f) < 510 0, 510 <g(f). Dominant sound to hear in each observation point inside the hall during the performance, and relations cause the distribution of the deep main frequency with the sound environment. Frequency is analyzed in each time in each position of the hall. And, a sound environment during the performance is visualized by assigning the color corresponding to the frequency. 4 Visualizing Sound Environment in the Hall during Performance At first the flow of the visualization is expressed in the following. * 1. Propagation route of the sound is found by using the sound beam tracing in each receiving point inside the hall. * 2. Performance in each receiving point is composed from propagation route of the sound. * 3. A time frequency analysis is given by using the composed performance. * 4. The space is colored it by the frequency which was analyzed in each time and which comes to take the greater part, and a sound environment during the performance is visualized. SN fi = fx F (7) 329

Page  00000330 rTm 7n 12m y::::::~ ~% Ifi~3r ~iiiiiii~llilx X Figure 5: Hall perspective. `x 2kI 4k9 125 I ~5O 1 500 1fki -~ --L _~ --~ --~ I: Figure 3: Hall ground plan. wail ceiling 026 0~-Q.14 0.-09$ 061 a_5809Dl~, St~ O.~-:1 O.14 VIQ~" 10$.08 jQO.BjO-$O7 z 5ni Figure 6: The main acoustic absorptivity hall wall side. coefficients in the 12m 1 2m~ Figure 4: Hall elevation. A hall to visualize a sound environment is made a multipurpose hall inside the university big academic meeting palace. Figure 3, figure 4 and figure 5 show a hall ground plan, an elevation and a perspective respectively. The main acoustic absorptivity coefficients are shown in the table 6. Each sound source which became the player who played was arranged as the figure 7 on the stage to the position of 1(m). A played tune is the Kanon (Pachelbel 1969) of Pachelbel composition. The first, the 2nd, and the 3rd violin and the cello are played. Music of the 12th bar that four all of the members joined in the figure 8 for the performance is shown as. Performance is done in the speed to play sixty quarter notes in one minute, and wavefile played in this tempo in each instrument is used for the sound source. The sound environment is measured in 13 points, 23 points and 7 points for the x, y and z direction. The total of measuring points is 2093 points. To compose performance in each receiving point from the reflection propagation route of the sound by the sound beam tracing. The standard distance D which the size of the sound source is the length of the diagonal of the hall as ( J142 + 242 + 2(m) where a place solid corner interval of a sound beam is defined as 0 10I. A sound environment from the performance start in every 1.0 seconds for 4 seconds is shown in 11 from the figure9. The time from [1] to [5] in the figures 11 and 9 corresponds to the position from [1] to [5] in the music figure 8. Figure 9 is a Z-section in figure 4, figure 10 is a X section in figure 3 and figure 1 s a Y section of figures 3 and 4. Which section is seen, it is understood that there are many blues which low frequency is shown in in [2] when the progress of the time was followed and these figures were seen. It knows the whole part to take out the sound of the middle sound and under when this position is confirmed by music of the figure 8. There are much yellow and orange in [4] [5], and the first the 2nd violin can be confirmed to take out high sound from the music. Next, from the figure 9 and 1 1, it is understood that a color changes in the right half of the hall and the left half. The violin which is a high music vessel in the stage in the left, the cello of the deeply toned instrument is arranged to the right, and such a left-right difference comes out as that result. A violin can be specially confirmed in [3] [4] when an upper sound is being played when it tries to see an influence by the height from the floor side in figure 10 and figure 11 that red and orange are concentrated around the ceiling. A figure 12 visualizes a sound environment in 330

Page  00000331 stage audience Figure 7: The arrangement of the player. Lii [2 1] 1:j [41 L[5 1 6]I'll 1M [81 Figure 8: Music score. each receiving point. Time analyzes the performance of each sound beam which it could get based on the sound beam tracing, and that an incident angle is being colored it. B is in the position of [e] from [a] of figures 3 and 4. Much sound reaches it from the back as well when [c] which is in the center of the hall is seen. Though it knows a thing, there is much front very much, and there is a little reflection sound from the back with [a] which is close to the stage in comparison with [c]. Generally there are a few sound beams with [e] behind the hall. The left-right balance of a sound beam to reach for wall side is bad with [b] and [e]. By the figure 13, in one cross line of Z-section and a Y-section in the figure 4, the state that a sound environment changes due to a change in time is shown as one sheet of figure. A figure shows change in time from 0 second until 8 seconds change toward the bottom from the top in every 0.25 seconds. Yellow which blue which low frequency is shown in shows high frequency with [5] in from [4] is shown even in this figure with [3] from [2]. And, dark u c w bluecan e coFigured t 8:]Msifromr[7. Figure 9: The sound environment is visualized during the performance in the hall plane (Z-section) according to the music score of figure 8. 5 Conclusions The following points were realized by composing performance by using the sound beam tracing and analyzing it for the whole inside the hall. A sound environment to change in accordance with the flow of the performance is visualized in all positions inside the hall. * What kind of sound is to see whether it reaches it from which direction during the performance in the receiving point of the sound. * It can cope with a change in the hall structure and the change of the program and the change of the instrument organization easily. By making it have pointing in the arrival direction of the sound in the receiving point. * The performance which it actually sounds like in that position in both ears is that it can be composed, too. It can be useful for the following technology from these. * 1. A sound environment can be examined at the hall designed newly in advance. * 2. A difference in the sound environment of the hall by the change of the tune is seen at the existent hall, even if it doesn't actually play, player's arrangement. * 3. The order of the seat in each position of the hall can be made clear from the number of sound beams and that propagation route. 331

Page  00000332 Figure 10: The sound environment is visualized during the performance in the hall elevation surface (X section) according to the music score of figure 8. performance in the hall elevation,~~\ surface (X section) accord-~ii'i' ing to the muic score of igure 8.':::::si [41J Figure 12: The sound environment is visualized in the receiving point shown in figure 3 and 4. The receiving point of sound is the center of the ball. Figure 11: The sound environment is visualized during the performance in the hall elevation surface (Y section) according to the music score of figure 8. It is mainly effective for the persons concerned in building corresponds in 1, for the player in 2, and for a hall administrator in 3. References Beranek, L. (1996). Concert Halls and Opera Houses. SpringerVerlag. M. Monks, B. M. Oh, J. D. (2000). Audioptimization goal-based acoustic design. IEEE Computer Graphics and Applications, pp. 76-91. N. Tsingos, T. Funkhouser, A. N. and I. Carlbom (2001). Modeling acoustics in virtual environments using the uniform theory of diffraction. In proc. ofSIGGRAPH2001, pp. 545-552. Pachelbel, J. (1969). Kanon fur drei Violinen und Basso continuo oder Streichorchester. B. Schott's Sohne Mainz. [II [2l --- 0.0 sec [51 gi i i 1 8.0 sec Figure 13: The sound environment in time of every 0.25 seconds is shown according to the music score of figure 8. 332