Page  00000001 SURVIVING ON PLANET CCRMA, TWO YEARS LATER AND STILL ALIVE Fernando Lopez-Lezcano, nando@ccrma.stanford.edu CCRMA Stanford University http://ccrma.stanford.edu/planetccrma/software/ ABSTRACT Planet CCRMA at Home [2] is a collection of packages that you can add to a computer running RedHat 9 or Fedora Core 1, 2 or 3 to transform it into an audio workstation with a low-latency kernel, current ALSA audio drivers and a nice set of music, midi, audio and video applications. It is also now a standalone distribution based on Fedora Core 3 which you can install by itself (without a previously running Fedora Core install). This presentation will outline the changes that have happened in the Planet over the past two years, focusing on the evolution of the linux kernel that is part of Planet CCRMA. 1. INTRODUCTION Creating worlds is not an easy task, and Planet CCRMA is no exception. The last two years have seen a phenomenal expansion of the project. The history of it will reflect, I hope, part of the recent history of Linux Audio projects and kernel patching. 2. A BIT OF HISTORY For those of you that are not familiar with Planet CCRMA [2] a bit of history is in order. At CCRMA (the Center for Computer Research in Music and Acoustics at Stanford University) we have been using Linux as a platform for research and music production since the end of 1996 or so. Besides the distribution I installed at the time, I started building and installing custom music software in our main server (we were dual booting PCs between Linux and NEXTSTEP, which was at the time the main computing platform at CCRMA). Sound support on Linux in 1997 was a bit primitive, not many sound cards were supported, and very few existed that had decent sound quality at all. Low latency was not a concern as just getting reliable sound output at all times was a challenge. Eventually the sound support in Linux evolved and patches became available for the kernel that enabled it to start working at low latencies suitable for reliable realtime audio work, so I started building custom monolithic kernels that incorporated those patches and all the drivers I needed for the hardware we used (easier than trying to learn the details of loadable kernel modules:-). At some point some adventurous CCRMA users started to install and try Linux in their home machines, and wanted an easy way to install the custom software available in all CCRMA workstations. I was installing RedHat so I started to use RPM (the RedHat Package Manager) to package a few key applications that were used in teaching and research (for example the Snd sound editor, the CM - CLM - CMN Common Lisp based composition and synthesis environment, Pd and so on and so forth). At first I just stored those packages in a network accessible directory and told potential users, "there you are, copy the packages from that directory and install them in your machine". A simple Web site with links to the packages was the next step, and installation instructions were added as I got feedback from users. Finally the project was made "public" with an email announcing it in the Cmdist mailing list - a list for users of Snd and CM/CLM/CMN. The announcement happened on September 14th 2001. This changed the nature of the project. As more people outside of CCRMA started using the packages I started to get requests for packaging music software that I would not have thought of installing at CCRMA. The number of packages started to grow and this growth benefited both CCRMAlites and external Planet CCRMA users alike. So Planet CCRMA was never an "official" project, it was just a side effect of me packaging software to install at CCRMA. The need for easier installation and upgrades grew and at the beginning of 2002 apt for rpm (a port of the Debian apt tool by Conectiva) was incorporated into Planet CCRMA, and used for all package installation and management. For the first time Planet CCRMA was reasonably easy to install by mere mortals (oh well, mere geek mortals). Fast forward to today: there are more than 600 individual packages spanning many open source projects in each of the supported branches of RedHat/Fedora Core. You can follow the external manifestation of these changes over time by reading the online ChangeLog that I have maintained as part of the project (a boring read, to say the least).

Page  00000002 3. AT THE CORE OF THE PLANET Since the announcement of the project outside CCRMA on September 2001, the base distribution on which it was based (RedHat) has seen significant changes. In July 2003 RedHat stopped releasing consumer products (the last RedHat consumer version was 9, released on March 2003). The Fedora Project was created, with the aim of being a community driven distribution with a fast release cycle that would also serve as a testbed for new technologies for the Enterprise line of RedHat products. Fedora Core 1 was the first release, followed by Fedora Core 2 and 3, at approximately 6 month intervals. In particular, Fedora Core 2 (released on May 2004) saw the introduction of the 2.6 kernel, which created a big problem for rebuilding the Planet CCRMA package collection on top of it. The problem was: the stock kernel was not good enough for low latency operation and there were no additional patches to fix it. 4. THE KERNELS Up to Fedora Core 1 the base distribution used a 2.4 kernel, and Planet CCRMA provided custom kernel packages patched with the well known low latency (by A. Morton) [6] and preemptible kernel (by R. Love) [5] patches (the last originally created by Monta Vista [4]), in addition to the tiny capabilities patch that enabled non-root users to run applications such as the Jack Audio Connection Kit server [15] with realtime scheduling. Fedora Core 2 changed the equation with the introduction of the 2.6 kernel. Running a 2.4 kernel on top of the basic distribution presented enough (small) compatibility problems that I discarded the idea very early in my testing cycle. And 2.6 had very poor latency behavior, at least in my tests. For the first 2.6 kernels I tested (March 2004) I used a few additional patches by Takashi Iwai [7] that solved some of the worst latency problems. But the results were not very usable. Ingo Molnar and Andrew Morton again attacked the problem and a very good solution evolved that is now available and widely used. Ingo started writing a series of patches for realtime preemption of the 2.6 kernel [8] (named at the beginning the "voluntary preemption" patchset). This set of patches evolved on top of the "mm" patches by Andrew Morton [9], the current equivalent of the old unstable kernel series (experimental kernel features first appear in the "mm" patches and then slowly migrate - the successful ones, that is - to the official release candidates and finally to the stable releases of the Linux kernel). Ingo did very aggressive things in his patches and the voluntary preemption patches (later renamed realtime preemption patches) were not the most stable thing to run in your computer, if it booted at all. I finally released a preliminary set of kernel packages on December 24 2004, using version 0.7.33-04 of Ingo's patches, one of the first releases that managed to boot in all my test machines:-) What proved out to be interesting and effective in Ingo's patches gradually percolated to the not so bleeding edge "mm" patches by Andrew Morton, and bits and pieces of "mm" gradually made it upstream to the kernel release candidates and then to the stable kernel tree. Eventually the realtime preempt patch itself started to be released on top of the official release candidates of the linux kernel and distanced itself from the more bleeding edge "mm" patchset. So, little by little the latency performace of the stock kernel improved. By the time of the release of 2.6.10 (December 24 2004) it was pretty good, although perhaps not as good as a fully patched 2.4 kernel. But keep in mind that this is the stock kernel with no additional patches, so the situation in that respect is much much better than it was in the old stock 2.4 kernel. The end result of this quest for the perfect kernel for Planet CCRMA dwellers at the time of this writing are two sets of kernels, currently available on both Fedora Core 2 and 3. 4.1. The "stable" kernel The current version is 2.6.10-2.1.11 (quite old by now). 2.6.10 turned out to be an unexpected (at least by me) milestone in terms of good low latency behavior. Finally, a stock kernel that has good low latency performance, out of the box. I would say it is close to what a fully patched 2.4 kernel could do before. 4.2. The "edge" kernel Currently 2.6.11-0.16.rdt based on Ingo Molnar's realtime preempt patch version 0.7.47-17. This is a more bleeding edge kernel, with significantly better low latency performance and based on Ingo Molnar's realtime preemption patches. The downside of trying to run this kernel is that it still (at the time of this writing) does not work perfectly in all hardware configurations. But when it works, it works very well, and users have reported good performance with no xruns running with two buffers of 64 or even 32 samples! I'm still being a bit conservative in how I configure and build this kernel, as I'm not currently using the REALTIME_RT configuration option, but rather the REALTIME_DESKTOP option (thus the "rdt" in the release). The penalty in low latency behavior is worth the extra stability (at this time). 4.3. Small details that matter But a kernel with good low latency is not nearly enough. You have to be able to run, for example, Jack, from a normal non-root account. Enter Jack O'Quinn [10] and Torben Hohn. Their efforts created a kernel module, part of the kernel security infrastructure, that enabled appli cations run sgid to a certain group, or run by users belonging to a group, or run by any user (all of this configurable, even at runtime), to have access to realtime priv

Page  00000003 ileges without having to be root. This is more restrictive and secure than the old capabilities patch, and at the time of this paper's initial submission, and after a very long discussion in the Linux Kernel mailing list (see [11I] and [12]), had been incorporated into the "mm" kernel patches. We all hoped it would eventually make its way to the stock kernel. But kernel gurus were not happy with the whole approach of the realtime Ism kernel module (although it was used successfully in Planet CCRMA from the very start of the 2.6.x kernel releases). Fortunately two alternative approaches appeared. One by Ingo Molnar called the rtlimits patch, in which the limits mechanism of the kernel is used to allow users to have access to realtime scheduling. This was more acceptable to the kernel gurus but had the disadvantage that it needed a patch to PAM (Pluggable Authentication Modules) infrastructure, which is a core package (that is, if the distribution used PAM, of course). The second alternative approach was a set of new scheduler classes by Con Kolivas, a potentially better approach but a much more intrusive patch from what I understand, which will relegate its inclusion or the consideration of its inclusion to the 2.6.13+ timeframe. To make a long story short the rtlimit patch also made it to the "mm" series patch and was eventually included in the linux kernel release candidates. So, regretfully, the early approach of the realtime Ism kernel module is now deprecated. But the good news is that now a stock kernel (2.6.12, just released at the time of this writing) has decent low latency performance (although not as good as with the realtime preempt patch) and a way to access realtime scheduling and memory locking from a normal non-root account through PAM. It was a tough sell advocating any of these options in the Linux Kernel mailing list, many thanks to Jack O'Quinn, Lee Revell and others for leading that effort and to Ingo Molnar and Con Kolivas for proposing more workable alternatives. Planet CCRMA is using (as of May 10 2005) the rtlimit patch and a patched PAM package for access to realtime scheduling and memory locking by mere mortals (ie: nonroot users). But this was not enough for a Planet CCRMA release. Ingo Molnar's realtime preemption patch changed the behavior of interrupt requests, the lower half of the interrupt processes are now individual processes with their own scheduling class and priorities, and a part of tuning a system for good low latency behavior is to give them, and Jack itself, the proper realtime priorities so that the soundcard and its associated processes have more priority than other processes and peripherals. I was trying to find a solution to this that did not involve users looking around /proc and tuning things by hand, when Rui Nuno Capela sent me a neat startup service script called rtirq that does just that, it sorts all interrupt service routines and assigns them decent priorities. All of this together makes it possible to package an easy to install turn-key solution to a low latency 2.6 based kernel. 4.4. The core packages The end result in Planet CCRMA are two sets of meta packages that reduce the installation and configuration of a 2.6 kernel to two apt-get invocations (installing planetccrmacore for the safer kernel and planetccrma-core-edge for the more risky one that offers better low latency performance). This, coupled to the fact that due to 2.6 both Fedora Core 2 and 3 use ALSA by default, made installing Planet CCRMA is a much easier process when compared to Fedora Core 1 or RedHat 9 and their 2.4 kernels. 5. OTHER WORLDS Planet CCRMA is one of many package repositories for the RPM based RedHat / Fedora family of distributions. Freshrpms [13], Dag [14], Atrpms, Dries and many others constitute a galaxy of web sites that provide easy to install software. Planet CCRMA is in the process of integrating with several of them (the so called RpmForge project) with the goal of being able to share spec files (the building blocks of RPM packages) between repositories. That will make my work, and that of the other packagers, easier, will reduce the inevitable redundancy of separate projects and will increase compatibility between repositories. Another world in which I also want to integrate parts of Planet CCRMA is the Fedora Extras repository. This Fedora sponsored project will be a centralized and more official repository of packages, but exclusively dedicated to augmenting the latest Fedora Core release (as opposed to the more distribution agnostic RpmForge project). With the availability of Fedora Extras the "community" part of the Fedora Project is finally arriving. 6. PLANET FORGE At the beginning of 2005 I finally got all the remaining components, and finished building a new server here at CCRMA completely dedicated to the Planet CCRMA project. The original goal was to create a fast build machine in which to queue packages to be rebuilt, as that process was fast becoming one of my productivity bottlenecks in maintaining Planet CCRMA. A secondary goal is to try to create a collaborative environment in which more people could participate in the development and maintenance of Planet CCRMA packages and associated documentation. We'll see what the future brings. 7. PLANET CCRMA 3, THE DISTRIBUTION One of the many things that are requested from time to time in the Planet CCRMA lists is the mythical "single media install" of Planet CCRMA (ie: "do I have to download all these cdroms?"). Up to a couple months ago (and completely on purpose), a potential user of Planet CCRMA has to first install Fedora Core, and then add the kernel, drivers and packages that make up Planet CCRMA

Page  00000004 (this additional installation and configuration work has been substantially reduced in Fedora Core 2 and 3 releases as they use ALSA by default instead of OSS). While this is not that hard, specially with the help of meta packages and apt-get or synaptic, it appears that sometimes it is too much work:-) And I have to agree, it would be much nicer to have a single cd (hmm, actually a dvd given the size of current distributions) and at the end of the install have everything ready to go, low latency kernel active, just start the applications and make some music. I had long avoided going down this road and becoming a "distribution" because of the additional work that would involve. It is hard enough trying to keep up to date with the very fast evolution of Linux audio software. But on and off I've been thinking about this idea, and since November 2004 I've been actually doing something about it. At the time of this writing (mid June 2005) I already have a single "Planet CCRMA 3" dvd with everything in it, all of Fedora Core 3 including current updates plus all of Planet CCRMA. This dvd is not small, about 2.7G of stuff, but remember, all of Fedora Core is included! I have also created a split distribution on 4 cdroms for those that do not have access to dvdrom drives. Installing Planet CCRMA from these images entails booting into the dvd or cdrom, selecting the Planet CCRMA desktop or workstation (which includes all development tools) installation target, customizing the packages installed if desired and pressing "Install" (while going through the normal installation choices of a stock Fedora Core system install, of course). One reboot and a few more configuration screens and you are up and running. Furthermore, the dvd and cdrom creation process is pretty much automatic at this point (start a series of scripts, wait for some time and out comes a dvd iso image and the split cdroms). Of course things are not that easy. What kernel should I select for installation? The more stable or the more risky that has better latency performance? How will the idiosyncracies of this non-standard kernel interact with the Fedora Core install process? (for example, it may happen that it will fail to boot in some machines, while the original Fedora Core kernel would have succeeded - and I don't think Anaconda, the RedHat installer, would be able to deal with more than one kernel at install time). For the current dvd and cdrom images I'm using the more bleeding edge low latency "rdt" kernel series and so far it seems to be working fine. 9. ACKNOWLEDGEMENTS The Planet CCRMA project would never have been possible without the support for GNU/Linux and Open Source at CCRMA, Stanford University, and in particular, the support of to Chris Chafe, CCRMA's Director. It goes without saying that I extend my heartfelt thanks to the hundreds of commited developers whose software projects I package. Without them Planet CCRMA would not exist and I would live in much more boring world. 10. REFERENCES [1] The Fedora Project. http://fedora.redhat.com/ [2] The Planet CCRMA Project. http://ccrma.stanford.edu/planetccrma/software/ [3] Ingo Molnar: Low latency patches for 2.2/2.4. http://people.redhat.com/mingo/lowlatencypatches/ [4] MontaVista: The Preemptible Kernel Patch. http://www.mvista.com/, see also http://www.linuxdevices.com/news/ NS7572420206.html [5] Robert Love: The Preemptible Kernel Patch. http://rlove.org/linux/ [6] Andrew Morton: Low latency patches for 2.4. http://www.zip.com.au/ akpm/linux/schedlat.html [7] Takashi Iwai: low latency tweaks. http://kerneltrap.org/node/view/2702 [8] Ingo Molnar: Realtime Preemption patches for 2.6. http://people.redhat.com/mingo/realtimepreempt/ [9] Andrew Morton: the "mm" patches for 2.6. http://kernel.org/pub/linux/kernel/people/ akpm/patches/2.6/ [10] Jack O'Quinn: the realtime Ism kernel module. http://sourceforge.net/projects/realtimeIsm/ [11] Linux Weekly News: Merging the realtime security module. http://lwn.net/Articles/118785/ [12] Weekly News: Low latency for Audio Applications. http://lwn.net/Articles/120797/ 8. CONCLUSION [13] Freshrpms: http://freshrpms.net/ package repository. It is easy to conclude that Planet CCRMA is very cool, and very significant for research and production environments where software freedom is important. Planet CCRMA as a project is alive and well. As a maintainer I'm (barely) alive, but have made it to another conference, no small feat. [14] Dag: package repository. http://dag.wieers.com/home-made/apt/ [15] The Jack Audio Connection Kit, a low latency sound server. http://jackit.sf.net/