Some notes on how to build a Linux cluster

"SUPERCOMPUTER: what it sounded like before you bought it."
On this page:

Cluster And Grid Computing

Linux based clusters are pretty much in fashion these days. The ability to use cheap off-the-shelf hardware together with open source software makes Linux an ideal platform for supercomputing. Linux clusters are either: "Beowulf" clusters, "MOSIX" clusters or "High-Availability" clusters. Cluster types are chosen depending on the application requirements which usually fall in one of the following categories: computational intensive (Beowulf, Mosix), I/O intensive (supercomputers) or high availability (failover, servers...). In addition to clusters, there are two related architectures: distributed systems (seti@home, folding@home, Condor...) that can run on computers with totally different architectures spread out over the Internet; and multiprocessor machines (those, like the SGIs are much superior to clusters when memory needs to be shared fast between processes).

New 2006: Although I don't work on clusters anymore, I try to follow what's going on. There are several recent developments like Starfish or Amazon EC2 (Elastic Compute Cloud). The first one is implemented in Ruby and only takes a few lines of code to get up and running. The latter runs a Linux virtual image remotely on as many servers as you need. There are many grid services available, but the economics of them is sometimes hard to fathom; Sun's service has been a failure for instance as it takes way too much work on the programmer's part.

High Availability

High Availability (HA) clusters are highly fault tolerant server systems where 100% up-times are required. In the event of failure of a node, the other nodes which form the cluster will take over the functionality of the failed node transparently. HA clusters are typically used for DNS server, proxy and web servers.

High-Availability cluster Links:

Good evening Mr Gates, I'll be your server today.

Beowulf clusters

Clusters are used where tremendous raw processing power is required like in simulations and rendering. Clusters use parallel computing to reduce the time required for a CPU intensive job. The workload is distributed among the nodes that make up the clusters and the instructions are executed in parallel. More nodes means faster execution and less time taken.

The difference between the two main cluster types, Mosix and Beowulf is that OpenMosix is a kernel implementation of process migration, whereas Beowulf is a programming model for parallel computation. Beowulf clusters need distributed application programming environments such as PVM (Parallel Virtual Machine) or MPI (Message Passing Interface). PVM is the standard interface for parallel computing. But MPI is becoming the industry standard. PVM still has an upper edge over MPI as there are more PVM aware applications when compared to MPI based ones. But this could soon change as MPI becomes popular.

Beowulf Links:

Mosix clusters

MOSIX is a software package that was specifically designed to enhance the Linux kernel with cluster computing capabilities. The core of MOSIX are adaptive (on-line) load-balancing, memory ushering and file I/O optimization algorithms that respond to variations in the use of the cluster resources, e.g., uneven load distribution or excessive disk swapping due to lack of free memory in one of the nodes. In such cases, MOSIX initiates process migration from one node to another, to balance the load, or to move a process to a node that has sufficient free memory or to reduce the number of remote file I/O operations. MOSIX clusters are typically used in data centers and data warehouses.

Mosix works best when running plenty of separate CPU intensive tasks. Shared memory is its big drawback, like Beowulf: for applications using shared memory, such as Web servers or database servers, there will not be any benefit from [Open]Mosix because all processes accessing said shared memory must resided on the same node.

MPI and openmosix play very nicely together, the processes that MPI spawns on remote nodes are migrated by the OpenMosix load balancing algorithm like any other process. So in a sense it's better in a dynamic environment where different MPI programs will be running that are unaware of each other and they could both try to overload any individual node.

Mosix Links:

Cluster Hardware

What's in a cluster ?
Commodity hardware components (usually PC), interconnected by a private high-speed network. The nodes run the jobs. It is sometimes connected to the outside world through only a single node, if at all. Overall cost may be a third to a tenth of the price of a traditional supercomputer.
What's the most important: CPU speed, memory speed, memory size, cache size, disk speed, disk size, or network bandwidth ? Should I use dual-CPU machines ? Should I use Alphas, PowerPCs, ARMs, x86s or Xeons ? Should I use Fast Ethernet, Gigabit Ethernet, Myrinet, SCI, FDDI, FiberChannel ? Should I use Ethernet switches or hubs ? ...
The short answer is: it all depends upon your application! Benchmark, profile, find the bottleneck, fix it, repeat.
Some people have reported that dual-CPU machines scale better than single-CPU machines because your computation can run uninterrupted on one CPU while the other CPU handles all the network interrupts.
Can I make a cluster out of different kinds of machines: single-processor, dual-processor, 2.0GHz, 2.4GHz, etc ?
Yes. Depending upon the application mix, the cluster can be configured to look like multiple clusters, each optimized for the specific application.
Is it possible to make a cluster of already used workstations ?
Yes, you can use your secretary's computer's unused CPU cycles. Install a cluster package like Mosix or Condor on it and add its IP to the overall cluster. This kind of architecture is called a COW (Cluster Of Workstations). The main drawbacks are security (any compromised machine compromises the entire cluster), unstable apps (if you crash a machine, part or even the entire cluster could be affected) and deciding who gets to run the CPU intensive jobs.
How do you connect the nodes ?
Lately, with the fashion of simple clusters of low-quality hardware, the most common type of interconnect has been the cheap Gb ethernet LAN. But there are other type that can be faster or with lower latency or both: Inifiniband, Myrinet, SCI, Quadrix... Those can be expensive though.
Hardware Links:

Hardware cost for a sample cluster (March 2003)

Here's an Excel spreadsheet of the hardware list and total cost involved in our cluster at CSU. About 24000$ (including shipping) for a 12 nodes, 24 processor cluster. Some of the key characteristics are: AMD Athlon MP 2400 with better heatsinks, Tyan Thunder K7X Pro motherboards (without SCSI), internal 2Tb RAID-5 IDE/ATA array, Gb LAN, 24Gb of RAM, Linux RedHat 8.0 with OpenMosix...

Software / distributed libraries

Where's the Beowulf software?
There isn't a software package called "Beowulf". There are, however, several pieces of software many people have found useful for building Beowulf clusters: MPI, LAM, PVM, Linux....
How much faster will my software run?
It's very application dependent. You'll need to split it into parallel tasks that communicate using MPI or PVM. Then you need to recompile it. Some "highly parallel" tasks can take advantage of running on multiple processors simultaneously, but then a Mosix cluster can do that more easily.
What are PVM and MPI?
PVM and MPI are libraries that allow you to write message-passing parallel programs in Fortran or C to run on a cluster. PVM used to be the defacto standard until MPI appeared. But PVM is still widely used and very good. MPI (Message Passing Interface) is a defacto standard for portable message-passing parallel programs standardized by the MPI Forum and available on all massively-parallel supercomputers.
Is there a compiler that will automatically parallelize my code for a Beowulf?
No. But there is BERT from which will help you manually parallelize your Fortran code. And NAG's and Portland Group's Fortran compilers can also build parallel versions of your Fortran code, given some hints from you (in the form of HPF and OpenMP directives). These versions may not run any faster than the non-parallel versions. The gnu gcc, Portland Group, Intel, Fujitsu and Absoft are the most commonly used compilers.

Other distributed OSs

With ROCKS, you get:
Mandrake 'Click'
The new Mandrake 'Click' Turn-Key Clustering Distribution (in beta). Pretty easy to install.
ClusterKnoppix and Quantian are Knoppix boot CDs with OpenMosix self-discovery daemons and network boot. It's an excellent way to have an instant cluster: put the CD in, boot the master, reboot the slaves and you are clustering !!! Includes mathematical software. You don't need hard drives to run it and it's very easy and quick to install. If you want to try out clustering, start with this first.

Distributed Numerical processing

Installing OpenMosix on RedHat

  1. Install RedHat
  2. Install openMosix rpms
  3. Install kernel source to /usr/src/linux and patch with openMosix-2.4.20-4.gz
  4. Compile and install openMosixUserland-0.2.4
  5. Configure /etc/
  6. Reboot new openMosix kernel
  7. setpe -W -f; edit /etc/ on each node
  8. Install libdshconfig and DSH
  9. Put the other nodes' hard drives as slaves in your just configured box, format it and remount your main drive as read-only (mount -o remount,ro /), dd if=/dev/sda of=/dev/sdb. Mount the new disk to update a few files (like /etc/sysconfig/network-scripts/ifcfg-eth0).

Using OpenMosix

What do you need to know to use OpenMosix ?
In short: nothing. Log in the Master node, compile your programs (with optional AMD Athlon optimization), run them, they will migrate to an available node. You never need to login into the slave nodes.
In long: it depends if the task is CPU intensive or I/O intensive. Read on...
CPU intensive tasks
Those are perfect for Mosix and should migrate well (there are some exceptions, like Java applications) and make the cluster feel like an SMP machine. You can use pmake or make -j 24 to compile in parallel.
I/O intensive tasks
Use mosmon to find a node (N) that doesn't have much running, copy your data to its disk "/mfs/N/dir" and just launch your code, it will move to that node in order to minimize I/O.
The Mosix Filesystem (MFS)
All disks are shared between nodes in the following way: /mfs/Node#/Dir. In addition you can refer to process dependent directories: /mfs/here (the process' current node), /mfs/home (the process' home node).
The Distributed Shell (DSH)
DSH allows you to run the same command on all the nodes: dsh -r ssh "uname -a"
More Mosix control
the following commands give you info on running processes, or some control on how they are migrated by OpenMosix: migrate, mon, mosctl, mosrun, setpe, tune, or the graphical programs openMosixview, openMosixprocs, openMosixcollector, openMosixanalyser, openMosixhistory, openMosixmigmon... Use man.
More info ?
Read the Mosix HowTo (only 99 pages).

Possible hostnames for a cluster

Lists kindly generated by Google sets.

Northern mythology
Accalia, Aegir, Aesir, Aldwin, Asgard, Avalon, Balder, Baldric, Beowulf, Bragi, Brahma, Dolvar, Drexel, Fenrir, Forseti, Frey, Freya, Freyr, Frigg, Fulla, Grendel, Heimdall, Hel, Hermod, Hildreth, Hod, Hofstra, Hrothgar, Hyglac, Hyoga, Hyster, Idun, Jord, Loki, Megalon, Midgard, Naegling, Njord, Norns, Odin, Ragnarok, Ran, Rhung, Sif, Skadi, Snotra, Thor, Tyr, Ull, Unferth, Valhalla, Valkyrie, Vidar, Vor, Wiglaf, Yggdrasil...
More technical
criticalregion, dde, deadlock, distributed, globallock, messagepassing, multiprocess, mutex, namedpipe, parallel, rendezvous, rpc, semaphore, setiathome, sharedmemory, thread...
Anise, Basil, Chamomile, Chive, Coriander, Dill, Fennel, Hyssop, laurel, Lavender, Mint, Oregano, Parsley, Rosemary, Tarragon, Thyme...

IP Choice

Private IP Networks
RFC 1918 reserved three IP network addresses for private networks. The addresses, and can be used for anyone for setting up their own internal IP networks.
Class 'E' Addresses
The Class 'E' IP address is reserved for future use. In Class 'E' addresses, the 4 highest-order bits are set to 1, 1, 1, and 1 (in practice through Routers and Firewalls currently ignore class 'E' IP addresses so you can use them between the machines, and have the master node also have a normal IP as the only way in the cluster.

Cluster Books

Except for the first one, most of those books are either outdated, very vague, too specific (only source code) or with bad reviews. You've been warned.

And a little wallpaper as a gift:

A cluster of penguins...

Welcome to the Neumann cluster


There is a bug in mfs that causes some recursive commands to fail painfully. For instance the following commands will cause the node to hang: find /, du -ks /... In other words, do not lauch recursive commands from / (the root directory) or /mfs. A workaround is to use find / \! -fstype mfs or locate. Note: that problem may be solved now (by unmounting a shared memory module), but I don't want to risk it...

rpm -Uvh --oldpackage fileutils-4.1-10.1.i386.rpm to avoid bug in cp and move returning "cp: skipping file `foo', as it was replaced while being copied".


Neumann is an x86 Linux cluster running OpenMosix. There are twelve nodes in the cluster, each node having two AMD Athlon MP 2400+ CPUs, 2Gb of RAM and 60Gb of disk space. The main node, on which you login, also has a 2Tb RAID array. A 1Gbps ethernet switch interconnects the nodes, and only the master node is connected to the outside via a 100Mbps LAN. Each CPU has 3 floating point units able to work simultaneously.

Each node runs Red Hat Linux 8.0, fully patched, with a customized kernel 2.4.20. OpenMosix is a kernel modification that makes a group of computers act like a multiprocessor machine (SMP), so the simplest way to use Neumann is to just ignore it, it will run your CPU intensive jobs on whatever node has spare CPU.

Working with OpenMosix

In addition to the current page, you might want to take a look at the following information:

The simplest way to use OpenMosix is to just ignore it. Just compile and run your CPU intensive jobs on the master node, and they will be dispatched automagically to whatever node has spare CPU cycles. And when a node becomes overloaded, processes will migrate to a more idle node.


One very useful part of OpenMosix if the MFS (Mosix FileSystem). You can access any drive on any node from any node using a path like /mfs/3/usr/src/. Just prepend /mfs/NodeNum to the normal path. Note that you don't need to do this for your home directory as it's already linked to /mfs/1/home/username. Note that when doing I/O from a local disk (like /), the processes need to migrate back so they can read the proper drive (/ on node 1, not / on node N); if on the other hand you use mfs, processes can read without migrating. So Command </mfs/1/dir/InputFile >/mfs/1/dir/OutputFile is much preferable to Command <InputFile >OutputFile Remember this syntax when hard-coding pathnames in your programs, it is the simplest and best way to optimize your programs for OpenMosix.

In addition to /mfs/NodeNum/, there are virtual directories accessible on a 'per process' base such as /mfs/home or /mfs/here for quick access. Read the HowTo for more.

Mosix Commands

If you want more in depth control, here are a few tools, both X-windows and command line:

X-windows Command line
Display the state of the various nodes and their load (the load includes both CPU and outstanding I/O).
Display the various jobs running on the various nodes. The green squares are jobs that have been migrated away from the master node. You can click on them and drag them onto another 'penguin' to migrate them manually. Or issue migrate PID node# from the command line. I would advise you to run your large jobs with mosrun -j2-12 -F job and then move them around manually with openmosixmigmon. In other words, try not to run your big jobs on the master node (neumann, 1) as it's already busy handling network I/O, interactive users and more.
Process box (similar to to Linux system monitor). Double click a process to be able to change its settings (nice, locked, migration...)
Process history for the cluster
Similar to xload for the entire cluster
use this to start a job in a specific way, for instance mosrun -j2-12 -F job will start a job on one of the slave nodes and it will not migrate automatically after that. mosrun -c job for CPU intensive jobs, mosrun -i job for I/O intensive... man mosrun for more. Various of the mosrun options have command line aliases, for instance nomig to lauch a process not allowed to migrate.
migrate PID [Node]
Manually migrate a process to another node. Note that if the process is not node-locked it may change node later.
mps and mtop
OpenMosix aware replacements for the ps and top utilities
Controls the way a node behaves in the cluster (if it's willing to run remote jobs and more...) man mosctl for more.

Advanced clusterization

I you write shell scripts, you can use the power of OpenMosix with the simple '&' to fork jobs to other nodes. In C, use the fork() function; there is a code sample chapter 12 of the OpenMosix HowTo. Also see the HowTo file for more information on Python, Perl, Blast, Pvm, MPICH... Note that Java doesn't migrate.

OpenMosix is designed to allow you to run multiple jobs simultaneously in a simple way. You cannot run a single job (even multithreaded) on more than one processor. If you need to do something like that, you need to use the MPI (Message Passing) library. MPICH is the implementation of MPI which is compatible with OpenMosix, version 1.2.5 is installed in /usr/local/mpich for use with the PGI compilers (not the GNU compilers). There's also an MPI library coming with RedHat Linux, but there's no warranty about its OpenMosix compatibility, so make sure you are using the proper one. Also read this short page about MPICH on OpenMosix (it tells you to use a MOSRUN_ARGS environment variable). A more recent discussion about Looking for Portable MPI I/O Implementation.

If you need checkpoint and restart facility for extremely time consuming problems (the cluster can reboot and the computation continues), you can ask for Condor to be installed on top of OpenMosix.

You normally do not need to log onto the slave nodes. But it is possible. You just need to type ssh neumann2 (3..12). Note that you can use the usual ssh-keygen -t dsa, cat ~/.ssh/*.pub >>~/.ssh/authorized_keys, ssh-agent $SHELL and ssh-add to define a (possibly empty) passphrase so that you won't have to enter your password each time. Note that you can access neumann# only from neumann, '#' being the node number (2 to 12). See below.

DSH, The Distributed Shell

DSH allows you to run a command on all the nodes of the cluster simultaneously. For instance:

dsh -a reboot
Reboot the entire (-a) cluster.
dsh -c -g slaves "shutdown -h now"
Shut down gracefully all the slave nodes.
dsh -a df
Execute df on every node, in order (option -c executes concurrently)
dsh -c -g custom /home/CustomJob
Run a custom job on the computers defined in the file ~/.dsh/group/custom, so that even if the master node crashes, they shouldn't be affected (but bear in mind that /home is a symlink to /mfs/1/home
dsh -a passwd
the password files need to be the same on all the nodes, so you need to change your password everywhere at once... Sorry, but you need to type it in 24 times !

Setting up password-less access (necessary for dsh).

Cluster administration


Several kernels are available to boot from, depending on use and issue to troubleshoot:

Optimized OpenMosix kernel. This is the default kernel. Note that this kernel is slightly different on the master than on the slaves (although named the same). Everything that wasn't strictly necessary has been removed from the slave kernels (no CD, no floppy, no SCSI, no enhanced video, no sound...) and almost all from the master kernel.
Optimized 2.4.20 kernel
Basic openmosix kernel
Vanilla 2.4.20 kernel, no customization.
Default multiprocessor RedHat 8.0 kernel
Basic RedHat 8.0 kernel, if all else fails.
A tarred copy of the slave kernel is kept in /boot on the master node. Note that when you make xconfig dep clean bzImage modules modules_install install, the last phase (install) cannot work if you try to recompile the kernel currently in use; reboot and use another kernel first.

Important note: if you recompile a kernel, you need to edit /boot/grub/grub.conf after running make install and change root=LABEL=/ to root=/dev/hda3 (master) or hda7 (slaves). If you get a kernel panic on boot, you probably forgot to do that.

There is a complete disk image of the neumann2 slave in the file Slave.img.gz. It was created by putting the slave drive in the master node and doing dd if=/dev/hdd | gzip >Slave.img.gz. You can create a new slave by putting a blank hard drive in the master (or any node) and doing zcat Slave.img.gz >/dev/hdd. It takes about 12 hours and OpenMosix should be disabled while you do that (/etc/init.d/openmosix stop). After completion, you need to change the node hostname and IP with /usr/bin/neat and also various settings that may have changed since the image file was created (last kernel...). Note: if you connect the drive as primary slave it is /dev/hdb, secondary master is /dev/hdc and secondary slave is /dev/hdd.

Boot sequence

OpenMosix administration

The user IDs must be the same on all the nodes. In other words /etc/passwd, /etc/shadow and /etc/group must be identical on all nodes. You can edit and run /root/ to add a user on all nodes.

Use /etc/init.d/openmosix {start|stop|status|restart} to control openmosix on the current node.

/etc/ contains the list of nodes and must be identical on all nodes. You want to keep it consistent with /etc/hosts. Autodiscovery is not used.

If a slave node crashes, the jobs that had been migrated to it are lost (and the cluster may hang for a while while it sort things out). If the master node crashes, all jobs are lost (except if they were started somewhere else). This is because a job that has been migrated away keeps some references/pointers/resources on its home node.

To keep the master node from running too many jobs, either migrate them manually with migrate or openmosixmigmon, or set its speed to some low value with mosctl setspeed 7000

Hard Drives


Partitions are slightly different on the master than on the slaves.


Note that there's no specific use for the 11 /spare partitions (totaling about 500Gb) present on the slaves. Here is my suggestion: if you have I/O intensive jobs, put your data on one of those drives (say /mfs/7/spare/) and then run your job in a similar way: mosrun -j7 -F -i "job </mfs/7/spare/LargeDataFile" so that the job stays close to its data and doesn't use the network. Remember to delete your data afterwards (or back it up on /raid).

RAID management

The RAID is based on a 3ware hardware controller model Escalade 7500-8 with 8 250Gb Maxtor hard drives. The RAID is configured as a RAID-5 where data parity is spread over all the drives. One drive may fail without causing any data loss. After a drive failure, the array must either be rebuilt on the remaining drives, or a blank drive may be inserted to replace the failed one prior to rebuilding. In case of drive failure, an email is sent to the admin.

Configuration is done both on boot in the 3ware utility (for basic configuration), or with cli or preferably with 3dm through a web browser. From neumann itself, type mozilla http://localhost:1080/ to manage the array.

U: drive and remote mounts

Some NFS mounts are available under /mnt/, for instance personal network files are mounted under /mnt/udrive/yourusername. Note that this is the Engineering network username, not the Neumann username, which may be different.

Installed software and services

Compilers and their options

The AMD Athlon MP processors are able to run native i86 code, but for full performance, you should use the special athlon compiling options (SSE and 3DNow!) whenever applicable in your compiler. One of the main difference between Intel and AMD is that the Athlon has 3 floating point units (FPUs) leading to a theorical performance of 2 double precision multiplications and 2 single precision additions per clock cycle. The actual performance is 3.4 FLOPs per clock, leading to a peak performance of 160 Gflops for the entire cluster.

For the GNU compilers are the default Linux (g77, gcc, gcj). Use g77 --target-help or gcc --target-help to get the relevant compiling options for optimizing your programs (add them to your makefile):

For the PGI compilers (pgcc, pgCC, pgdbg, pgf77, pgf90, pghpf...) you need the proper variables set ($PGI, $MANPATH...) in your ~/.profile or ~/.bash_profile before you can use them or use man. Note that PGI workstation is limited to 4 processors per thread and that you can compile only from node 1.

gcc-O3 -march=athlon-mpGNU compiler 3.2
pgcc-O2 -tp athlonxp -fastPortland group C compiler
pgCC-O2 -tp athlonxp -fastPortland group C++ compiler
bcc-3Bruce's C compiler
g77-O3 -march=athlon-mpG77 3.2
pgf77-O2 -tp athlonxp -fastPortland group fortran 77 compiler
pgf90-O2 -tp athlonxp -fastPortland group fortran 90 compiler
pghpf-O2 -tp athlonxp -fastPortland group High Performance Fortran compiler
Other languagesDescription
gcj?Ahead-of-time GNU compiler for the Java language
perl?The Perl Compiler
python?An interpreted, interactive, object-oriented programming language
pgdbg?The PGI X-windows degugger

Services running on Neumann (5 is the normal runlevel)

Active servicesDisabled services
xinetd based services:

Specific software

Type /mnt/fluent/bin/fluent to start the fluent application (remotely mounted from gohan). Also installed is xfig. I'm having trouble compiling Grace, a successor to xmgr.

If you need to work with MS Office documents, you can start OpenOffice with /usr/lib/openoffice/program/soffice or /usr/lib/openoffice/program/scalc or more in that directory...

The PGI Workstation suite of compilers are installed in /usr/pgi (see above).

Software packages installed on Neumann

Here's the short list below. For a complete list with version numbers and comment, see the complete RPM list with rpm -qa --info at the command line.

4Suite          gal             libf2c          netconfig       setuptool
a2ps            gawk            libgal19        netpbm          sgml
acl             gcc             libgcc          newt            sh
alchemist       gconf           libgcj          nfs             shadow
anacron         GConf           libghttp        nmap            slang
apmd            GConf2          libglade        nscd            slocate
ash             gd              libglade2       nss_ldap        slrn
aspell          gdb             libgnat         ntp             sox
at              gdbm            libgnome        ntsysv          specspo
atk             gdk             libgnomecanvas  oaf             splint
attr            gdm             libgnomeprint   octave          star
audiofile       gedit           libgnomeprint15 Omni            statserial
authconfig      gettext         libgnomeprintui omtest          strace
autoconf        gftp            libgnomeui      openjade        stunnel
autofs          ggv             libgtop2        openldap        sudo
automake        ghostscript     libIDL          openmosix       swig
automake14      gimp            libjpeg         openmosixview   switchdesk
automake15      glib            libmng          openmotif       sysklogd
basesystem      glib2           libmrproject    openoffice      syslinux
bash            glibc           libogg          openssh         SysVinit
bc              Glide3          libole2         openssl         talk
bdflush         glut            libpcap         orbit           tar
bind            gmp             libpng          ORBit           tcl
binutils        gnome           libpng10        ORBit2          tcpdump
bison           gnumeric        librpm404       pam             tcp_wrappers
bitmap          gnupg           librsvg2        pam_krb5        tcsh
blas            gnuplot         libstdc++       pam_smb         telnet
bonobo          gpg             libtermcap      pango           termcap
byacc           gphoto2         libtiff         parted          tetex
bzip2           gpm             libtool         passivetex      texinfo
cdecl           gqview          libungif        passwd          textutils
chkconfig       grep            libunicode      patch           time
chkfontpath     groff           libusb          patchutils      timeconfig
compat          grub            libuser         pax             tk
comps           gtk             libvorbis       pciutils        tmpwatch
control         gtk+            libwnck         pcre            traceroute
cpio            gtk2            libwvstreams    perl            transfig
cpp             gtkam           libxml          php             ttfprint
cracklib        gtkhtml         libxml2         pilot           tux
crontabs        gtkhtml2        libxslt         pine            umb
cups            guile           lilo            pinfo           units
curl            gzip            linc            pkgconfig       unix2dos
cvs             hdparm          linuxdoc        pmake           unzip
cyrus           hesiod          lockdev         pnm2ppa         up2date
db4             hotplug         logrotate       popt            urw
desktop         hpijs           logwatch        portmap         usbutils
dev             htmlview        lokkit          postfix         usermode
dev86           httpd           losetup         ppp             utempter
dhclient        hwbrowser       LPRng           procmail        util
dia             hwcrypto        lrzsz           procps          VFlib2
dialog          hwdata          lsof            psmisc          vim
diffstat        ImageMagick     ltrace          pspell          vixie
diffutils       imlib           lvm             psutils         vte
docbook         indent          lynx            pvm             w3c
dos2unix        indexhtml       m4              pygtk2          webalizer
dosfstools      info            magicdev        pyOpenSSL       wget
doxygen         initscripts     mailcap         python          which
dump            intltool        mailx           pyxf86config    whois
e2fsprogs       iproute         make            PyXML           WindowMaker
ed              iptables        MAKEDEV         qt              wireless
eel2            iputils         man             quota           words
eject           irda            maximum         raidtools       wvdial
emacs           isdn4k          mc              rcs             Xaw3d
eog             jadetex         memprof         rdate           xdelta
esound          jfsutils        metacity        rdist           xfig
ethereal        kbd             mingetty        readline        XFree86
ethtool         kbdconfig       minicom         redhat          Xft
evolution       kernel          mkbootdisk      reiserfs        xinetd
expat           krb5            mkinitrd        rhl             xinitrc
expect          krbafs          mktemp          rhn             xisdnload
fam             ksymoops        mod_perl        rhnlib          xml
fbset           kudzu           mod_python      rhpl            xmltex
fetchmail       lam             mod_ssl         rmt             xmlto
file            lapack          modutils        rootfiles       xpdf
filesystem      less            mount           rp              xsane
fileutils       lesstif         mouseconfig     rpm             xscreensaver
findutils       lftp            mozilla         rpm404          xsri
finger          lha             mpage           rsh             Xtest
firstboot       libacl          mrproject       rsync           yelp
flex            libaio          mt              samba           yp
fontconfig      libart_lgpl     mtools          sane            ypbind
foomatic        libattr         mtr             screen          zip
fortune         libbonobo       mutt            scrollkeeper    zlib
freetype        libbonoboui     nautilus        sed
ftp             libcap          ncurses         sendmail
gail            libcapplet0     nedit           setserial
gaim            libelf          net             setup