Network Booting

Forget about burning EEPROMs. The network boot capabilities in new PCs make this powerful technique easier to use than ever.

Network booting — often called “diskless booting” — can dramatically simplify some tough system administration headaches. In a computer lab, a classroom, an Internet cafe, a Beowulf-style compute cluster, or a render farm, network booting can centralize system configuration and management, making it easier and faster to maintain, repair, and upgrade those networks of machines. (See the sidebar “Network Booting Scenarios” for a few of the many uses for network booting.)

Using network booting, you can configure new machines just by plugging them in and turning them on; you can upgrade a raft of machines in a couple of hours; you can even eliminate the headaches caused by well-meaning user “customizations.” More than just a convenience, network booting is essential for any computing environment that has a large number of essentially-identical systems. And, recent developments are making network booting simpler and more accessible.

Network booting can’t install new SIMMs in a PC — you’ll want to hang on to your trusty screwdriver for that — but network booting can save you time, prevent frustration and hassles, and let you focus on more important tasks, like building that petaflop supercomputer cluster from spare 486s.

Let’s examine how network booting works, learn how to configure a Linux boot server, and set up a Linux machine to boot over the network. First, we’ll develop a working configuration where each boot client has its own separate set of directories on the server. Then, we’ll look at ways to improve the configuration by sharing critical directories among all clients.

Network Booting Scenarios

Network booting is really a collection of techniques that can be combined in many different ways. Once you understand the core principles involved, you’ll certainly see even more ways to put these ideas to use.

Clusters: Clusters consist of a lot of identical machines that require identical software configurations. Centralizing configuration for cluster nodes on a single network boot server greatly simplifies administration. Unlike diskless workstations, cluster nodes typically use a local hard disk for swap and critical files to boost performance and reduce network traffic.

Instant Classroom: For an instant classroom, configure a portable machine as a network boot server. When you arrive on-site, plug it into the classroom network, reboot each classroom machine, and immediately have the required setup. After the class, take away the server and leave the classroom machines in their original condition.

Secure Desktops: Machines in classrooms, computer labs, and Internet cafes all vulnerable to unauthorized changes to system configurations. By configuring the central server to export critical filesystems as read-only, you can easily limit what changes are possible on client machines. Moreover, any damage that does occur can be erased with a simple reboot.

Virtual Rescue Disk: On some newer PCs, you can initiate a network boot by pressing F12 at the BIOS screen. Even if a machine is otherwise unbootable, F12 can always access a known-good configuration from the network.

Give Disks the Network Boot

A typical Linux system boots by loading a series of increasingly powerful programs from disk into memory, culminating in the loading of the Linux kernel itself. With network booting, one or more of the programs loaded during the boot sequence is retrieved from a network server. In some cases, only the kernel or the root filesystem will be accessed over the network, with everything else residing on a local hard disk. The extreme case is the so-called “diskless workstation,” in which everything is accessed over the network, including the boot loader, kernel, root filesystem, and sometimes even swap.

To set up your own network-boot system, you’ll need one server to run the DHCP, TFTP, and NFS server software — if you have an existing DHCP server, you can use that instead — and at least one client machine that contains a PXE ROM, as described in the sidebar “Intel’s PXE.” If you don’t have a client machine with PXE, you can use Etherboot instead. (Network booting with Etherboot is slightly different than the PXE process. The sidebar “PXE-Free Booting” provides the details.)

Intel’s PXE

Older network booting systems used ROM code that accessed and loaded an operating-system kernel directly. The end result was a selection of network boot ROMs that were specific to particular operating systems.

To avoid this problem, Intel’s Preboot eXecution Environment (PXE) breaks the network boot code into two portions: a hardware-specific PXE ROM, and an OS-specific network boot program (NBP).

The PXE ROM provides a basic network API and a simple initial download capability. It is the network counterpart to the standard PC BIOS, which provides basic disk I/O and a rudimentary initial boot. The NBP corresponds to the traditional boot loader (such as LILO or GRUB); it uses the PXE API to locate, download, and execute an actual operating system.

PXE is now commonly included in the motherboard or NIC BIOS. If you’re trying to locate such capabilities, watch for MBA (Managed PC Boot Agent), which is the name of one popular implementation. MBA supports multiple boot conventions — make certain your MBA is configured to support PXE.

PXE capabilities are sometimes well hidden. As a concrete example, the Dell Dimension 8100 BIOS requires you to reboot three times to properly enable network booting. The first time, enter Setup and enable the onboard NIC “with MBA.” The second time, enter Setup and enable network booting on the boot devices list. After the third boot, wait for the network boot message and hit Ctrl-Alt-B to access the network boot options, where you can configure MBA to support PXE. Hopefully, network boot configuration will be more cleanly integrated into next-generation PC BIOSes.

PXE-Free Booting

If you don’t have a PXE-capable NIC or motherboard, you should investigate Etherboot (http://www.etherboot.org) or Netboot (http://www.netboot.org).

Unlike a PXE ROM, which loads an intermediate network boot program, these other network booting solutions are designed to load a specially-marked Linux kernel directly. There are only three differences in the server configuration:

  • You don’t need PXELinux at all.

  • In the DHCP configuration, the filename parameter contains the name of the client kernel file.

  • The client kernel file needs to be processed by the mknbi program to correctly recognized by the ROM.

Other than these variations, all of the instructions in this article apply.


Network booting requires a DHCP server and a boot server. The DHCP server provides a variety of critical information, including the IP address of the boot server and the names of the bootstrap files. The boot server provides a simple download facility (TFTP) for accessing the bootstrap files and NFS access to the root filesystem.

If you have PXE ROM support, the system boots in the following steps:

  1. The PXE ROM queries the DHCP server to obtain an IP address, the IP address of the boot server, and the name of a network boot program (NBP) to download.

  2. The PXE ROM uses TFTP to load the NBP from the boot server.

  3. The NBP obtains DHCP information from the PXE ROM.

  4. The NBP loads a configuration file from the boot server using TFTP.

  5. The NBP loads the kernel using TFTP.

  6. The kernel queries the DHCP server to obtain the IP address of the boot server and the name of the root filesystem.

  7. The kernel mounts the root filesystem from the boot server using NFS.

  8. The kernel begins the init process, which typically involves mounting other NFS filesystems.

This article configures the server in this same order, which simplifies testing. You should be able to reboot your client after each configuration step and see each time that it gets a little further in the boot sequence.


If you already have a DHCP server on your network, you can simply add the next-server, filename, and root-path options shown in Listing One to your existing configuration. Otherwise, you can install a DHCP server on the same machine as your boot server. (Please note that you should avoid having two DHCP servers on your network, and you should never mix BOOTP and DHCP. DHCP was designed to replace BOOTP.)

Listing One: Minimal dhcpd.conf file

# For ISC DHCPD 3.0, you may need to uncomment the following line
#ddns-update-style none;

group {
# Common options for all network-boot clients
# IP address of TFTP/NFS server
# The name of the file to be downloaded by the ROM
filename “/tftpboot/pxelinux.0″;
# A default server directory to be used as / by the clients
#option root-path “/clients/shared/root”;
# IP addresses of DNS servers
#option domain-name-servers XX.XX.XX.XX, YY.YY.YY.YY;
# IP addresses of routers
#option routers AA.AA.AA.AA, BB.BB.BB.BB;

# An entry for one specific client
host sample-client {
# Ethernet address of the client machine
hardware ethernet EE:EE:EE:EE:EE:EE;
# IP address to assign
# Override root-path option for this machine
# Our initial configuration will use separate
# root directories for each client.
option root-path “/clients/”;

# Entries for other clients go here:
# host another-client {
# etc, etc, etc
# }

Most Linux systems use the Internet Software Consortium’s DHCP server software (ISC DHCPD). To configure ISC DHCPD, you’ll need to create a suitable dhcpd.conf file, which is generally stored in /etc or /usr/local/etc. The minimal dhcpd.conf file, shown in Listing One, provides basic information for a single network-boot client that’s been assigned IP address The only entries specific to network booting are the next-server, filename, and option root-path entries. Note that network-boot clients should have fixed IP addresses.

The next-server field provides clients with the IP address of the boot server. The filename field specifies the name of the network boot program that should be downloaded. The /tftpboot directory prefix here is strongly recommended; you can sometimes use a different prefix or omit the prefix, but not all software permits this. The root-path option names the directory on the boot server that the kernel will use for its root filesystem. We’ll create these directories later.

Note that you’ll need the Ethernet address of your client machine before you can use this DHCP configuration. There are several ways to get this information. If the client machine already has Linux installed, use ifconfig and look for the HWaddr value. If Windows is on the machine, look in the Network control panel. The PXE ROM will often show such information either in the configuration screen or during an attempt to boot from the network. If none of those work, use arpwatch or tcpdump on the server to watch the network packets from the client as the client tries to network boot.


With DHCP configured to tell the PXE ROM the location of the network boot program, you need to find and install a suitable NBP. PXELinux is a network boot program developed by the author of the SYSLinux boot loader. It uses the PXE API to locate and download a configuration file, and then uses that configuration file to download and execute the Linux kernel. You can download the newest version of SysLinux from www.kernel.org and copy the file pxelinux.0 to the /tftpboot directory.

When PXELinux boots on the client, it tries to load a configuration file. Create an initial configuration file /tftpboot/pxelinux.cfg/default containing the following lines:

LABEL linux
KERNEL vmlinuz-2.2.20
APPEND root=/dev/nfs


The PXE ROM uses TFTP (which is completely unrelated to FTP) to download the network boot program. (Intel’s PXE specification includes a new Multicast TFTP protocol, but that’s not yet widely supported.) PXELinux also uses TFTP to download it’s configuration files and the Linux kernel. Unfortunately, PXELinux does not work with all TFTP programs. It will work with either tftpd-hpa or atftpd.

The tftpd program is usually started from inetd. If you install one of these TFTP servers from a package, the inetd configuration should be handled for you. If not, you should read the documentation for the TFTP server and manually edit the inetd.conf file.

Note that TFTP offers essentially no security. Don’t put anything in the /tftpboot directory that you don’t want other people to read!

To test the TFTP server, try using the tftp client program on the local host to download a file:

$ tftp
> get /tftpboot/pxelinux.0

Of course, you should also try this from another machine on your network.

If you get errors here, make sure that you restarted inetd after the configuration change. Check /var/log/syslog for error messages. Make sure that everything in the /tftpboot directory is world-readable. Finally, if you’re using TCP wrappers, you may need to adjust the permissions for TFTP in hosts.allow and hosts.deny.


At this point, you should be able to boot a PXE-capable PC from your server and see something work. The client should make a DHCP request and use TFTP to download and execute PXELinux. PXELinux will stop at the boot: prompt when it fails to find a Linux kernel.

Next, let’s look at building a custom kernel for network boot clients.

Popping New Kernels

The network boot server and network boot clients generally have quite different requirements for a kernel. In particular, many Linux distributions do not include Ethernet drivers in their default kernel. Instead, they load suitable modules from the hard disk. This does not work for a machine that’s booting from the network.


Although there are other approaches, let’s recompile the kernel to include all of the network drivers that are needed, plus a few other useful tidbits. Using make menuconfig, make sure you have the items shown in Figure One compiled into the kernel (do not compile them as modules):

Figure One: Building a kernel for the network boot clients

Network Device Support/Ethernet

No real harm comes from including unused PCI Ethernet drivers, so go ahead and include any that are in common use on your network.

Networking Options

The kernel needs to perform a DHCP request when it boots to initialize the network interfaces. You should enable IP: kernel-level configuration support (CONFIG_IP_PNP) and the two sub-options Enable by default (CONFIG_IP_PNP_ENABLE) and DHCP support (CONFIG_IP_PNP_DHCP).

Filesystems/Network File Systems

The kernel needs to be able to access its root filesystem using NFS. You should enable NFS filesystem support (CONFIG_NFS _FS) and the two sub-options NFS version 3 support (CONFIG _NFS_V3) and Root file system on NFS (CONFIG_ROOT_NFS).

Block Devices

While not strictly necessary, ram disk support can be very useful. Enable RAM disk support (CONFIG_BLK_DEV_RAM).

Once you’ve configured the client kernel of your dreams save the configuration, compile the new kernel, and copy the kernel file to /tftpboot:

# make dep bzImage
# cp arch/i386/boot/bzImage \ /tftpboot/vmlinuz-X.X.X

Remember to replace X.X.X with the correct version for this kernel. Also remember that this is the kernel for the client, so you should not use make install to install it.


Double-check the PXELinux configuration in /tftpboot/ pxelinux.cfg/default to make sure the kernel file specified there matches the kernel file in the /tftpboot directory. Then try booting your client machine.

This time, PXELinux should boot and load the Linux kernel file. Watch to make sure that the kernel successfully detects the Ethernet card in the client machine and successfully obtains startup information from DHCP.

If anything goes wrong, double-check your kernel configuration. You may have accidentally compiled a critical component as a module.

This time, the boot will fail when Linux is unable to mount the root filesystem. That’s ok. We’re making progress. Next, we mount filesystems from the server.

Managing the Clients’ Filesystems

With the kernel booting, we’re ready to create a root filesystem and other filesystems for the client. Our first attempt will use a completely separate set of directories for each client.

For example, the client with IP address will mount /clients/ as its root filesystem. Similarly, each client will have its own /usr, /var, and /tmp filesystems. Only the /home filesystem will be shared among the clients in this initial configuration.

Note that there is no overlap between the server’s files and the clients’ files. This is partly a security issue: the clients should not be able to modify or even read any file that is used by the server. It’s also a maintenance issue: server upgrades need to be handled much more carefully than client upgrades.


Start by creating a separate directory for each client, for example, /clients/ for the machine assigned IP address Within that directory, create root, usr, var, and tmp directories. Populate these directories with a complete client installation.

Unfortunately, this is not simple. Many people start by copying the existing server directories and hand-customizing them. This works, but is rather tedious. A simpler approach is to do a standard install onto a hard disk of one of your client machines. Then use NFS to mount your server directories onto the client and copy the entire client system over. Alternatively, you can temporarily install a second hard disk into your server and accomplish the same thing without using NFS. Advanced system administrators may even be able to use chroot trickery to fool a standard installer into installing directly into these directories.

The best way to copy an entire hard disk partition over to a new directory is to use find to select the files to be copied (the -mount option prevents find from crossing mount points) and cpio -p to do the actual copying. For example, to copy the filesystem mounted at /mnt to /clients/, you might use:

# cd /mnt
# find . -mount | cpio -pd \

Whichever approach you choose, some tweaking will be necessary before these filesystems are suitable for network booting:

  • /clients/ must be edited. Remove any entries that refer to the local hard disk (such entries generally start with /dev/hd), and add the new entries shown in Listing Two. Again, should be replaced with your server’s IP address, and with the IP address of the test client.

  • Edit the client’s network configuration files to ensure that eth0 does not get configured. Remember that by the time these configuration files are read, the network is already running, and attempts to reconfigure the NIC can break the system. (This is why we chose to have the DHCP server assign fixed IP addresses to network-boot clients.)

  • Make sure that /etc/resolv.conf contains correct DNS information. Remember that DHCP is being handled by the kernel before /etc is available. As a result, DHCP cannot be used to update /etc/resolv.conf.

Listing Two: Entries to add to /etc/fstab / nfs rw,hard,intr,nolock 0 0 /usr nfs rw,hard,intr,nolock 0 0 /var nfs rw,hard,intr,nolock 0 0 /tmp nfs rw,hard,intr,nolock 0 0 /home nfs rw,hard,intr,nolock 0 0

Configure and Test NFS

If you’re not running NFS, install and configure the NFS server software. (If you have a choice, make sure you get the “kernel” server. The old user-space NFS server does not work well.) Then create an /etc/exports file on the server to specify which filesystems should be available to clients. Listing Three shows a typical exports file.

Listing Three: The /etc/exports file


After editing /etc/exports, restart the NFS server and use exportfs (with no options) to list the currently exported filesystems. You can then use the (localhost) loopback device,, to test your NFS configuration directly on the server:

# mount /mnt

If ls /mnt produces the same output as ls /clients/, then NFS is working okay and you can umount /mnt.

If not, check that NFS is running (if you’ve just installed it, you may need to start it manually or reboot the server). If you have other machines on your network, you should also try mounting the /clients/ root/ directory on one of them as well. If that fails, you may have a mismatch between your NFS server software and NFS client software or you may have the wrong permissions in your /etc/exports file. Check your NFS configurations and retry these instructions until NFS works.


You should now be able to successfully boot one of your clients. Although you’ll probably see a lot of errors at this stage, everything should be minimally functional. Expect to spend at least several hours testing the client. Since everything is mounted read-write on the client, you should be able to log into the client as root to fix anything that isn’t working correctly. More severe problems may need to be addressed from the server side.

This approach can be used for a production system. Setting up a new client requires a few easily-automatable steps:

  • Clone the /clients/ directory tree.

  • Add the new directories to the server’s /etc/exports file.

  • Modify the root/etc/fstab file for that client.

  • Add a new host section to the DHCP server for this client.

Sharing /usr

While this configuration works, a few additional changes would make it significantly easier to maintain. In particular, it would be nice to share the root and /usr filesystems across clients. That would allow most configuration changes to be done only once yet apply to every client. It would also make it easier to add new clients.

Sharing /usr is fairly simple. Move /clients/ to /clients/shared/usr. Similarly, add a line to /etc/exports to make this shared directory available to every client:

/client/shared/usr *(ro,no_root_squash)

You should also edit the per-client /etc/fstab files. Note that the directory is now being exported read-only. Generally, it’s best for shared directories to be read-only to prevent the various clients from interfering with each other. (Obviously, /home is an exception.) This may require some attention to software configuration, but a read-only /usr is common enough that you should have few problems.


A shared read-only root directory is more troublesome, however. Start by making changes similar to those for /usr: move /clients/ to /clients/shared/root and make appropriate changes to the server’s /etc/exports. Since we’re moving the root, you’ll also need to update the DHCP root-path option.

Sharing /etc/fstab is a problem. Directories such as /var and /tmp cannot be shared among clients. This means that these directories will always be mounted from places that depend on the particular client. To handle this, simply delete those entries from /etc/fstab entirely and add a script to the boot process that computes the appropriate names, as shown in Listing Four.

Listing Four: mount.script boot script to mount client-specific directories

# File: /clients/shared/root/etc/init.d/netboot-dirs
# Mount directories whose names depend on the client

IP=`ifconfig eth0 | grep inet | sed ‘s/.*inet //;s/
/bin/mount$IP/var /var
/bin/mount$IP/tmp /tmp

To add this script to the boot process, you need to add it to the startup sequence. For Debian, the following suffices:

# cd /clients/shared/root/etc/rcS.d
# ln -s ../init.d/netboot-dirs S09netboot-dirs

You should now be able to minimally boot your clients, although you will notice a lot of errors from programs that expect to be able to modify files in the root filesystem. Fortunately, there are standard ways to solve the most serious such issues:

  • /etc/mtab: The mount and umount commands expect to update /etc/mtab. Fortunately, /proc/mounts largely duplicates this, so you can change /etc/mtab into a symbolic link to /proc/mounts. You may also need to edit some of the startup scripts to use the -n option to mount or umount.

  • /dev/log: The syslogd daemon creates a log socket when it starts. By tradition, this log socket is in /dev. To change this, make /dev/log a symbolic link to /var/run/log and change the startup of syslog to include the -p /var/run/log option (for example, in Debian, edit the etc/init.d/sysklogd script.)

  • /dev: There are a number of other problems involving /dev. Some programs expect to be able to change ownership and/or permissions of files here. Also, /dev is a significant administrative problem since your clients will have some hardware variation. Fortunately, starting with the 2.4 kernels, the devfs filesystem is now standard. Use it.

  • /etc/passwd: If the root filesystem is read-only, then the password file cannot be changed. In many environments, this is actually a good thing. If you need to allow users to change their passwords, use NIS to centralize password administration.


Every computer must have its very own writable /var and /tmp directories. Earlier, we provided those by creating such directories on the server for each client. If your clients have enough memory, it’s possible to eliminate this by placing those directories in a ramdisk. As above, you build a script to mount the /var and /tmp directories, but you have to create them first. Listing Five shows how to do this.

Listing Five: boot.script, a boot script to create memory-based writable directories

# File: etc/init.d/netboot-dirs
# Initialize a 1MB /var
/sbin/mke2fs -q /dev/ram12 1024
/bin/mount /dev/ram12 /var
/bin/chown 0:0 /var
/bin/chmod 755 /var

# Create some needed dirs
cd /var
/bin/mkdir -p –mode=755 run lib log/news spool/cron/crontabs
/bin/mkdir -p –mode=1777 tmp lock

# Initialize a 4MB /tmp
# -N option tunes this for small files
/sbin/mke2fs -q -N 2048 /dev/ram11 4096
/bin/mount /dev/ram11 /tmp
/bin/chown 0:0 /tmp
/bin/chmod 1777 /tmp


Swap is trickier. Ideally, your clients will have enough RAM that you can run them without swap. Otherwise, you can have swap space either on a local hard disk or on the server.

Swapping to a local hard disk is easy and fast and does not strain your network. For clusters or server farms, it works very well, since you can reasonably ensure that every machine has the swap partition in the same place. For example, you might have a convention that /dev/hda3 is always the swap partition. This works less well for desktop workstations where you don’t always have sufficient control over the clients.

Swapping to the server is a poor idea. It generates a heavy load on your network, it is slow, and it opens some troubling security holes. Because of these issues, savvy administrators avoid it, which in turn means that the support for it is spotty and not well tested. If you must swap to the server, there are a few descriptions on the Internet explaining various kernel patches or other techniques that can provide network swapping.

Other Issues

A memory-based /var will lose its entire contents on each reboot. This can break software that uses /var for persistent storage. You can sometimes work around this by either reconfiguring such software to use /home for persistent storage or by using an NFS-mounted /var.

NFS file-locking is notoriously troublesome. Many programs will break if you have everything mounted from NFS because they can’t lock files. Fortunately, most older software avoids file locks because of this exact issue. One workaround is to use memory-based filesystems for those directories where file-locking is necessary. The /var/lock directory is a good candidate for this. Some software packages allow you to specify the location for lock files. NFS version 3′s file locking works for some people.

These two problems together break many package-management tools, which store their database in /var (so you can’t use memory-based /var) and attempt to lock their database file (so you can’t use NFS-mounted /var).

Developers should note that software packages that need locks to control access to database files should not lock the database file itself. Instead, they should always use a separate lock file. That allows the database file to be stored on a persistent NFS-mounted filesystem with the lock file being stored on a separate memory-based filesystem.

Sniny new Boots

Network booting allows you to dramatically simplify the management of large, homogenous networks. Although the emerging PXE standard has simplified many aspects of network booting, there are still tough issues.

The server configuration described in this article is reasonably straightforward. Almost any Unix-like system can provide the required DHCP, TFTP, and NFS services with minimal fuss. However, the client setup inevitably requires a lot of manual work. Because network booting is still fairly unusual, you should expect to spend some time fidgeting with boot scripts and trying various combinations of memory-based and NFS-based storage before everything works exactly the way you want.

However, the advantages are well worth it. Network booting has been used for many years by computer labs, and computing clusters. As Linux systems adapt to better support network booting — such as the recent addition of devfs in the kernel — these techniques will become even more useful.

Tim Kientzle is an independent consultant based in Oakland, California. He can be reached at kientzle@acm.org. You can download the listings used in this article from http://www.linux-mag.com/downloads/2002-10/netboot.

Comments are closed.