pxeboot and nfsroot with debian

[ link: nfsroot-on-debian | tags: pxe nfs boot linux debian | updated: Tue, 14 Dec 2010 16:06:32 ]

I have two boxes (i386 and amd64) in the lab that I use for testing of drivers I work on. Recently another Maxtor hard disk died on me, and I decided to get network booting working. I already have a file server from which I host my $HOME directories and do all backups from. It sounded like a win.

I've never done this before, so it took me a few hours to get the first host going, the second took 10 minutes plus the amount of time to build the kernel for it.

Below, I describe steps I took to get pxe-enabled hardware to boot a debian image, from a debian DHCP, TFTP and NFS servers.

My setup includes three machines:

  • gluon - gateway and DHCP server,
  • meson - TFTP and NFS server, and
  • nitrogen - disk-less client.

You can run these services on one machine, or split them over four boxes if you need to.

Let's start by looking at the sequence of steps the disk-less client will go through to get to the boot prompt:

  • BIOS boots from PXE-enabled firmware image.

    • you need to configure your BIOS to enable PXE booting. PXE is a protocol that enables DHCP booting. If you don't have PXE in your BIOS, you can still use a special PROM on an older NIC. I will not cover that.
  • PXE uses DHCP to get the IP address for the host.

    • DHCP server uses the MAC address of the NIC to match it up with an IP configuration, and further info on how to boot. In these details is the IP address of the TFTP server to use.
  • PXE uses TFTP server to fetch the initial boot image, or pxelinux.0.

    • pxelinux.0 is provided by syslinux package.
  • pxelinux.0 image gets configuration from TFTP server.

    • this is where the disk-less client learns about the kernel to boot and boot parameters to use.
  • kernel parameters tell it to fetch the root file system over NFS.

  • profit.

And now on to the setup. I am going to cover things in the boot order, but feel free to skip around. This might be a good time to start building that kernel and do the rest while it builds. I am not sure if distro kernels support nfsroot. Mine didn't seem to.

Don't forget to change the IPs below to match your network configuration.

BIOS

You need to enable PXE booting in your BIOS. I cannot really help here. It should work on modern PC's and laptops.

DHCP

Install dhcpd.

    apt-get install dhcpd

On the DHCP server (in /etc/dhcpd.conf) you will need network booting options enabled:

    allow bootp;
    allow booting;

You will need to define a subnet, and teach the disk-less box where all the other systems are on the network:

    subnet 10.0.0.0 netmask 255.255.255.0 {
            range 10.0.0.150 10.0.0.200;                            # dynamic IP range

            default-lease-time 3600;
            max-lease-time 7200;

            option domain-name "yourdomain.com";

            option routers 10.0.0.254;                              # default gateway(s)
            option subnet-mask 255.255.255.0;
            option domain-name-servers 10.0.0.250, 10.0.0.250;
            option time-offset -18000;                              # Eastern Standard Time

    }

Next define a name host that boots using the file pxelinux.0 named on NFS server 10.0.0.2.

    host llama0 {
            hardware ethernet 01:23:45:67:89:ab;                    # client MAC
            option host-name "diskless";                            # client name
            fixed-address 10.0.0.100;                               # client IP
            next-server 10.0.0.2;                                   # TFTP server
            filename "pxelinux.0";                                  # file on TFTP server
    }

Restart dhcpd, if it fails to start up check the logs.

    /etc/init.d/dhcp restart

At this point you should have your disk-less client getting an IP address.

TFTP

Next is the TFTP server. The TFTP server gets launched from inetd. I am using openbsd-inetd.

    $ apt-get install atftpd openbsd-inetd

The inetd daemon passes to TFTP the directory where all the files are held. This configuration is in /etc/inetd.conf.

    tftp            dgram   udp     wait    nobody  /usr/sbin/tcpd  /usr/sbin/in.tftpd /tftpboot

I kept mine in /tftpboot for historical reasons. If the directory does not exist, you will have to create it.

    $ mkdir /tftpboot

PXE

So far the TFTP server is useless. Let's install the initial boot image (pxelinux.0).

    $ apt-get install syslinux
    $ cp /usr/lib/syslinux/pxelinux.0 /tftpboot/

At this point you can have the disk-less client boot the pxelinux.0 image. It still needs a configuration. It looks for it in /pxelinux.cfg/ on the TFTP server.

    $ mkdir /tftpboot/pxelinux.cfg/

If uses the MAC address of the client to key on. Failing that it uses the IP address (in hex). We will use the MAC. Note that the actual MAC address is preceeded by 01- and the colons are replaced by dashes. MAC letters must be in lower case!.

The configuration will look something like this:

    $ cat /tftpboot/pxelinux.cfg/01-01-23-45-67-89-ab
    IPAPPEND 2                      # I don't know :)
    SERIAL 0 115200                 # show on serial console

    PROMPT 1                        # show boot: prompt
    TIMEOUT 10                      # boot default after 1 second
    DEFAULT linux

    label linux
      kernel vmlinuz-diskless
      append rw console=ttyS0,115200n81 console=tty0 netdev=irq=11,name=eth0 netdev=irq=10,name=eth1 root=/dev/nfs ip=::::diskless:eth0:dhcp nfsroot=10.10.10.200:/nfsroot/diskless panic=100

    label another
      kernel another
      append ...

Note that the config file can support multiple images. The default is linux. The kernel image is vmlinuz-diskless, once we build it it will reside in /tftpboot/vmlinuz-diskless.

The kernel append command line is a bit long, so let me explain:

  • rw - mount read/write, you can use a read-only nfsroot, but that might require more work.
  • console=ttyS0,115200n81 console=tty0 - setup two consoles, on serial and on the terminal (screen).
  • netdev=irq=11,name=eth0 netdev=irq=10,name=eth1 - I have two network cards and wanted to force their order based on IRQ.
  • panic=100 - should booting fail, reboot after 100 seconds -- nice since my "lab" is in the basement.
  • root=/dev/nfs - root is on NFS.
  • ip=::::diskless:eth0:dhcp - IP configuration for nfsroot comes from DHCP, we will use interface eth0, hostname is diskless.
  • nfsroot=10.0.0.3:/nfsroot/diskless - nfsroot will be mounted from this host/path.

Note that the kernel parameters can be no longer then 256 characters. If you do have a initrd image, it can be listed on the append line.

At this time, the disk-less client will know to look for a kernel, but it will not find it.

ROOTFS

I keep my network booting root filesystems organized under /nfsroot/${hostname}/, and I install them using debootstrap.

    $ mkdir /nfsroot/diskless
    $ apt-get install debootstrap
    $ debootstrap --include=nfsbooted,dhcp3-client,procps,passwd,vim,less,configure-debian  etch  /nfsroot/diskless

This will take a while.

nfsbooted is a boot script that changes permissions of ramfs partitions so they can be accessed by everyone. For it to work you will need to create a directory in the rootfs image.

    $ mkdir /nfsroot/diskless/.nfsroot

Once it's done, you have to modify a few files. You might as well do it from the chroot, if you don't make sure you're not chaning files in / but in /nfsroot/diskless.

    $ chroot /nfsroot/diskless su -
  • /etc/hosts needs to contain the following

    127.0.0.1       localhost
    
  • /etc/hostname should be set to your hostname

    diskless
    
  • /etc/resolv.conf should contain something sane, copy it from the NFS server.

  • /etc/nfsbooted/mountfix.conf needs to point to the right directory:

    NFSROOTDIR=/.nfsroot
    
  • /etc/fstab will contain a few tmpfs partitions so that temp files don't have to go over NFS. Here is what mine looks like:

    /               /.nfsroot       none    bind,ro         0 0
    proc            /proc           proc    defaults        0 0
    
    
    # copied from /etc/nfsbooted/fstab
    /dev/ram        /tmp            ramfs   defaults,rw,auto,dev            0 0
    /dev/ram1       /var/run        ramfs   defaults,rw,auto,dev            0 0
    /dev/ram2       /var/state      ramfs   defaults,rw,auto,dev            0 0
    /dev/ram3       /var/lock       ramfs   defaults,rw,auto,dev            0 0
    /dev/ram4       /var/account    ramfs   defaults,rw,auto,dev            0 0
    /dev/ram5       /var/log        ramfs   defaults,rw,auto,dev            0 0
    /dev/ram6       /var/lib/gdm    ramfs   defaults,rw,auto,dev            0 0
    /dev/ram7       /var/tmp        ramfs   defaults,rw,auto,dev            0 0
    
  • finally, you should finish off the debian installation

    $ configure-debian --all
    

Then exit the chroot.

    $ exit

I think that's it. I hope I didn't forget anything :)

KERNEL

Let's build a kernel. There are so many ways to build one. Here is just one.

    $ apt-get install git-core build-essential
    $ git clone git://git2.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.20.y.git linux
    $ cd linux
    $ make allmodconfig

This will create a configuration with everything enabled as a module. But some things are required to be built into the kernel. So, edit and/or add the following:

  • DHCP support

    CONFIG_IP_PNP=y
    CONFIG_IP_PNP_DHCP=y
    
  • NFS client support

    CONFIG_NFS_FS=y
    CONFIG_NFS_V3=y
    CONFIG_NFS_V4=y
    CONFIG_NFS_DIRECTIO=y
    
  • NFS root support

    CONFIG_ROOT_NFS=y
    
  • your network interface

    CONFIG_E1000=y                          # if you have Intel E1000
    CONFIG_BNX2=y                           # if you have Broadcom NetXtreme
    

If you don't know what NIC you have, enable a bunch. It costs little.

Next we build...

    $ make -j2 bzImage modules

Then install...

    $ make INSTALL_MOD_PATH=/nfsroot/diskless modules_install
    $ make INSTALL_PATH=/nfsroot/diskless/boot install

And lastly copy the vmlinuz file to the TFTP server.

    $ cp /nfsroot/diskless/boot/vmlinuz* /tftpboot

At this point the disk-less client will be able to boot the kernel and will die when it tries to mount the nfsroot.

NFS

Last step is to configure the NFS server to export our new nfsroot.

    $ apt-get install nfs-kernel-server

This is done through /etc/exports

    /nfsroot/diskless 10.0.0.100(rw,no_root_squash,no_subtree_check)

And export it

    $ exportfs -a

And now, it should all work.

References

I used (at least) the following documents to get the above procedure:

And these were found to be also useful:

Bart's Blog

/nfsroot-on-debian

About

I am an embedded Linux software developer and consultant operating under Jukie Networks Inc in Ottawa, Canada.

RSS Feed - Full Content

RSS Feed - Full Content

ipv6 ready

Valid XHTML 1.0 Transitional

Created with Vim

Created with Perl




Tags

abiword adam android apt asm ata audio bacula bash bios board-game boot bug busybox bzr c code conference debian debug desktop devel disk dns dpkg dream email embedded firefox font fs fun g1 gdb geode git git scm git-find git-graft go google gpg grub hardware html hw inkscape ion3 ipsec ipv6 ipv6summit.ca irc irssi joke kdb kernel klips kqemu kvm laptop lastfm ldap lighttpd linux lua lvm mail make mercurial meta mmc mouse-free mp3 mpd music mutt net nexusone nfs oclug ogre ols oom OOo openswan opteron osx ottawa perl photography power pxe qemu raid redhat rogers ruby sata sbc scm screen script scsi security sheeva shell skype spam squid ssd ssh suse svn tags talk thinkpad ubuntu uml unix urxvt usb uzbl vi video vim vimgit virt vpn vserver web weechat wifi wind wmii wmiirc-lua work x x86 x86emu xen xen-box-setup xfs xterm zsh

Posts

[ 20130409092758 ]
automount mmcblk devices

[ 20130316190507 ]
tunnelbroker vs IRC

[ 20110616180255 ]
HDD -> SDD

[ 20110517204617 ]
how to manually create a 6in4 tunnel

[ 20110501202915 ]
Presentation slides published!

[ 20110320190458 ]
What is IPv6, and why should I care?

[ 20110309230106 ]
Ottawa IPv6 Summit 2011

[ 20110309225149 ]
I am now an IPv6 Sage

[ 20101214145558 ]
It's a Holiday Miracle

[ 20101103095130 ]
ipv6 on your desktop in 2 steps

[ 20101020113026 ]
growing a live LVM volume

[ 20100915175649 ]
distributing DNS list through radvd

[ 20100908173402 ]
Canadian ipv6 drought

[ 20100903190623 ]
ipv6 certification

[ 20100723192259 ]
m4a to mp3

[ 20100723134522 ]
git 1.7.2 is out

[ 20100523092001 ]
console=ttyS0 with grub2

[ 20100426162644 ]
How many times is my function used within an executable?

[ 20100412100135 ]
vmlinux on Ubuntu

[ 20100328152951 ]
where your WIND coverage ends

[ 20100326113354 ]
Nexus One live in Ottawa on WIND Mobile

[ 20100311133411 ]
the WIND excitement

[ 20100302160743 ]
sata hotswap pico-HOWTO

[ 20100227093435 ]
serving http content out of a git repo

[ 20100225104049 ]
pimped out zsh prompt

[ 20100221184126 ]
live termcasting of your terminal over telnet

[ 20100215083725 ]
Debian on UBIFS upgrade on SheevaPlug

[ 20100214205532 ]
nexus one

[ 20100130150527 ]
skype on Debian Linux (64bit)

[ 20100125191021 ]
I am so peeved at Rogers

[ 20091205130455 ]
running really nice

[ 20091120225902 ]
notmuch for vim

[ 20091113102221 ]
squid and apt

[ 20091104195344 ]
using WIP branches to save every edit

[ 20091104194146 ]
using git workflows to avoid loosing intermediate changes

[ 20091019183503 ]
reflections on ACM Reflections

[ 20091013174248 ]
bacula rejected Hello command

[ 20091007235046 ]
pimping out git log

[ 20091002094400 ]
virtualization primer

[ 20090922110756 ]
adding an external encrypted volume under Debian

[ 20090905112529 ]
the dreaded process of rooting Rogers Dream

[ 20090830174551 ]
Cyanogen's recipe for Cupcake/Donut-like pastry

[ 20090827100232 ]
simpler android rooting

[ 20090821095427 ]
androids don't like water

[ 20090820215358 ]
prettier function tracing

[ 20090816151854 ]
Baking a cupcake

[ 20090814131559 ]
Hello Android!

[ 20090813100025 ]
sim unlocking a G1

[ 20090731185522 ]
rooting the droid

[ 20090727210307 ]
getting into android

[ 20090714094340 ]
importing an old project into git

[ 20090702113222 ]
Why pick Git?

[ 20090628131255 ]
select loop for X events

[ 20090625181315 ]
portable printf

[ 20090623225800 ]
switching to uzbl

[ 20090622223147 ]
Linux Symposium

[ 20090622214023 ]
bringing git-format-patch to bzr

[ 20090612215638 ]
nfs local caching with fscache and cachefilesd on Lenny

[ 20090610202041 ]
Scott Chacon smacks git around

[ 20090610150039 ]
how would you read a file into an array of lines

[ 20090609215141 ]
libguestfs

[ 20090609000208 ]
tiding up the PATH

[ 20090608232531 ]
only showing relevant messages in mutt by default

[ 20090608010405 ]
git-vim hacking

[ 20090530223801 ]
mark-yank-urls: fix bug allowing shell to interpret the url

[ 20090509212648 ]
wmiirc-lua v0.2.8 release

[ 20090509113125 ]
wmiirc-lua moving to github

[ 20090504101605 ]
two terminals one PWD

[ 20090501172645 ]
splitting files out of a commit

[ 20090409155905 ]
git workflow: git amend

[ 20090401112030 ]
how old are these files in git?

[ 20090330181138 ]
sles 11 on kvm

[ 20090323194942 ]
android true type font

[ 20090322203939 ]
popen with stdin, stdout, and stderr

[ 20090320214228 ]
shrinking URLs

[ 20090304004744 ]
readlater

[ 20081112150409 ]
splitting patches with git

[ 20081011081638 ]
creating busybox symlinks

[ 20081002215121 ]
wmiirc-lua v0.2.5 release

[ 20080916155113 ]
git-svn strangeness

[ 20080915112959 ]
installing git man pages quickly

[ 20080913112345 ]
wmiirc-lua v0.2.4 release

[ 20080825100454 ]
Kernel Walkthroughs - booting

[ 20080813210205 ]
Linux Kernel Booting

[ 20080719211329 ]
printable OLS/2008 schedule

[ 20080715214447 ]
wmiirc-lua updates

[ 20080713194704 ]
Git Screencast

[ 20080713143429 ]
four steps to reproducible Debian installs

[ 20080705150651 ]
USB2.0 enclosure benchmark

[ 20080703230924 ]
Linux Kernel Walkthroughs posted

[ 20080702113602 ]
Introducing the Ottawa Ruby folks to Git

[ 20080628160732 ]
Authenticating Linux against OSX LDAP directory

[ 20080627142123 ]
Canada Day Events 2008

[ 20080613162541 ]
Linux Kernel Walkthroughs

[ 20080510083828 ]
is my usb device connected to a fast port?

[ 20080509111534 ]
show more git info on zsh prompt

[ 20080430104202 ]
git-vim

[ 20080412100337 ]
color your word

[ 20080404105620 ]
show current git branch on zsh prompt (2)

[ 20080303200359 ]
how to track multiple svn branches in git

[ 20080301134220 ]
fixing X for GeodeLX

[ 20080108002540 ]
kvm nfs hang

[ 20080107160836 ]
screen -c relative path bug

[ 20080105132854 ]
WeeChat spell suggestions

[ 20071219221358 ]
show current git branch in zsh

[ 20071217141037 ]
wmiirc-lua kitchen sink repository

[ 20071212100316 ]
protecting sshd from OOM killer

[ 20071204234232 ]
wmiirc-lua v0.2.1 remembers a bit more

[ 20071013205336 ]
wmiirc-lua v0.2 has suspend and raw modes

[ 20070929112345 ]
wmiirc-lua debianization

[ 20070924104140 ]
zsh tip of the day - global aliases

[ 20070915094213 ]
wmiirc-lua v0.1.1

[ 20070913130838 ]
comparing two directories

[ 20070909204125 ]
git-rebase --interactive

[ 20070908115905 ]
zsh tab completion awesomeness

[ 20070902000736 ]
wmiirc in lua

[ 20070831150306 ]
debugging with -dbg libraries

[ 20070831142646 ]
svn status like output in git

[ 20070829141847 ]
Git Cheat Sheet

[ 20070821142038 ]
switching to abiword

[ 20070811105746 ]
forwarding ssh and X through screen

[ 20070807112531 ]
git-svnup

[ 20070724082355 ]
reducing power consumption

[ 20070722123734 ]
Makefile template

[ 20070722002649 ]
less, colourful

[ 20070719162359 ]
irssi docs

[ 20070716114553 ]
qemu eats up /dev/shm

[ 20070710214512 ]
git-clean in svn land

[ 20070705113139 ]
ipw2200 not working

[ 20070627191916 ]
git slides updated

[ 20070618220649 ]
unpopular debian packages on my system

[ 20070611125852 ]
git-svn with multiple branches

[ 20070517085321 ]
Linus on Git at Google

[ 20070510134551 ]
vim modelines insecure

[ 20070504205042 ]
bios disassembler

[ 20070504124124 ]
dd hex arguments

[ 20070503013555 ]
urxvt mouseless url yanking

[ 20070502211941 ]
gitdiff.vba v2

[ 20070419234350 ]
india

[ 20070418155857 ]
zsh fun

[ 20070418143632 ]
pipe to pastey.net

[ 20070418094151 ]
vimgrep alias

[ 20070416202545 ]
mouse-free

[ 20070406141850 ]
ATA messages via SCSI layer

[ 20070330221019 ]
GITDiff vim plugin

[ 20070329011735 ]
git presentation for OCLUG

[ 20070328123631 ]
fixing vim's [[ and ]] for bad code

[ 20070316092236 ]
pxeboot and nfsroot with debian

[ 20070312134706 ]
etc snapshots with git

[ 20070307004041 ]
remote power switch

[ 20070222215355 ]
klips-less openswan git tree

[ 20070221041316 ]
git caching for v1.5.x

[ 20070218002214 ]
klips loses zlib

[ 20070209172606 ]
vim and linux CodingStyle

[ 20070207205427 ]
my kqemu install

[ 20070204100100 ]
leaner meaner openswan

[ 20070115111917 ]
wmii+ruby xlock action

[ 20070112131252 ]
cloning xterms in wmii+ruby

[ 20070102010551 ]
wmii+ruby talk for OCLUG

[ 20061228220641 ]
dump and restore

[ 20061218100219 ]
C style

[ 20061101002027 ]
fetching all git branches from remote

[ 20061028111607 ]
local caching for git repos

[ 20061020145437 ]
automatic version creation with git

[ 20061018213306 ]
wmii w/ ruby wmiirc

[ 20061018201907 ]
small fonts

[ 20061007151802 ]
google-codesearch from vim

[ 20060928020813 ]
shell commands

[ 20060920093957 ]
letting screen apps use the ssh-agent

[ 20060908223613 ]
mpdscribble stream support

[ 20060907125149 ]
glGo on ubuntu/dapper amd64

[ 20060906163240 ]
lbdb and mutt

[ 20060902145643 ]
vim tutorial

[ 20060902135722 ]
fixing your terminal

[ 20060828124713 ]
apt-get pdiffs

[ 20060824224842 ]
256 colour xterms

[ 20060824152658 ]
dynamic IPcomp

[ 20060824145428 ]
inkscape++

[ 20060818150516 ]
tags/cscope for system headers

[ 20060805131557 ]
opteron 170, part 4

[ 20060805101941 ]
opteron 170, part 3

[ 20060803233234 ]
opteron 170, part 2

[ 20060802210126 ]
opteron 170

[ 20060729144129 ]
OLS keysigning / 2006

[ 20060728105500 ]
git-find findings

[ 20060727162941 ]
starting on git-find

[ 20060727113632 ]
git-graft and git-find brainstorm

[ 20060726224531 ]
pretty function tracing

[ 20060713174723 ]
uml and multiple network segments

[ 20060707182236 ]
lastfm artist and title to clipboard

[ 20060706162256 ]
reverting a git changeset

[ 20060629212003 ]
user #3

[ 20060628083456 ]
firefox crashes with form input

[ 20060621151402 ]
git vs hg

[ 20060612222204 ]
ldap account management

[ 20060612194523 ]
stupid ldap

[ 20060608092157 ]
rpm hell is right

[ 20060605095726 ]
OpenSSH VPNs

[ 20060604114317 ]
Lenovo lost a customer

[ 20060601234010 ]
generating html colourized sourcecode

[ 20060601211716 ]
ion3 greatness and acting on X selections

[ 20060526085644 ]
software RAID10 performance

[ 20060525234148 ]
learning to love git

[ 20060524121638 ]
recent vim7 articles

[ 20060516095748 ]
bootstrapping debian on my sbc

[ 20060428145140 ]
entropy injection

[ 20060423140628 ]
Adam

[ 20060414202507 ]
converting mp3s to CD

[ 20060413232836 ]
secure apt-get

[ 20060412194423 ]
xen domain0 on debian

[ 20060410220525 ]
LVM2 on RAID1 mirror

[ 20060410102824 ]
building a RHEL4 kernel w/ kdb support

[ 20060407230939 ]
xen on debian

[ 20060407230818 ]
Upgraded look

[ 20060330131334 ]
Flattered by a copy

[ 20060328165153 ]
vim7 from source

[ 20060308123539 ]
Perl, Catalyst, CPAN, and Debian

[ 20060308123302 ]
last.fm

[ 20060128124841 ]
carcassonne and zombies

[ 20060120135931 ]
Election Humour

[ 20050925130002 ]
ldap on debian

[ 20050528190034 ]
Error while mapping shared library sections

[ 20050124130158 ]
IRC over email gateway

[ 20050110225522 ]
brute force attacks sshd?

[ 20050108095026 ]
LDAP authentication (part 1)

[ 20041124130146 ]
sawfish workspace themes

[ 20041113082651 ]
Mini-DV to divx using mencoder

[ 20041004084525 ]
notes on vserver

[ 20040922104334 ]
fast kernel logging

[ 20040803104122 ]
Digital Rebel for sale... GONE

[ 20040714202912 ]
OpenOffice resources

[ 20040603175746 ]
photo editing

[ 20040601082817 ]
WRT54G

[ 20040503205227 ]
Digital Rebel

[ 20040420200136 ]
Open Office templates

[ 20040326082602 ]
bash vi editing mode

[ 20040315204142 ]
debian install CDs

[ 20040312155542 ]
change of jobs

[ 20040308091554 ]
spamassasin extras

[ 20040305163216 ]
cool debian tools

[ 20040305155708 ]
first post!





Bart Trojanowski
http://www.jukie.net/~bart
bart@jukie.net