The boot loader that phoned home

My network boot server has a habit: every time I actually need it, it's broken. Not down — broken in a specific, boring way that takes twenty minutes to rediscover. This time I stopped fixing the symptom and went looking for why it kept coming back.

The complaint was "it isn't authenticating." That's an odd thing for a PXE server to do, since nothing in the chain asks for a password.

What the logs actually said

Stage one was flawless. The client's firmware broadcast a DHCP discover, the proxyDHCP responded, and TFTP handed over the iPXE binary:

PXE(eth0) 00:e0:4c:xx:xx:xx proxy
tags: efi64, eth0
bootfile name: ipxe.efi
...
dnsmasq-tftp: sent /config/menus/ipxe.efi to 10.0.0.109

And then nothing. No HTTP request for the boot menu. Not a failed request — no request at all. Whatever iPXE did after it loaded, it wasn't talking to my server.

The two-stage trick, and its load-bearing assumption

A PXE chainload has two stages that look identical from the outside and are completely different underneath:

The NIC firmware asks DHCP for a boot file, gets ipxe.efi, TFTPs it, runs it.
iPXE itself now asks DHCP the same question — and if it gets the same answer, it downloads and runs itself again, forever.

Everyone solves stage two the same way: tag the second request and answer it differently. iPXE announces itself with user-class iPXE and DHCP option 175, so dnsmasq can match on that and hand back an HTTP menu URL instead of a binary:

dhcp-match=set:ipxe,175
dhcp-userclass=set:ipxe,iPXE
pxe-service=tag:ipxe,x86-64_EFI,"local menu",http://10.0.0.5/menu.ipxe

That works. It also means the health of your boot server depends on a tag match inside a second DHCP conversation you never see, on a network where a router firmware update can quietly wipe the options out from under you.

Where "authentication" came from

Here's the part I'd missed for months. The binary I was serving was the stock upstream release, and stock iPXE builds ship with a chain URL compiled in: https://boot.netboot.xyz/menu.ipxe.

So when the stage-two tag match failed, iPXE didn't error out. It fell back to its built-in URL, went out to the internet over TLS, failed to validate the certificate chain, and printed:

Permission denied (https://ipxe.org/0216...)

Permission denied. To a person standing at a machine that won't boot, that reads exactly like an authentication failure — which is why every previous debugging session started in the wrong place. The message described the fallback path, not the path I'd configured.

The fix: stop asking

The real problem isn't the tag match. It's that the boot loader had an opinion about where the menu lives, and my DHCP server had to argue it out of that opinion on every boot. Remove the argument: compile the answer into the binary.

$ cat embed.ipxe
#!ipxe
set attempt:int32 0
:top
inc attempt
dhcp || goto retry
chain --autofree http://10.0.0.5/menu.ipxe || goto retry
exit 0
:retry
iseq ${attempt} 4 && goto giveup ||
sleep 3
goto top
:giveup
echo Could not reach the local boot menu - dropping to iPXE shell
shell

$ git clone --depth 1 https://github.com/ipxe/ipxe
$ cd ipxe/src
$ make -j4 EMBED=/path/to/embed.ipxe \
       bin-x86_64-efi/ipxe.efi bin-i386-efi/ipxe.efi bin/undionly.kpxe

Ten minutes of build time. Now iPXE still does DHCP — it needs an address — but it no longer asks where to go. There is no fallback to the public internet, no TLS, no certificate chain, and nothing for a router firmware update to break. Worst case it can't reach my menu and drops to a shell that says so, instead of producing a misleading error about permissions.

Two things I'd have done sooner:

Build the BIOS and 32-bit EFI binaries too. I'd fixed the x86-64 path months ago and left the other pxe-service lines pointing at stock binaries, which meant the same trap was still armed for half my clients.
Assert the artifact, not the service. The old monitoring checked that the container was up and TFTP answered. Both were true the entire time it was broken. The check that would have caught this greps the binary the server is actually handing out for the URL it's supposed to contain:

bash curl -s --max-time 20 -o "$T" tftp://10.0.0.5/ipxe.efi || fail "TFTP fetch failed" grep -qa "10.0.0.5/menu.ipxe" "$T" || fail "binary has no embedded local menu URL"

"Is the service running" and "does the service do the thing" are different questions, and only one of them was ever going to catch a boot loader that phoned home.