2 Jul 2015
Fork Bomb!
Using shell builtin commands to deal with a fork-bombed machine
A few days ago I had to deal with my first ever real-life fork-bombed server.
By logging in to the console, I was somehow able to get a shell (one process). Having got that shell, even though I was root, it was difficult to be able to spawn other processes. It turned out that this was because we had restricted the CPU count on the kernel command line (maxcpus=2) so that a dual processor, 16-core machine had only one eighth of its processing power available. The dynamic change to the nproc value does not take this into account, so this unprivileged user was able to fork-bomb the entire machine.
The first thing you might want to do in this situation is to run ps -eaf. That’s another process, and even as root, you don’t get to do it. Being Linux, you can see how many processes exist on the system by listing /proc:
$ cd /proc $ echo *
Neither of these commands spawn a new shell, they are both shell builtin commands, so they will work. In this case, with over 69,000 processes, I killed the output before I got too bored. Since there are usually around 200 processes running, that was enough to tell me that something was wrong.
After many attempts, a ps command did work, and confirmed that a certain shell script was being run a lot of times. I couldn’t cat that file, and didn’t even have its full name (ps truncates output to match the terminal’s width; you can bypass this by piping the output to cat, but that involves spawning yet another process). I had the PID, so /proc/$PID/fd gave the filename.
It’s not possible to cat the script to see what it’s doing, so more builtin commands are required. This loop displays the contents of a file without spawning any further processes:
$ while read f > do > echo $f > done < /path/to/script.sh
This uses all shell-builtin commands (without spawning an extra cat command), and tells you the full content of the script. From there, you may have some insight into what it is doing, and how to stop it.
Invest in your career. Buy my Shell Scripting Tutorial today:
(ported from my nixshell blog)
