<<< Back to Tips Index

2 Jul 2015

Fork Bomb!

Using shell builtin commands to deal with a fork-bombed machine

A few days ago I had to deal with my first ever real-life fork-bombed server.

By logging in to the console, I was somehow able to get a shell (one process). Having got that shell, even though I was root, it was difficult to be able to spawn other processes. It turned out that this was because we had restricted the CPU count on the kernel command line (maxcpus=2) so that a dual processor, 16-core machine had only one eighth of its processing power available. The dynamic change to the nproc value does not take this into account, so this unprivileged user was able to fork-bomb the entire machine.

The first thing you might want to do in this situation is to run ps -eaf. That’s another process, and even as root, you don’t get to do it. Being Linux, you can see how many processes exist on the system by listing /proc:

$ cd /proc
$ echo *

Neither of these commands spawn a new shell, they are both shell builtin commands, so they will work. In this case, with over 69,000 processes, I killed the output before I got too bored. Since there are usually around 200 processes running, that was enough to tell me that something was wrong.

After many attempts, a ps command did work, and confirmed that a certain shell script was being run a lot of times. I couldn’t cat that file, and didn’t even have its full name (ps truncates output to match the terminal’s width). I had the PID, so /proc/$PID/fd gave the filename.

It’s not possible to cat the script to see what it’s doing, so more builtin commands are required.

$ while read f
> do
>   echo $f
> done < /path/to/script.sh

This uses all builtin commands, and tells you what the script is. From there, you may have some insight into what it is doing, and how to stop it.

Invest in your career. Buy my Shell Scripting Tutorial today:

 

(ported from my nixshell blog)

Steve's Bourne / Bash shell scripting tips
Share on Twitter Share on Facebook Share on LinkedIn Share on Identi.ca Share on StumbleUpon
Join the Facebook page for more tips and tricks:


Buy my Shell Scripting Tutorial

(Free Sample)
(more info)

$9.99 US Dollars

£4.99 UK Pounds

€6.99 Euros