Email Address Password
Remember Me

Or Create a (Free) Account.
2004JanFebMarAprMayJunJul Aug Sep Oct Nov Dec
2005 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
2006 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Oct Oct

Sat 30th Apr 2011 @ 00:34 2011: RPM Reporter

shell bookI have been meaning to write something like this for a long time; the upcoming Shell Scripting Recipes book has forced me into it. It only works with Bash version 4 or above, as it uses associative arrays (it would be fairly easy to convert it to work without associative arrays, but this is partly about having fun with the things that Bash is capable of beyond straight POSIX compliance, and associative arrays are so much more flexible, even if we don't have multi-dimensional arrays yet.

The script to produce this is 114 lines, including all of the HTML and CSS necessary for the HTML report; it also produces messages to stdout, like this:

RPM MATCH: yum-updatesd 0.9-2.el5
RPM yum-utils
node1 : Not Installed
node2 : 1.1.16-13.el5_4.1
node3 : Not Installed
node4 : Not Installed
node5 : Not Installed

So you can grep for "RPM MATCH:" or for "^RPM " |grep -v "MATCH:" to get the packages which do or do not match. Of course, the point of these recipes is that you can take the code and do what you want with it, so this output can be whatever suits the task at hand, and you can make it as greppable as you like.

RPM can be a tricky thing to handle; these are all valid RPM names:

The key is to deal with that cleanly, in a way which provably works, in its own separate function. Everything else is then safe in not having to worry about the details of how RPMs are named.

The other big thing about this script is keeping data formats and output formats separate; in the screenshot below, you see that (although rpm -qa does not provide all of the detail), there are multiple versions of alsa-lib installed. These are displayed on separate lines for readability, but the "
" code is only added when the HTML output is generated. The raw data, that there are multiple versions of this RPM installed, is not tainted by any assumption that the output will be in HTML.

I have written a few scripts and programs like this over the years, and dealt with quite a few more. Most are limited to comparing two systems, or are hard-coded to some arbitrary limit of maybe five, which it would be difficult to change, as the references to node1, node2, node3, node4, node5 would all have to be changed to add node6, node7 and node8.
By using arrays, this script is totally flexible in either direction; the number of hosts that it can report on, and the number of RPMs it can deal with. Of course, you could in theory run out of memory, but these 5 servers are represented in 97Kb of `rpm -qa` output.

The HTML output shows a plain white background for packages which match on all servers, or a grey background for those which differ. It would be trivial to add extra CSS and scripting for "lowest revision", "highest revision", "odd one out" and so on.
RPM Comparisons

If the screenshot looks distorted, it is probably because it is 993 pixels wide with 5 servers compared. Browsers may shrink images down without sorting the proportion; if it looks odd, right-click and view the image in a new window.

Post a Comment               

Sun 24th Apr 2011 @ 23:16 2011: Manchester Passion

Five years ago, the BBC produced the Manchester Passion.
It is available for viewing at (the Musical Director). Thanks to Google Video, it is also available here:

This is the BBC Web page about the event

  • Jesus sings You're Gonna Need Someone On Your Side (Morrissey)
  • Mary sings Cast No Shadow (Oasis)
  • Jesus sings Love Will Tear Us Apart (Joy Division) at the Last Supper.
  • Mary sings Search For A Hero (M People)
  • Judas sings Heaven Knows I’m Miserable Now (The Smiths) as he is about to betray Jesus.
  • Jesus sings Sit Down (James) to the sleeping disciples
  • Jesus and Judas sing Blue Monday (New Order)
  • Peter sings I Am The Resurrection (The Stone Roses) when he betrays Jesus
  • Mary sings Angels (Robbie Williams)
  • Jesus and Pontius Pilate sing Wonderwall (Oasis)
  • Mary sings Sunshine After The Rain (Elkie Brooks)

If you only have a short amount of time, watch the first few minutes then skip forward to 41 minutes for Jesus and Pilate.

Post a Comment               

Tue 19th Apr 2011 @ 00:23 2011: Friar Tuck!

shell bookThe Jargon File describes the meaning of "hack" in hacker culture. One of the tales it tells is about Robin Hood and Friar Tuck.

Back in the mid-1970s, several of the system support staff at Motorola discovered a relatively simple way to crack system security on the Xerox CP-V timesharing system. Through a simple programming strategy, it was possible for a user program to trick the system into running a portion of the program in ‘master mode’ (supervisor state), in which memory protection does not apply. The program could then poke a large value into its ‘privilege level’ byte (normally write-protected) and could then proceed to bypass all levels of security within the file-management system, patch the system monitor, and do numerous other interesting things. In short, the barn door was wide open.

Motorola quite properly reported this problem to Xerox via an official ‘level 1 SIDR’ (a bug report with an intended urgency of ‘needs to be fixed yesterday’). Because the text of each SIDR was entered into a database that could be viewed by quite a number of people, Motorola followed the approved procedure: they simply reported the problem as ‘Security SIDR’, and attached all of the necessary documentation, ways-to-reproduce, etc.

The CP-V people at Xerox sat on their thumbs; they either didn't realize the severity of the problem, or didn't assign the necessary operating-system-staff resources to develop and distribute an official patch.

Months passed. The Motorola guys pestered their Xerox field-support rep, to no avail. Finally they decided to take direct action, to demonstrate to Xerox management just how easily the system could be cracked and just how thoroughly the security safeguards could be subverted.

They dug around in the operating-system listings and devised a thoroughly devilish set of patches. These patches were then incorporated into a pair of programs called ‘Robin Hood’ and ‘Friar Tuck’. Robin Hood and Friar Tuck were designed to run as ‘ghost jobs’ (daemons, in Unix terminology); they would use the existing loophole to subvert system security, install the necessary patches, and then keep an eye on one another's statuses in order to keep the system operator (in effect, the superuser) from aborting them.

One fine day, the system operator on the main CP-V software development system in El Segundo was surprised by a number of unusual phenomena. These included the following:
  • Tape drives would rewind and dismount their tapes in the middle of a job.
  • Disk drives would seek back and forth so rapidly that they would attempt to walk across the floor (see walking drives).
  • The card-punch output device would occasionally start up of itself and punch a ‘lace card’ (card with all positions punched). These would usually jam in the punch.
  • The console would print snide and insulting messages from Robin Hood to Friar Tuck, or vice versa.
  • The Xerox card reader had two output stackers; it could be instructed to stack into A, stack into B, or stack into A (unless a card was unreadable, in which case the bad card was placed into stacker B). One of the patches installed by the ghosts added some code to the card-reader driver... after reading a card, it would flip over to the opposite stacker. As a result, card decks would divide themselves in half when they were read, leaving the operator to recollate them manually.

Naturally, the operator called in the operating-system developers. They found the bandit ghost jobs running, and killed them... and were once again surprised. When Robin Hood was gunned, the following sequence of events took place:

!X id1

id1: Friar Tuck... I am under attack! Pray save me!
id1: Off (aborted)

id2: Fear not, friend Robin! I shall rout the Sheriff
of Nottingham's men!

id1: Thank you, my good fellow!

Each ghost-job would detect the fact that the other had been killed, and would start a new copy of the recently slain program within a few milliseconds. The only way to kill both ghosts was to kill them simultaneously (very difficult) or to deliberately crash the system.

Finally, the system programmers did the latter — only to find that the bandits appeared once again when the system rebooted! It turned out that these two programs had patched the boot-time OS image (the kernel file, in Unix terms) and had added themselves to the list of programs that were to be started at boot time (this is similar to the way Windows viruses propagate).

The Robin Hood and Friar Tuck ghosts were finally eradicated when the system staff rebooted the system from a clean boot-tape and reinstalled the monitor. Not long thereafter, Xerox released a patch for this problem.

It is alleged that Xerox filed a complaint with Motorola's management about the merry-prankster actions of the two employees in question. It is not recorded that any serious disciplinary action was taken against either of them.

This is the kind of thing that people discuss when talking about clusters, and in particular, the comment is often made that 90% of what most commercial clustering products do could be written in an afternoon with the shell; it is just the final 10%, dealing with failure fencing and forced reboots that is the troublesome part.

Well, I can now report that the process monitoring side of things alone seems to take a good few hours to write and get working. This should all be in Chapter 21 of the shell scripting recipes book which is due out in August. I have provisionally named Friar Tuck's role as friartuck; I assume from the original American reference quoted above that it will be understandable to a USA-based audience. Also I think that it's good to have a little bit of levity in such things; process names like "watchdog" and "monitor" are so dull!

Post a Comment               

Sat 16th Apr 2011 @ 23:11 2011: My first Arduino Project

I had promised myself that I would focus on the book before playing with the Arduino but I gave in today, and wrote this simple "sketch". It uses a basic wiring of 8 LEDs in pins 2-9, with 560-ohm resistors attached to each one.

It alternates between counting 0-255 in binary, and doing a simple Knight Rider style flashing LEDs.

// counts bits on 8 LEDs
int ledPins[]={2,3,4,5,6,7,8,9};
int sleeptime=100;
int countsleep=20;
#define CHECK_BIT(var,pos) ((var) & (1<<(pos)))

void setup()
for (int i=0; i<8; i++) {

void countup(int cs)
for (int i=0; i<8; i++)
digitalWrite(ledPins[i], HIGH);
if (i>0) digitalWrite(ledPins[i-1],LOW);

void countdown(int cs)
for (int i=7; i>=0; i--)
if (i<7) digitalWrite(ledPins[i+1],LOW);
delay (cs*countsleep);

void shownum(int n)
for (int i=0; i<8; i++)
if (CHECK_BIT(n,i)) digitalWrite(ledPins[i], HIGH);
else digitalWrite(ledPins[i],LOW);

void loop()
int i;
for (i=0; i<255; i++) {
for (i=8; i>0; i--)

Post a Comment               

Tue 12th Apr 2011 @ 23:21 2011: umop apisdn

umop apisdn is "upside down", upside down

Post a Comment               

Mon 11th Apr 2011 @ 00:09 2011: Proprietary software and Piracy

I am so glad to be spend most of my digital life in the Free (and Open Source) Software community. I can't really see how the Proprietary Software system is supposed to work, other than depending on users pirating their software in order to gain a monopoly (or rather, to emulate the pre-existing Free Software ecosystem whereby users are used to your software because they can get it for free). There was an article posted on Slashdot last week; I don't often read Slashdot these days, but the discussion was quite interesting, as was the article itself.

I just stumbled upon, which claims to have a tool to thwart the Windows Genuine Advantage anti-piracy software. This reminded me of a lot of the comments from the article above. The page says, under a heading of "Virus scanner results", "All virus scanner detection's[sic] are a false-positive. Simply turn off your anti-virus while installing the loader."

This is closed-source, unverifiable software written by a developer who clearly has no interest in honouring copyright law, telling people to trust their work over the anti-virus software which does indeed sometimes detect false positives. The user has no way of verifying what is true any longer. The old saying about "honour amongst thieves" comes to mind; the OS vendor, the antivirus developer and the "free" (in all senses other than the one truly necessary to its user) hack to the OS all have their own self-interest, and (other than executable code, whether for a dollar price or not) publish nothing but what the user wants to hear.

The user has no way of knowing whether or not this code is trustworthy, and should now have alarm bells ringing that other, independent software flags it as a virus. These users are used to downloading unvalidated software from random web sites, and are also used to false alarms from antivirus software, so a fair number of people could be lead into following these instructions. I have no idea whether or not this will achieve the goal they desire, or infect their machine, and nor do they.

With Free Software (Debian specifically, but the same goes for most other GNU/Linux distributions) I have a single channel which provides tens of thousands of applications, complete with source code. If I am at all unsure of an application, I can download the source and inspect it myself before installing, or even change it before installing it.

And on top of that, I don't need to bypass some restriction tool imposed by the OS in the first place, because my OS is also Free (and free) so it does not need to impose any hidden restrictions upon me. Why would anybody choose any other way of computing?

Post a Comment               

Steve's urandom blog
Share on Twitter Share on Facebook Share on LinkedIn Share on Share on StumbleUpon
My Shell Scripting Book:
    Shell Scripting, Expert Recipes for Linux, Bash and more
is available online and from all good booksellers: