<<< Back to the Linux Tips Index

30th March 2017

Bad Public Keys

What does "error: RSA_public_decrypt failed: error:0407006A:lib(4):func(112):reason(106)" mean? Read on...

SSH Public Key authentication is reasonably well-understood (or so I thought). You have a private key in id_rsa, a corresponding public key in id_rsa.pub, and if that key is also found in the user's authorized_keys file on the remote server, then you can SSH in to that server, as that user, without being challenged for a password.

I came across a situation recently, where SSH keys were being accepted at times, and not at other times.

FYI: In all of these tests, no changes were made to the server. It had the correct public key in its authorized_keys at all times. When the Public/Private Key login failed, it fell back to a password challenge, and in every situation, although that test is classed here as a "fail", I provided the correct password, and successfully logged in. Just to help you to make sense of the logs. The Public/Private key authentication failed, but the login was successful because a valid password was provided.

Looks like a bad pub to me.

Case One: Normal: Works

The first test, was a normal setup: The client has both id_rsa and id_rsa.pub. Everything worked fine. Good.

Server Log | Client Log

Case Two: No Local Public Key: Works

If you delete the local id_rsa.pub from the client, then the connection still works. That makes sense, right? You only need your private key to authenticate to the server.

Server Log | Client Log

Case Three: Invalid Local Public Key: Works

If the local id_rsa.pub is not a valid SSH Public Key, then it is ignored, and everything continues as in Case Two.

Sorry, I didn't track logs for this, but it looks almost identical to Case 1.

Case Four: Incorrect Local Public Key: Fails

This is the part I really struggled with: If you take the id_rsa.pub from a different public-private key pair, and put it into id_rsa.pub on the client, the connection fails.

If you don't have an id_rsa.pub at all, the connection works; if you have a bad one, it works, but if you have one in a valid format, but which doesn't correspond to the id_rsa (and indeed, doesn't correspond to the remove authorized_keys file), then your connection is rejected.

Server Log | Client Log

It seems that what is happening, is that in Cases Two and Three, there is no valid-looking public key provided, so SSH has to do an extra bit of work. It wouldn't be efficient to do this all the time, but a public key can be derived from its corresponding private key. I didn't know this before, but ssh-keygen -y id_rsa > id_rsa.pub will create id_rsa.pub from id_rsa. It's more computationally expensive, so if you provide a valid-looking id_rsa.pub, SSH will use that, assuming that you know what you are doing. It's a shame that it doesn't, at that point, try to derive the valid public key from the private key, but there you go.

Either way, SSH has a public and a private key to use, and it uses them both in the process of establishing a connection.

An important point to note, is that the client has to say which public key it's wanting to use, because the server will only take that line from authorized_keys; SSH can use keys to say things like "This client can connect, and do these things; another client can connect, and do other things.". So you have to establish who you are attempting to connect as, before starting the authentication phase.

Notes

The logs of all of these tests are in notes directory.

Root Cause Analysis

Frustratingly, the cause of this, in my case, was that Puppet was running periodically, and overwriting the id_rsa file, but not the id_rsa.pub file. I had created a new keypair, which would work for a while, and then stop working (as Puppet put the old id_rsa file in place of mine). Each time, I'd create a new keypair, and copy the public portion to the remote authorized_keys, and everything would work for a while, then stop working. Hopefully my pain will help to save somebody else the time!

The symptom in the log file was this (note: this will be /var/log/auth.log on Debian, Ubuntu, and other derivates; it will be /var/log/secure on RedHat, CentOS, and other derivatives):

Mar 31 00:04:36 centos7 sshd[3266]: error: RSA_public_decrypt failed: error:0407006A:lib(4):func(112):reason(106)
Mar 31 00:04:36 centos7 sshd[3266]: debug1: ssh_rsa_verify: signature incorrect
Mar 31 00:04:36 centos7 sshd[3266]: Failed publickey for jenkins from 192.168.1.99 port 56025 ssh2: RSA 0e:62:63:c9:8f:c4:68:7a:39:e9:87:52:d6:b2:62:dc

Of course, the "debug1: ssh_rsa_verify: signature incorrect" line will only be shown if you have "LogLevel DEBUG" in /etc/ssh/sshd_config.

The image at the top? Well, it looks like a pretty bad pub, to me.

 

Invest in your career. Buy my Shell Scripting Tutorial today:

Steve Parker - Linux / DevOps Consultant
Share on Twitter Share on Facebook Share on LinkedIn Share on Identi.ca Share on StumbleUpon