30th March 2017
Bad Public Keys
What does "error: RSA_public_decrypt failed: error:0407006A:lib(4):func(112):reason(106)" mean? Read on...
SSH Public Key authentication is reasonably well-understood (or so I thought). You have a private key in id_rsa, a corresponding public key in id_rsa.pub, and if that key is also found in the user's authorized_keys file on the remote server, then you can SSH in to that server, as that user, without being challenged for a password.
I came across a situation recently, where SSH keys were being accepted at times, and not at other times.
FYI: In all of these tests, no changes were made to the server. It had the correct public key in its authorized_keys at all times. When the Public/Private Key login failed, it fell back to a password challenge, and in every situation, although that test is classed here as a "fail", I provided the correct password, and successfully logged in. Just to help you to make sense of the logs. The Public/Private key authentication failed, but the login was successful because a valid password was provided.
Looks like a bad pub to me.
Case One: Normal: Works
The first test, was a normal setup: The client has both id_rsa and id_rsa.pub. Everything worked fine. Good.
Case Two: No Local Public Key: Works
If you delete the local id_rsa.pub from the client, then the connection still works. That makes sense, right? You only need your private key to authenticate to the server.
Case Three: Invalid Local Public Key: Works
If the local id_rsa.pub is not a valid SSH Public Key, then it is ignored, and everything continues as in Case Two.
Sorry, I didn't track logs for this, but it looks almost identical to Case 1.
Case Four: Incorrect Local Public Key: Fails
This is the part I really struggled with: If you take the id_rsa.pub from a different public-private key pair, and put it into id_rsa.pub on the client, the connection fails.
If you don't have an id_rsa.pub at all, the connection works; if you have a bad one, it works, but if you have one in a valid format, but which doesn't correspond to the id_rsa (and indeed, doesn't correspond to the remove authorized_keys file), then your connection is rejected.
It seems that what is happening, is that in Cases Two and Three, there is no valid-looking public key provided, so SSH has to do an extra bit of work. It wouldn't be efficient to do this all the time, but a public key can be derived from its corresponding private key. I didn't know this before, but ssh-keygen -y id_rsa > id_rsa.pub will create id_rsa.pub from id_rsa. It's more computationally expensive, so if you provide a valid-looking id_rsa.pub, SSH will use that, assuming that you know what you are doing. It's a shame that it doesn't, at that point, try to derive the valid public key from the private key, but there you go.
Either way, SSH has a public and a private key to use, and it uses them both in the process of establishing a connection.
An important point to note, is that the client has to say which public key it's wanting to use, because the server will only take that line from authorized_keys; SSH can use keys to say things like "This client can connect, and do these things; another client can connect, and do other things.". So you have to establish who you are attempting to connect as, before starting the authentication phase.
Notes
The logs of all of these tests are in notes directory.
Root Cause Analysis
Frustratingly, the cause of this, in my case, was that Puppet was running periodically, and overwriting the id_rsa file, but not the id_rsa.pub file. I had created a new keypair, which would work for a while, and then stop working (as Puppet put the old id_rsa file in place of mine). Each time, I'd create a new keypair, and copy the public portion to the remote authorized_keys, and everything would work for a while, then stop working. Hopefully my pain will help to save somebody else the time!
The symptom in the log file was this (note: this will be /var/log/auth.log on Debian, Ubuntu, and other derivates; it will be /var/log/secure on RedHat, CentOS, and other derivatives):
Mar 31 00:04:36 centos7 sshd[3266]: error: RSA_public_decrypt failed: error:0407006A:lib(4):func(112):reason(106) Mar 31 00:04:36 centos7 sshd[3266]: debug1: ssh_rsa_verify: signature incorrect Mar 31 00:04:36 centos7 sshd[3266]: Failed publickey for jenkins from 192.168.1.99 port 56025 ssh2: RSA 0e:62:63:c9:8f:c4:68:7a:39:e9:87:52:d6:b2:62:dc
Of course, the "debug1: ssh_rsa_verify: signature incorrect" line will only be shown if you have "LogLevel DEBUG" in /etc/ssh/sshd_config.
The image at the top? Well, it looks like a pretty bad pub, to me.
Invest in your career. Buy my Shell Scripting Tutorial today:
