[c-nsp] ssh Crypto key broke ??

Tue Mar 1 04:39:26 EST 2011

We had seen this issue in our network. I believe that CSCtc41114 is the bug you are seeing.

We had a tac case open on this for about half a year as there is basically no testing of any of these "advanced" features in IOS. (You should have seen the epic failures on the part of TAC in supporting us, this one still burns at the heart of my soul).

The lab folks pretty much don't use inband communication with the devices, so don't realize when things like this are broken.

The only way I've seen to "save" your key is to reload without saving the config.  There is no independent mechanical operation to save the config other than doing a "wr mem" or "copy run start".  This has been lacking for a few years now. (eg: you can save the mib-variables in a different operation, but not the crypto keys).

BTW: The workaround was to create a ssh key with a "label", which is fine if you want to change all the keys in your network, as they likely impact all your operators machines with saved ssh keys.

- Jared

(Cisco folk that care: C3 613392533 - Start reading the foolish advice starting at Service Request LOG 2010-01-15 20:40:30.0 GMT, XXCTS_SRM_ADMIN_PROCESS, then stuff like - Service Request LOG 2010-02-06 00:41:34.0 GMT)

On Feb 28, 2011, at 5:00 PM, Jeff Fitzwater wrote:

> If there is a phantom key you can't see it using that the command "show crypto key mypubkey rsa"
> 
> No matter what I do I cant build a key that works.
> 
> I'll open TAC case to see if they can see phantom key in tech dump.
> 
> 
> Jeff
> On Feb 28, 2011, at 16:51 , Vinny_Abello at Dell.com wrote:
> 
>> show crypto key mypubkey rsa
>> 
>> Also, I have found you can change the hostname if you specify the new keypair-name using:
>> 
>> ip ssh rsa keypair-name <keypair-name>
>> 
>> Just be sure to do this after changing the hostname or domain suffix but BEFORE you disconnect. Otherwise new SSH connections will be broken. Test it of course prior to disconnecting your working session. :) The new keypair-name can be seen using the show crypto key mypubkey rsa command, but should be the FQDN  of the device.
>> 
>> I don't know if that process is officially supported or works in all circumstances, but I've stumbled across it has worked for me when renaming devices running SSH. Perhaps it will help you out. Your IOS must also support specifying the ssh rsa keypair-name as well. Not all of them do. SXI should from what I see.
>> 
>> -Vinny
>> 
>> -----Original Message-----
>> From: cisco-nsp-bounces at puck.nether.net [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Jeff Fitzwater
>> Sent: Monday, February 28, 2011 3:40 PM
>> To: Bill Blackford
>> Cc: cisco-nsp at puck.nether.net
>> Subject: Re: [c-nsp] ssh Crypto key broke ??
>> 
>> I just added VRF interface SVI and still have old non-vrf SVI.
>> 
>> If there is a phantom key, that may have been introduced (name unknown), is there a way to see them in some kind of dump?
>> 
>> 
>> Jeff
>> On Feb 28, 2011, at 15:19 , Bill Blackford wrote:
>> 
>>> This could be way off base here, but if changing to a new VRF is 
>>> anything like changing a hostname, then you're require a reboot.
>>> 
>>> -b
>>> 
>>> 
>>> 
>>> On Mon, Feb 28, 2011 at 11:08 AM, Jeff Fitzwater <jfitz at princeton.edu> wrote:
>>>> Running 12.2.33-SXI3 on 6500
>>>> 
>>>> 
>>>> Config had one IP interface.
>>>> 
>>>> Also had SSH enabled with crypto key mod 1024 and all has been working for ever.
>>>> 
>>>> NOW COMES THE CHANGE....
>>>> 
>>>> 
>>>> Had to add new VRF interface.
>>>> 
>>>> Made VTY vrf-aware and added new IP to VTY ACL.
>>>> 
>>>> 
>>>> Initially I could SSH using new IP and OLD.
>>>> 
>>>> About an hour later SSH stopped working with log errors shown below.
>>>> 
>>>> 
>>>> 
>>>> SSH2 1: RSA_sign: private key not found
>>>> SSH2 1: signature creation failed, status -1
>>>> 
>>>> 
>>>> 
>>>> I cleared crypto keys but no luck.
>>>> Also cleared my local .ssh2 hostkeys.
>>>> 
>>>> Also see there is still crypto bug that truncates last character of key name leaving a phantom key.
>>>> 
>>>> -----
>>>> This is fix for bug, but it did not work either...
>>>> 
>>>> ---------  This was note from my CISCO rep.
>>>> 
>>>> For example, our switch was named "switch-core1" with a domain of "ox.com". The fqdn was "switch-core1.ox.com". After the upgrade, the hidden corrupted key was labeled "switch-core1.ox.co".
>>>> 
>>>> The solution is to create a key with the bad label that will overwrite the phantom, then delete it:
>>>> 
>>>> switch-core1(config)#crypto key generate rsa general-keys label 
>>>> switch-core1.ox.co modulus 512 switch-core1(config)#crypto key 
>>>> zeroize rsa switch-core1.ox.co
>>>> 
>>>> and the phantom key will be gone.
>>>> 
>>>> ------------
>>>> 
>>>> 
>>>> 
>>>> 
>>>> Need help... any ideas???
>>>> 
>>>> 
>>>> 
>>>> Jeff Fitzwater
>>>> OIT Network Systems
>>>> Princeton University
>>>> _______________________________________________
>>>> cisco-nsp mailing list  cisco-nsp at puck.nether.net 
>>>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>>>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Bill Blackford
>>> Network Engineer
>>> 
>>> Logged into reality and abusing my sudo privileges.....
>> 
>> 
>> _______________________________________________
>> cisco-nsp mailing list  cisco-nsp at puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
> 
> 
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/