[j-nsp] MX960 power supply stopped during ISSU

[Aaron Gould]

Last night I had a successful ISSU upgrade. BUT. "show chassis alarm" showed
me that PEM0 power supply had issues.  Searching logs didn't turn up any
previous issues so I think that this happened during the ISSU process.
Anyway ever seen something like that before?  I would've thought that a
software upgrade wouldn't do much with power, but I'm wondering now. 

 

17.4R1-S2.2 - old

17.4R2-S1.2 - new

 

agould at blvr-960> show chassis alarms

3 alarms currently active

Alarm time               Class  Description

2019-01-29 00:33:12 CST  Major  PEM 0 Input Failure

2019-01-29 00:33:12 CST  Major  PEM 0 Not OK

2019-01-29 00:32:27 CST  Minor  Backup RE Active

 

.this morning CO Tech went on site and said power feeds to PEM0 were fine,
and no tripped fuzes or anything.  "show chassis power" showed 2 feeds
expected and connected, and good power but not putting anything out.

He removed the bad PEM0 and put it into lab MX960, and it works!

 

Some messages seen were. I wonder what "bump volt" means ?  .wondering if
that is an action to actually hit the voltage of each PEM, and if so, wonder
if that would've tripped on offline.

 

Jan 29 00:32:20  hwdb: entry for cbd 2988 at slot 2 inserted

Jan 29 00:32:20  acb_add: CB 2, initializing SGLS SGLINk type 2 Local ACB
type 4

Jan 29 00:32:20  acb_sglink_init: GE 8374 PHY PMC ctrl 2 : 0xa300 at slot 2

Jan 29 00:32:20  acb_sglink_init: GE 8374 PHY PMC ctrl 2: set TXCLK4 at slot
2

Jan 29 00:32:20  acb_sglink_init: GE 8354 PHY Auto Neg Status 2: 0x2a for
slot 2

Jan 29 00:32:20  acb_sglink_init: GE 8354 PHY is byte aligned for slot 2

Jan 29 00:32:21  acb_sglink_init: GE 8374 PHY Auto Neg Status 2: 0xa0 at
slot 2

Jan 29 00:32:21  acb_sglink_init: GE 8374 PHY is byte aligned at slot 2

Jan 29 00:32:21  acb_sglink_init: CB slot 2 SGLS version 0

Jan 29 00:32:21  acb_sglink_init: CB slot 2 SGLS type 2, acb type 4

Jan 29 00:32:21  acb_add: CB 2, initializing PCIe hub

Jan 29 00:32:21  acb_add: setting CB 2 cache type and i2c 0xbac

Jan 29 00:32:21  ch_probe_frus: Routing Engine 1 added

Jan 29 00:32:21  reading RE 1 initial state

Jan 29 00:32:21  reading host processor dimms

Jan 29 00:32:22  hwdb: entry for re 3087 at slot 1 inserted

Jan 29 00:32:22  ch_probe_frus: PEM 0 added

Jan 29 00:32:22  reading PEM 0 initial state

Jan 29 00:32:22  Bump volt: reset structure for pem 0 during add

Jan 29 00:32:22  ch_probe_frus: PEM 1 added

Jan 29 00:32:22  reading PEM 1 initial state

Jan 29 00:32:22  Bump volt: reset structure for pem 1 during add

Jan 29 00:32:22  ch_probe_frus: PEM 2 added

Jan 29 00:32:22  reading PEM 2 initial state

Jan 29 00:32:22  Bump volt: reset structure for pem 2 during add

Jan 29 00:32:22  ch_probe_frus: PEM 3 added

Jan 29 00:32:22  reading PEM 3 initial state

Jan 29 00:32:22  Bump volt: reset structure for pem 3 during add

Jan 29 00:32:22  ch_probe_frus: FPM 0 added

Jan 29 00:32:22  reading FPM 0 initial state

Jan 29 00:32:22  check_and_carp_on_i2cs_version I2CS version=0x29

 

Jan 29 00:33:12  blvr-960 alarmd[16028]: Alarm set: Pwr supply color=RED,
class=CHASSIS, reason=PEM 0 Not OK

Jan 29 00:33:12  blvr-960 craftd[13352]:  Major alarm set, PEM 0 Not OK

Jan 29 00:33:12  blvr-960 chassisd[13337]: CHASSISD_PEM_INPUT_BAD: status
failure for power supply 0 (status bits: 0x2); check circuit breaker

Jan 29 00:33:12  blvr-960 alarmd[16028]: Alarm set: Pwr supply color=RED,
class=CHASSIS, reason=PEM 0 Input Failure

Jan 29 00:33:12  blvr-960 craftd[13352]:  Major alarm set, PEM 0 Input
Failure

Jan 29 00:33:12  blvr-960 chassisd[13337]: CHASSISD_PEM_INPUT_BAD: Input
failure for power supply 0 (status bits: 0x2); check circuit breaker

Jan 29 00:33:17  blvr-960 chassisd[13337]: CHASSISD_PEM_INPUT_BAD: status
failure for power supply 0 (status bits: 0x2); check circuit breaker

 

Jan 29 00:33:12  send: red alarm set, device PEM 0, reason PEM 0 Not OK

Jan 29 00:33:12 CHASSISD_PEM_INPUT_BAD: status failure for power supply 0
(status bits: 0x2); check circuit breaker

Jan 29 00:33:12  send: red alarm set, device PEM 0, reason PEM 0 Input
Failure

Jan 29 00:33:12 CHASSISD_PEM_INPUT_BAD: Input failure for power supply 0
(status bits: 0x2); check circuit breaker

 

-Aaron