[c-nsp] Cisco 7600/RSP720 high fabric channel utilization

Mon Oct 30 22:41:45 EDT 2017

>-----Original Message-----
>From: James Bensley [mailto:jwbensley at gmail.com] 
>Sent: Monday, October 30, 2017 11:33 PM
>To: cisco-nsp at puck.nether.net; Валерий Н. Лутошкин
>Subject: Re: [c-nsp] Cisco 7600/RSP720 high fabric channel utilization
>
>On 30 October 2017 at 14:52, Валерий Н. Лутошкин <lvn at nts.su> wrote:
>> Hi,
>>
>> I have two identical 7609 with RSP720-3CXL-10G.
>>
>> "show platform hardware capacity fabric" command on any of 7609 gives me a similar output:
>>
>>   Fabric utilization:     Ingress                    Egress
>>     Module  Chanl  Speed  rate  peak                 rate  peak
>>     1       0        20G   63%   70% @22:41 29Oct17   35%   51% @20:35 15Oct17
>>     1       1        20G   24%   39% @21:25 29Sep16   41%   59% @21:17 21Dec16
>>     2       0        20G   18%   37% @21:34 16Sep17   53%   69% @21:33 16Sep17
>>     2       1        20G   33%   57% @03:21 14Oct17   77%   90% @20:32 27Oct17
>>     3       0        20G   66%   97% @03:22 14Oct17   47%   67% @20:32 06Jul17
>>     3       1        20G   59%   98% @11:21 04Aug17   31%   84% @20:42 01Feb17
>>     5       0        20G   71%   89% @21:54 20Oct17   83%   98% @22:24 29Oct17
>>     5       1        20G    0%    4% @10:45 17Jan17    0%   30% @10:03 25Jul16
>>
>> I wonder why the RSP uses only one channel as fabric connection.
>> As you can see, it was overloaded (98% - it is overloading) yesterday.
>>
>> Is there any way to balance traffic between two fabric channels?
>>
>> Best regards,
>> Valery Lutoshkin
>
>
>Hi Valery,
>
>I'm happy to be corrected but I think the answer is "no". The RSP720-3CXL-10G has 3x1Gbps ports and 2x10Gbps ports, so having them mapped to a single 20Gbps fabric channel is only slightly overscribed at 23Gbps:20Gbps (1.15:1).
>
>If you check on your box you will see all the ports on the RSP card are mapped to the same fabric channel (you should see output similar to below):
>
>
>7606-S-RSP720-3CXL-10G#show fabric fpoe map
>
>slot   channel   logical fpoe    physical fpoe
> 1        0             0                0
> 1        1             1                9
> 2        0             2                1
> 2        1             3                10
> 3        0             4                2
> 3        1             5                11
> 4        0             6                6
> 4        1             7                15
> 5        0             8                18  < Slot 5 is an RSP720-3CXL-10G
> 5        1             9                19  < Slot 5 is an RSP720-3CXL-10G
> 6        0             10               5
> 6        1             11               14
>
>7606-S-RSP720-3CXL-10G#show fabric fpoe slot 5 port 1 logical fpoe for 5/1 is 8, physical fpoe:18
>
>7606-S-RSP720-3CXL-10G#show fabric fpoe slot 5 port 2 logical fpoe for 5/2 is 8, physical fpoe:18
>
>7606-S-RSP720-3CXL-10G#show fabric fpoe slot 5 port 3 logical fpoe for 5/3 is 8, physical fpoe:18
>
>7606-S-RSP720-3CXL-10G#show fabric fpoe slot 5 port 4 logical fpoe for 5/4 is 8, physical fpoe:18
>
>7606-S-RSP720-3CXL-10G#show fabric fpoe slot 5 port 5 logical fpoe for 5/5 is 8, physical fpoe:18
>
>7606-S-RSP720-3CXL-10G#show platform hardware capacity fabric ...
>    5       0        20G    0%   11% @10:03 04Oct17    3%   28% @09:04 14Jul17
>    5       1        20G    0%    0%                   0%    5% @11:32 22Dec16
>
>
>I don't believe any command exists to remap ports to fabric channels (because I think it's a physical circuit mapping)
>
>7606-S-RSP720-3CXL-10G#show asic-version slot 5 Module in slot 5 has 8 type(s) of ASICs
>        ASIC Name      Count      Version
>             KUMA          1      (3.0)
>      METRO_ARGOS          1      (3.0)
>    METRO_KRYPTON          1      (3.0)
>              SSA          2      (9.0)
>       SANTA_CRUZ          1      (3.0)
>          TELESTO          1      (7.0)
>          KAILASH          1      (2.4)
>             R2D2          2      (3.0)
>
>
>There are 2x R2D2 port ASICs in the card, presumably one for each 10G port as I believe they are 10G rated ASICs, and no ROHINI ASIC which is commonly used for 1Gbps ports, so the 1Gbps ports must also feed into those R2D2 ASICs some how. So that could indicate some oversubscription on the card. >There are 2 SSA chips which are 20Gbps fabric channel chips which is why I guess that card shows as having two fabric channels however I also think the Telesto ASIC is related to fabric connectivity and there is only one. Again I'm not sure of the impact of this though but it doesn't look good for getting wire >rate out of all ports on the RSP card.
>
>Cheers,
>James.

Hi, James,

Thank you for your detailed answer.

I realized that I have too much traffic on the fabric channel. I combined 2*10G on RSP in LAG and the peak traffic in that LAG is 10/6Gbps in/out. It couldn't overload 20G duplex fabric channel.
I also have 2 cards without DFC, but the summary traffic from them is about 3-4Gbps duplex. So I need to find an explanation for that huge amount of traffic from fabric.

I'm going to start from IOS upgrade :) I've heard smth about high fabric load with multicast on young versions of IOS15, and I still have 15.1(3)S3.

Thank you again.

Best regards, 
Valery Lutoshkin