[c-nsp] Cisco 7600/RSP720 high fabric channel utilization

[James Bensley]

On 30 October 2017 at 14:52, Валерий Н. Лутошкин <lvn at nts.su> wrote:
> Hi,
>
> I have two identical 7609 with RSP720-3CXL-10G.
>
> "show platform hardware capacity fabric" command on any of 7609 gives me a similar output:
>
>   Fabric utilization:     Ingress                    Egress
>     Module  Chanl  Speed  rate  peak                 rate  peak
>     1       0        20G   63%   70% @22:41 29Oct17   35%   51% @20:35 15Oct17
>     1       1        20G   24%   39% @21:25 29Sep16   41%   59% @21:17 21Dec16
>     2       0        20G   18%   37% @21:34 16Sep17   53%   69% @21:33 16Sep17
>     2       1        20G   33%   57% @03:21 14Oct17   77%   90% @20:32 27Oct17
>     3       0        20G   66%   97% @03:22 14Oct17   47%   67% @20:32 06Jul17
>     3       1        20G   59%   98% @11:21 04Aug17   31%   84% @20:42 01Feb17
>     5       0        20G   71%   89% @21:54 20Oct17   83%   98% @22:24 29Oct17
>     5       1        20G    0%    4% @10:45 17Jan17    0%   30% @10:03 25Jul16
>
> I wonder why the RSP uses only one channel as fabric connection.
> As you can see, it was overloaded (98% - it is overloading) yesterday.
>
> Is there any way to balance traffic between two fabric channels?
>
> Best regards,
> Valery Lutoshkin


Hi Valery,

I'm happy to be corrected but I think the answer is "no". The
RSP720-3CXL-10G has 3x1Gbps ports and 2x10Gbps ports, so having them
mapped to a single 20Gbps fabric channel is only slightly overscribed
at 23Gbps:20Gbps (1.15:1).

If you check on your box you will see all the ports on the RSP card
are mapped to the same fabric channel (you should see output similar
to below):


7606-S-RSP720-3CXL-10G#show fabric fpoe map

slot   channel   logical fpoe    physical fpoe
 1        0             0                0
 1        1             1                9
 2        0             2                1
 2        1             3                10
 3        0             4                2
 3        1             5                11
 4        0             6                6
 4        1             7                15
 5        0             8                18  < Slot 5 is an RSP720-3CXL-10G
 5        1             9                19  < Slot 5 is an RSP720-3CXL-10G
 6        0             10               5
 6        1             11               14

7606-S-RSP720-3CXL-10G#show fabric fpoe slot 5 port 1
logical fpoe for 5/1 is 8, physical fpoe:18

7606-S-RSP720-3CXL-10G#show fabric fpoe slot 5 port 2
logical fpoe for 5/2 is 8, physical fpoe:18

7606-S-RSP720-3CXL-10G#show fabric fpoe slot 5 port 3
logical fpoe for 5/3 is 8, physical fpoe:18

7606-S-RSP720-3CXL-10G#show fabric fpoe slot 5 port 4
logical fpoe for 5/4 is 8, physical fpoe:18

7606-S-RSP720-3CXL-10G#show fabric fpoe slot 5 port 5
logical fpoe for 5/5 is 8, physical fpoe:18

7606-S-RSP720-3CXL-10G#show platform hardware capacity fabric
...
    5       0        20G    0%   11% @10:03 04Oct17    3%   28% @09:04 14Jul17
    5       1        20G    0%    0%                   0%    5% @11:32 22Dec16


I don't believe any command exists to remap ports to fabric channels
(because I think it's a physical circuit mapping)

7606-S-RSP720-3CXL-10G#show asic-version slot 5
Module in slot 5 has 8 type(s) of ASICs
        ASIC Name      Count      Version
             KUMA          1      (3.0)
      METRO_ARGOS          1      (3.0)
    METRO_KRYPTON          1      (3.0)
              SSA          2      (9.0)
       SANTA_CRUZ          1      (3.0)
          TELESTO          1      (7.0)
          KAILASH          1      (2.4)
             R2D2          2      (3.0)


There are 2x R2D2 port ASICs in the card, presumably one for each 10G
port as I believe they are 10G rated ASICs, and no ROHINI ASIC which
is commonly used for 1Gbps ports, so the 1Gbps ports must also feed
into those R2D2 ASICs some how. So that could indicate some
oversubscription on the card. There are 2 SSA chips which are 20Gbps
fabric channel chips which is why I guess that card shows as having
two fabric channels however I also think the Telesto ASIC is related
to fabric connectivity and there is only one. Again I'm not sure of
the impact of this though but it doesn't look good for getting wire
rate out of all ports on the RSP card.

Cheers,
James.