[c-nsp] downlink bgp interconnect best practices

[Pete Templin]

On 5/31/2011 5:50 AM, Gert Doering wrote:
> We try to separate "core + uplink" and "customer connection" routers,
> so we can do works on core routers witout affecting customers - and
> vice versa, if we have to reboot a "customer connection" router, we
> know which customers are affected and that nothing else is.
>
> So a "text book" scenario would have two "core/uplink" routers here,
> fully meshed with two "customer access" boxes (so there's no single
> switch in between that could break), and customers are then connected
> to one or both of the "access" routers.
>
> uplink/IX  uplink/IX
>      |        |
>      CR1 -- CR2
>       | \  / |
>       |  \/  |
>       |  /\  |
>       | /  \ |
>      AR1    AR2
>       |      |
>       customers
>
Having "learned" in a multi-pop environment, I learned to separate into 
three groupings: "edge" routers for upstream transit/peer connections, 
"distribution" for downstream customer connections, and "core" as the 
glue that holds everything together.  Lots of little reasons why:

If another city's WAN link comes into CR1 and you have an uplink there, 
that city's outbound traffic will want to leave via the uplink there 
unless you really motivate BGP to punt the traffic to CR2.  In other 
words, if BGP doesn't make a decision before step 9 (Prefer the external 
BGP path over the internal BGP path), all of your traffic arriving on 
CR1 is leaving by CR1's upstream(s).

If you're using Cat6500s, separating the edge layer means you can use 
loose-mode uRPF without bothering the customer layer.

If you're using soft-reconfig, a memory issue doesn't blow up your 
core.  (Yes, there's a newer way, but if you want to see exactly what 
the neighbor is sending you pre-filtration...or, just in case someone 
accidentally turns the feature on.)

On a related note, if AR1 and AR2 are the same platform (or if AR3/AR4 
are the same platform, etc.), we'd usually do RSP/SUP redundancy in AR1 
and not in AR2.  Customers with one/odd connection(s) go on AR1 (for the 
odd-numbered connections), customers with an even number of connections 
get half on AR2.

pt