[c-nsp] Latest iteration of core upgrade - questions

[Rick Ernst]

On Thu, Oct 29, 2009 at 10:53 PM, Mark Tinka <mtinka at globaltransit.net>wrote:

> On Friday 30 October 2009 12:21:02 am Rick Ernst wrote:
>
> > - We do have some peering, but it was originally designed
> > at the customer/aggregation layer.
>
> Do you mean "at" or "in"?
>
As in, do you have a dedicated peering router connected to
> your edge layer, or do you have an edge router doubling as
> your peering router?
>
> ---  It was an "in", but now it's "at".  I can still argue it being
appropriate as a border/"upstream" device and also as
aggregation/"customer".


 > - The idea for the 7206s is as "lightbulb" devices.  One
> > upstream. One 7206. Two downlinks to the core.  The
> > single-point-of-failure remains within the individual
> > upstreams.
>
> Or the box itself, since with the exception of the power
> supplies, it has a single, integrated control and data
> plane.
>
> If you have the budget in the future, get a second router
> and terminate your other upstream there, for border router +
> upstream redundancy.
>
>
> This keeps max possible traffic within the
> > CPU/performance envelope. It also allows us to grow
> > horizontally as additional upstreams come in.  I'm
> > looking at going to 7201s(? the 1U NPE-G2 equivalent) as
> > bandwidth needs increase.
>
> 7201's might not be dense enough if you need to support
> additional Ethernet or non-Ethernet links. You can only use
> one additional PA.
>
> If you do decide to go with the 7201 and later realize
> you're out of ports, you'll be inclined to plant an Ethernet
> switch in there, stick the upstreams into it and run 802.1Q
> back to the 7201. This may or may not be ugly, depending on
> who's looking :-).
>
> --- One 720x per upstream, split into dual cores.  We had also considered
landing upstreams directly on the 7600s, but this allow for a core device
failure without losing upstream capacity.  Additionally, the 720x becomes a
"fuse" and intentional bottleneck for weird traffic.



> > - 7600/Sup720-3BXL is the top (currently only) contender
> > for core routing/switching.
>
> If you're talking about a collapsed edge router + core
> switch, then there are other options, even non-Cisco. But
> I'm guessing you're more Cisco-inclined :-). Shop around, if
> you can. There's always time to make the right decision :-).
>
> As for the 7600, be sure to consider all the features it can
> and can't support, and match those against what your current
> and future plans are.
>
> Talk to your Cisco SE on this until you're satisfied. Once
> these boxes are in, getting them out won't be easy. And that
> goes for all other options you may have.
>

--- I've looked at other vendors, but a big reason for sticking with Cisco
is we have the in-house knowledge.  Changing vendors while re-architecting a
production network seems to be a bad idea.


> > - I was planning on having an "core/border" and
> > "core/aggregation" VLAN on the 7600s.
>
> This is typical - in larger PoP's, both these functions
> would sit on different switches - so you end up having 4
> core switches with redundancy.
>
> In smaller PoP's, 2 core switches can collapse both these
> functions with redundancy, and then you may grow to 4 if
> necessary.
>

--- What is the benefit in having 4 devices instead of 2?  It seems like
you'd just be passing the same traffic through double the number of devices.


> As your network gets bigger and you have more peering and
> less transit, you'll find that you'll probably only need the
> 2. But that's a different level in the game :-).
>
> > Our customer TDM
> > needs are drying up and eveverything is moving to
> > ethernet.  New customer aggregation is Catalyst 4948s
> > with local-only BGP and OSPF.  Customers requiring BGP
> > ebgp-multi-hop to devices that are full-table capable.
>
> We tend to shy away from eBGP Multi-Hop as much as we can,
> but it's used a great deal in the field. Besides, it's a
> good way to go cheap-cheap at the edge (ask a well-known
> transit provider).
>
> > - Something the redesign/reimplentation will allow is
> > "core is glue only. Customers attach at the aggregation
> > layer and everything is a customer"
>
> That's the way you want it.
>
> > - I'm using IGP for loopback addresses, but also local
> > routing.  Not all devices can handle either BGP, or
> > full-tables.
>
> That's those Cisco 4948's you're talking about...
>
> That is a different upgrade project, but I
> > need to keep existing/legacy services running as I go
> > forward.
>
> Well, if you're looking at the 7600 or some such for the
> edge, you could use it as a Layer 2 aggregation edge router
> and service IP customers off their individual VLAN's. That
> way, you don't need to worry about having to support full
> BGP tables on your Cisco 4948's. Of course, the downside is
> turning the Cisco 4948 into a pure Layer 2 device means you
> have to deal with STP issues re: uplink redundancy.
>

-- I had actually considered another pair of 7600s at the aggregation layer,
but we currently have ~300 ports in use and the cable management is a
nightmare.   The 4948s let us to a "top-of-rack" design and run back to the
core.  We could have done the same thing with a pair of 7600s and dumb
layer-2 switches, but using the 4948s allows incremental upgrades/migration.



> - I'm on the fence with IPv6.  Of our current "name
> brand" providers, only one of them even sort-of supports
> v6.  v6 is also on my feature requirements list, but I'm
> planning on going dual-stack later rather than earlier;
> both to change as little as possible while upgrading and
> also to give me more time to digest how v6 really works
> and what it means.

 Well, if you're buying anything new now, insist that it
> support v6 for the features you (will) need. I'd consider it
> a show-stopper if any hardware/software we're buying today
> doesn't support v6.
>
> Yup. v6 is definitely part of the feature requirement, even if we don't
flip the switch right away.

Thanks,