[j-nsp] Provider segregation and subscribed services

[Paul Connally]

I've got something I'm working on that's giving me a bit of a
headache.  We have two upstream providers ("regular" Internet and
Internet2).  We're taking routes via BGP from both, and we have set
local preference so that routes learned from Internet2 are preferred
over regular Internet routes.  All of our downstreams peer with us and
their own Internet providers, and this tends to work pretty well; the
downstreams that subscribe (aka pay money) for the regular Internet
service get sent the full routing table, and those that don't only get
sent the Internet2 routes.  Since our big downstreams peer with their
regular providers, there's not really a problem.

We're going to start having some smaller customers who will only be
'subscribing' to the Internet2 service.  It's probably safe to assume
that some of them (due to limited hardware, expertise, or whatever)
won't be peering with their regular Internet provider, but will be
using a default route.  Here's a big problem I see happening:

Small customer peers BGP to us.  We advertise to them the Internet2
routes that are currently installed in our inet.0 table.  Say for
example Joe's Barber College is advertising 192.168.0.0/16 to
Internet2 and to the regular Internet.  Since we prefer the learned
Internet2 routes in our network, we'll advertise the /16 downstream
with an Internet2 AS path.

However, Joe's Barber College is, for whatever reason, also
advertising a smaller network (192.168.1.0/24) to the regular Internet
ONLY and not to Internet2 (I've seen this in practice; there's several
instances of this currently).  The problem comes with the customer
downstream from us who's not peering BGP to their regular provider
when they want to reach a host on that /24.

Since we advertise the aggregate /16 via BGP to them, packets destined
to that /24 are going to be preferred via their link to us using the
Internet2 BGP path over their default route to the regular Internet. 
The packet comes into our network, looks at the inet.0 routing table,
and sees the more specific /24 route via the regular Internet.  Since
our downstream has not subscribed to our regular Internet service, our
filter drops the packet.  That /24 is now basically black-holed to our
Internet2-only downstream customer.  The easy solution is to tell them
"peer with your regular Internet upstream" to learn that specific /24
from them, but there's a possibility that might not be an option.

I'm thinking the solution is going to be something VRF-ish; set up a
2547 VPN that holds the Internet2 routes from our upstream provider as
well as the Internet2-only customer routes.  We're not currently using
any routing-instance configurations, and I'd prefer not to have to
reconfigure my Internet2 peer under a separate routing-instance ( and
risk breaking connectivity to any current 'full' connectors).  I'd
like to be able to just copy the Internet2 learned routes from inet.0
into the separate Internet2 VRF, but haven't found a very intuitive
method for doing this.

There's also the chance that at some point in the future, we'll be
peering with another "private" network that will also be offered as a
subscribed service.  That means that basically I'll need to be able to
say "Customer A,  you can use these three different upstreams",
"Customer B, you can use these two upstreams", and "Customer C, you
can use only this one specific upstream".

Any advice?