[j-nsp] Network automation vs. manual config

[Pavel Lunin]

Antti Ristimäki wrote:

So, now that more and more of us are automating their network, there will
> be the question about how to manage the configurations, if they are
> partially automated and partially manually maintained.
>

...

One option is to generate the whole config [...]
>

...

Another option is to apply the auto-generated configuration via
> apply-groups [...]
>


There are two questions here: what to automate (whole config or a part of
it) and which tools to use to manage config templates (apply-groups, or
ansible/yaml/jinja2, or phython/bash/awk/grep/regexp/multi-cursor text
editors, or other off-box tools).

The second question is rather less interesting. Finally, the choice of
tools heavily depends on a particular network, number of people operating
it, their skills, how often they change etc, as well as the current fashion
trends. I personally wouldn't opt for any product-specific tool like JUNOS
apply-groups these days. In my experience, apply-groups often tend to mess
things up, and I recommend to only use it for product-specific technical
things like dual-RE configuration.

The first question (what to automate) is much more complex and a bit
philosophic. First it depends on what kind of config you are talking about.
In the case of a LAN switch, DC fabrics etc, you normally want to make your
config as static as possible, so that a whole switch would be a
reproducible building block of your network. In this case, the whole config
is normally the right option. And if you want some "dynamism" like
on-demand provisioning of EVPN VRF's, you certainly need some sort of an
API/protocol and not text config templates. Let me say that it's how SDN
differs from automation (API/protocol-based orchestration vs. per-device
text config automation).

In case of a multi-service MX-like router, of course, you can't have the
whole config as a building brick. So you need to define those bricks: a set
of BGP sessions and policies for an IP-transit customer, an L3VPN VRF
template, a core-interface, a customer VLAN-interface, or whatever else.

Finally, the art of automation is all about defining those reusable blocks
in order to implement your service model. And to implement a service model,
first you need a service model. *This* is where most people fail and not
the choice of tools to manage config templates.

Modern routers are too flexible and allow to configure nearly anything a
human being might imagine. Network engineers tend to love this "config
challenge", because routine copy-paste sucks and complex problem solving
rules. At the other end of the game you have business and customers who
also want "flexibility". All these factors result in complexity: manually
crafted configs, adapted for specific needs of each
customer/project/business-need etc. So the most difficult problem is to
find a balance between flexibility and standardization. Especially B2B
services tend to be hard to standardize, as major customers often create
enough business pressure, when asking for a custom solution.

It's also important to understand that automation is not the only tool to
implement a service model. A lot of things can be statically
pre-provisioned and encoded in the network data-plane instead of being
stored externally and configured on-demand. E. g. in the resident broadband
world, you don't create a new VLAN each time you have a new subscriber.
Neither manually nor automatically. Instead you pre-provision all
subscriber VLANs in advance so that a pair of Q-in-Q tags (or any other
sort of L1/L2 IDs) would say to the BRAS "hey, this packet is from port X
of MSAN Y". And then you only map those X/Y-s to subscribers. Instead of
mapping each piece of intermediate device config between the access port
and BRAS, using some external databases, all those intermediate elements
are bound together by a "numbering scheme", which is encoded into packet
headers.

Similar techniques can be used in the DC world, or even for B2B customers,
e. g. pre-provisioning static interfaces and access VLANs and than "just"
assign VRFs and policers to them, instead of creating them on demand.

However most people tend you treat services "one by one", putting each
"segment" into a database, like "on the router R service A corresponds to
the interface I, than, on the switch S1 it's VLAN V1, than on the next
switch S2 it's V2, than this VLAN is assigned to the physical interface K".
In the second database: "interface I of router R is tagged with V1, it's IP
address is J". Third database: "parent physical interface of I on router R
is a LAG of M1, M2, M3. M1 is connected to the M5, M2 to M7 and M3 to M11
of the switch S1".  And so on, and so on. Of course, when you try to
generate config templates out of this mess, it just never works.

This is why CLI-monkey approach often wins against automation. Bad
automation is when you just store your CLI in external databases and
translate one to another with templates generating templates over templates
and templates. CLI is the best place to store CLI parameters. If you don't
have a service model, automation will only make things worse.

In general, the power of automation tends to provoke "brute-force". People
think that they can overcome complexity by replacing humans with computers.
In fact, in order to make automation efficient, you first need to simplify,
standardize and formalize. Otherwise it only adds complexity, fragility,
bugs, costs, time and all that.

--
Pavel