[j-nsp] Automation - The Skinny (Was: Re: ACX5448 & ACX710)

[Mark Tinka]

On 26/Jan/20 22:46, adamv0025 at netconsultings.com wrote:

> You nailed it Mark,
> My opinion is that this new NetDevOps/NetOps initiative is the biggest
> blunder of the networking industry.
> If as a network engineer/architect you have some coding skills well good for
> you,
> But are programming skills a requirement to get into network
> engineering/architecture nowadays - that absolutely should not be the case. 
> We need skilled network engineers and architects to know how to build and
> operate complex networks 
> We need skilled developers and system architects to know how to build and
> operate complex systems (including network automation systems)
> We need these two groups to be able to talk to each other in a constructive
> manner - check out Model-driven engineering to get you started.

Totally agreed.


> Following these 3 simple premises you can then afford to have an army of
> web-ui clickers - provisioning network services not knowing the first thing
> about what' going on in the background of the network automation system. Or
> not, and you just handover the web-ui/API to your customers and have them
> self-service.  

Which is fine for me, as long as there are still some real network
engineers at the company that can make sure the network runs when the
automation system bombs out.


>
> Imagine a case where network engineer builds an automation solution based on
> number of hacks involving ansible, python, ydk, whatever... and this
> solution gets traction and is used by the company.
> Now that poor networking guy has a full-time job supporting the automation
> solution, fixing bugs, developing new functionality and you just lost one
> network engineer. This is a good example of jack of all trades but master of
> none.
> Even if you're a small operation or a start-up hiring a developer and make
> him talk to the network engineer in a virtual team is a much better option.

Agreed - but we, generally, have to start from somewhere. And if you are
going to start small with Ansible, I've found that network engineers
will do that better with limited sysadmin experience than trying to get
the sysadmin to stand up Ansible and get it to talk to routers. Over
time, if its successful, you can farm out bits about Ansible to the
software team that best suits their skills.

I'm not for the idea that network engineers are obsolete and the only
way to run an IP/MPLS network, over the next decade, is by giving it to
the software heads.


> Don't judge the book by its cover, in other words just give NETCONF and YANG
> a try, seriously.
>
> I'd say that NETCONF's biggest advantage over SNMP/CLI is it's transaction
> mechanism particularly atomicity and consistency ("all or nothing" and "all
> at once") from the full ACID, but all these are addressed by all NOS-es
> supporting two stage commit via CLI (As Saku mentioned below), so not a
> biggie. 
> Sorry Mark XE is not one of those NOS-es, but you could still get the
> functionality on XE using NETCONF ;)  
>
> YANG on the other hand gives one a common modelling language for
> representing services layer configuration and network layer configuration,
> which I find useful.
> But I'm a minority, I guess there aren't many of you using RFC8299 & RFC8466
> as bases for decomposing your L2 and L3 services and building a service
> abstraction layer, on top of network configuration layer, so YMMV.

I believed a lot in NETCONF/YANG in the middle of this past decade, and
actually insisted solutions support it. But as I've said before, over
time, you get to learn how to sniff the smell in the air, and while a
lot of those solutions paint a good picture, for our particular
use-case, it just felt like a whole lot of complexity for the 2 simple
things we wanted to achieve first - customer service
provisioning/de-provisioning + network deployment.

Which is not to say that NETCONF/YANG have no use-case. Down the line,
if what we are doing with Ansible becomes overly complex to require
models based on NETCONF/YANG, happy to consider. But to get off the
ground, I feel they are too heavy. I don't want to design overly
elaborate data models - we know what it takes to deploy or run a device;
we've been doing it for years. That doesn't change regardless of if it's
being done by humans in a semi- or fully hands-off approach.

The last 10 years have been bogged down by trying to figure how much
automation to do, when to do it and how. In the end, we are still in the
same place, and even more confused. If starting off slow with Ansible is
okay for me for the next decade, I'm good with that.


> I'd say there are different levels of automation, 
> At the entry level the aim might be just faster CLI interaction/simpler CLI
> scraping (Ansible and the like) and at the other extreme there's the full
> potential of model based engineering realized with frameworks like ONAP. 
> And operators naturally find themselves somewhere between these two poles in
> their automation efforts which is perfectly fine.

And this right here is my issue with where the industry is - we are
trying to define "automation" as a single thing that everyone should
aspire to in a particular way or set of ways.

I counter that if automation, to you, is loading a bunch of scripts into
RANCID and letting it run off and do stuff for you that you find
repetitive, great! I counter that if automation is you taking a Windows
Notepad and dumping it into a database with some templates, great! I
counter that if automation is using a vendor-proprietary solution to
provision and operate the network, great!

Let's not get bogged by trying to get everyone into the same boat, as
this is where, I feel, we are all failing each other.


> If you give your customers web-UI or API to your self-service portal that in
> turn talks to your automation system then it doesn't really matter whether
> it's your customers or partner operator using the API - these APIs have been
> around for quite some time, the MEF LSO is just an attempt on standardizing
> things. 

Which is my point - large operators have had homegrown solutions for
years. They really aren't struggling. We just think they are because we
don't know what they are running, but they aren't.

Now, if they want to open their systems up to other "standards-based"
solutions is totally up to them. And if they don't, I won't castigate
them for not wanting to conform. Ultimately, if they are happy with the
way they automate, who am I to tell them they are wrong for not running
it in some Docker image over Openstack in AWS?


> That's a pretty lonely network where customer change or new customer request
> won't come in months...
> No seriously, why would anyone spend any effort automating IP/MPLS core
> which as you rightly pointed out is implemented once and then just sits
> there for months without change?
> You'd automate the low hanging fruit i.e. repetitive tasks in large volume-
> like tasks associated with customer service lifecycle (i.e. PEs and CPEs and
> everything in between like aggregation/access networks or interaction with
> providers of access/aggregation networks). 
> And then there's the whole another universe of value added services in your
> telco cloud (that's where your customer gets a firewall VM spun up and
> provisioned when he/she clicks on the "I want a protected internet service
> add on") -which needs to tie in with your network service provisioning
> workflows.

I feel like I'm reading a vendor white paper :-).

Seriously though, if that makes you happy, go for it. What I'm saying is
if I find another way that leverages the industry without breaking my
back (or brain), I'll do it to my satisfaction. I won't be chasing the
panacea that is "automation", but rather, what automation means to me.


> I see it almost as two separate things -the "simple" plumbing between PEs
> and then there's the complex stuff happening at the customer facing side of
> PEs.

"Complex", like automation, can mean different things to different
people :-).

Mark.