[j-nsp] Automation - The Skinny (Was: Re: ACX5448 & ACX710)

[adamv0025 at netconsultings.com]

> Mark Tinka
> Sent: Friday, January 24, 2020 2:32 PM
> 
> On 24/Jan/20 12:10, Saku Ytti wrote:
> 
> > In my opinion we do roughly the same thing, the same way in networks,
> > with the same protocols since my start of career in 90s, very little
> > has changed and you could drop competent neteng from 90s to today and
> > they'd be immediately productive. Compare this to what has happened to
> > compute the difference is striking.
> 
> Agreed - but is it really enough to the extent that the common buzz
sentence
> nowadays is "Network engineers are dead, they'll all be replaced by
software
> [developers]"?
> 
> I mean, I'd wager that more than half of the problems you find with
> automation and tooling development is a total lack of protocol between
> software developers and network engineers; in the same company. While
> there has been plenty of success with a software developer reading a
> networking-related RFC and writing code for that without needing to
> understand, really, how IP/MPLS networks work, it's a whole other issue
> trying to teach a network engineer how to write code, or a software
> developer what IS-IS actually does.
> 
You nailed it Mark,
My opinion is that this new NetDevOps/NetOps initiative is the biggest
blunder of the networking industry.
If as a network engineer/architect you have some coding skills well good for
you,
But are programming skills a requirement to get into network
engineering/architecture nowadays - that absolutely should not be the case. 
We need skilled network engineers and architects to know how to build and
operate complex networks 
We need skilled developers and system architects to know how to build and
operate complex systems (including network automation systems)
We need these two groups to be able to talk to each other in a constructive
manner - check out Model-driven engineering to get you started.
Following these 3 simple premises you can then afford to have an army of
web-ui clickers - provisioning network services not knowing the first thing
about what' going on in the background of the network automation system. Or
not, and you just handover the web-ui/API to your customers and have them
self-service.  

Imagine a case where network engineer builds an automation solution based on
number of hacks involving ansible, python, ydk, whatever... and this
solution gets traction and is used by the company.
Now that poor networking guy has a full-time job supporting the automation
solution, fixing bugs, developing new functionality and you just lost one
network engineer. This is a good example of jack of all trades but master of
none.
Even if you're a small operation or a start-up hiring a developer and make
him talk to the network engineer in a virtual team is a much better option.



> > People who think that netconf and yang are solving big problems and
> > are key to solve automation probably haven't done much automation.
> 
> Totally agreed. But to also be fair, NETCONF/YANG are normally being
touted
> by vendors (much like Segment Routing, 5G and SD-WAN, but I digress). I've
> not really found actual operators with anything meaningful and useful to
say
> about NETCONF/YANG.
> 
> Raise your hands if I'm talking nonesense.
> 
> For us, we find this whole NETCONF/YANG thing to be too heavy for simple
> instructions you need to send to devices, not to mention the fact that
> support within and between vendors is questionable (FlowSpec, anyone?).
> 
> I mean, that's why Ansible was so pleasing to our fingertips - all you
need is
> SSH and a large-enough, repetitive problem you want to go away quickly.
> 
Don't judge the book by its cover, in other words just give NETCONF and YANG
a try, seriously.

I'd say that NETCONF's biggest advantage over SNMP/CLI is it's transaction
mechanism particularly atomicity and consistency ("all or nothing" and "all
at once") from the full ACID, but all these are addressed by all NOS-es
supporting two stage commit via CLI (As Saku mentioned below), so not a
biggie. 
Sorry Mark XE is not one of those NOS-es, but you could still get the
functionality on XE using NETCONF ;)  

YANG on the other hand gives one a common modelling language for
representing services layer configuration and network layer configuration,
which I find useful.
But I'm a minority, I guess there aren't many of you using RFC8299 & RFC8466
as bases for decomposing your L2 and L3 services and building a service
abstraction layer, on top of network configuration layer, so YMMV.

> > Roughly netconf is new snmp and yang is new mib, what ever they enable
> > could have been enabled by existing protocols decades ago, the
> > advantages are modest and will remain so.
> 
> Completely agreed!
> 
Regarding SNMP vs NETCONF similarities, 
For pulling operational data yes, for pushing configuration not really...
see ACID above.  


> >  The key enabler for
> > automation is device accepting arbitrary new B config when it is
> > running arbitrary new A config and transition there hitlessly.
Agreed.

> > Generating full new config from DB+template is trivial problem, 
>
Agreed.

> > trying
> > to be aware of network state and move from arbitrary state A to
> > arbitrary state B with minimal amount of changes is hard and
> > unnecessary problem.
> 
With using CLI scrubbing it most definitely is a hard problem, but with
using YANG model representations of service and network state it's as easy
as commit compare/rollback/check on a single network element.
Necessary or not I guess it depends on the level of automation desired.

> I tend to agree with you, Saku. What I've heard (from the vendors,
> again) is that Ansible is not great because you don't inherently get state
> confirmation feedback after posting the new configuration, and that adding
> that intelligence into Ansible requires time and energy to code. Okay,
fair
> point, I'll bite. But also, we are network engineers - we know what
> commands do when they run, and we've spent decades building templates
> from as simple as a Windows Notepad text to as complex as a MySQL
> database.
> 
I'd say there are different levels of automation, 
At the entry level the aim might be just faster CLI interaction/simpler CLI
scraping (Ansible and the like) and at the other extreme there's the full
potential of model based engineering realized with frameworks like ONAP. 
And operators naturally find themselves somewhere between these two poles in
their automation efforts which is perfectly fine.

> Then again, Terraform is meant to fix that downside of Ansible, but for
me, I
> don't really see that as a big issue. We aren't trying to provision
services
> across network domains (despite what MEF's LSO architecture will have you
> believe), and even if we were, do I really want you fiddling in my
network.
> We each know our networks better than outsiders know them, so what
> gives?
> 
If you give your customers web-UI or API to your self-service portal that in
turn talks to your automation system then it doesn't really matter whether
it's your customers or partner operator using the API - these APIs have been
around for quite some time, the MEF LSO is just an attempt on standardizing
things. 

> 
> > If/when network becomes more cloudified, more as-a-service, where you
> > use API to turn up your own active devices and circuits where you
> > want, when you want, instead of owning anything and once those
> > proprietary APIs get some subset standard APIs we'll probably start to
> > see openstack, kubernetes type of complexity explosion in networks
> > too.
> 
> MEF's LSO, which they've been pushing since about 2014. The concept is
> sexy, but honestly, I've not heard much ado in 6 years re: real-world
> deployment.
>
As I said the APIs at various levels for various functions have been around
for quite some time...   
 

> Also, while I'm wild enough to be one of the first maniacs to run a
network-
> wide Route Reflector on a VM on a server in 2014, you won't find me
> deploying said RR's in AWS or Azure, so I can access them over some API
into
> an Openstack/Kubernetes/Docker enclosure. Life is too interesting enough
> as it is :-).
> 
> 
> >  But as long as we keep owning the network most will keep running it
> > CLI jjockey network, touch when you must, but in many cases no one
> > touches it for weeks or months.
> 
> Words of wisdom.
> 
That's a pretty lonely network where customer change or new customer request
won't come in months...
No seriously, why would anyone spend any effort automating IP/MPLS core
which as you rightly pointed out is implemented once and then just sits
there for months without change?
You'd automate the low hanging fruit i.e. repetitive tasks in large volume-
like tasks associated with customer service lifecycle (i.e. PEs and CPEs and
everything in between like aggregation/access networks or interaction with
providers of access/aggregation networks). 
And then there's the whole another universe of value added services in your
telco cloud (that's where your customer gets a firewall VM spun up and
provisioned when he/she clicks on the "I want a protected internet service
add on") -which needs to tie in with your network service provisioning
workflows.

> I just want to get back to building, operating and optimizing IP/MPLS
> networks. I don't mind seeing the back of the last 10 years of SD-this and
5G-
> that.
> 
I see it almost as two separate things -the "simple" plumbing between PEs
and then there's the complex stuff happening at the customer facing side of
PEs.

adam