[j-nsp] Automation - The Skinny (Was: Re: ACX5448 & ACX710)

[Mark Tinka]

On 27/Jan/20 22:30, adamv0025 at netconsultings.com wrote:

> Very good point Robert,
> There are indeed two parts to the whole automation story 
> (it' obvious that this theme deserve a series of blog posts, but I keep on finding excuses).
>
> The analogy I usually use in presentations is the left brain right brain analogy,
> Where left brain is responsible for logical thinking  and right brain is responsible for creative thinking and intuition.
> So a complete automation solution is built similarly:
> Left brain is responsible for routine automated service provisioning 
> - and contains models of resources, services, devices, workflows, policies -and you can teach it by loading new/additional models.
> Right brain on the other hand is responsible for "self-driving" the network (yeah I know can't think of better term)
> - and collects data from network and acts on distributed policies, and also performs trending, analytics, correlation, arbitration etc...  
> Now left brain and right brain talk to each other obviously,
> Policies are defined in left brain and distributed to right brain to act on them.
> Also right brain can trigger workflows in left brain. 
>
> Major paradigm shift for our service designers here will be that they are now going to be responsible not only for putting the individual service building blocks together in term of config (and service lifecycle workflow -tbd), but also in terms of policies - determining the health of the provisioned service (including thresholds, post-checks, ongoing checks etc...)
> But following the MDE (Model driven Engineering) theme it's not just service designers contributing to the policy library, it's Ops teams, Security teams, etc...
> Main advantage is see is that some of the policies that will be created for the soon to be automated service certification testing could then be reused for the particular service provisioning post-test and service lifecycle monitoring and vice versa.    
> Then obviously there are policies defining what to do in various DDoS scenarios, and I consider the vendor solutions actually doing analytics, correlation, arbitration all part of the left brain).

Not to sound silly, but you're taking me back to where I was right
around the time Cisco decided to pick up Tail-f :-).


> Then nowadays there's also the possibility to enable tons upon tons of streaming telemetry -where I could see it all landing in a common data lake where some form of deep convolutional neural networks could be used for unsupervised pattern/feature learning, -reason being I'd like the system to tell me look if this counter is high and that one too and this is low then this usually happens. But I'd rather wait to see what the industry offers in this area than developing such solutions internally.

For me, this makes a lot of sense. I'm happy to support standardization
of telemetry streaming (box vendor) and decoding (NMS vendor) because
that enhances the NMS capabilities, which takes away a lot of the
corner-case issues Saku highlighted (well, that's the hope).


>  For now I'm glad I have automation projects going, when I asked whether we should have AI in network strategy for 2020 I got awkward silence in response. 

I'm not even going to touch the ML/AI pole :-).


> I don't know, my experience is that working in tandem with a devops person (as opposed to trying to figure it myself) gets me the desired results much faster (and in line with whatever their sys-architecture guidelines or coding principles are) while I can focus on WHAT (from the network perspective) not HOW (coding/system perspective). Although yes for some of the POC stuff I wish I had some coding skills. 
> But to give you a concrete example from my work, when I had a choice to read some python books or some more microservice architecture books I chose the latter as it was more important for me to know the difference between for instance orchestration and choreography among other aspects of microservice architectures to assess the pros and cons of each in order to make an educated argument for the service workflow engine architecture choice - so it lines up with what I had in mind for service layer workflows flexibility/agility.  

Totally agree with you there.

Network engineers should stop feeling the pressure about needing to
become software heads. I can tell you now, where we are with getting
software to solve of our operational problems, network engineers are
going to be in huge demand. Let's just not ruin the pot by teaching them
"GUI is the absolute answer" the moment they leave the university gates
and enter the real world. It's been hard enough disabusing them of
"Class A, Class B, Class C".


> Well we are starting to get a glimpse of it already with VM of a Route-Reflector running on a server - who owns the host (HW & SW) is it sys-admins or ip-ops, which Mark could shed some light on based on his experience running vRRs. 

As you know, we started running CSR1000v on HP servers under ESXi back
in 2014.

Ultimately, the overall system is owned by the Network team. However,
day-to-day support is handled by the IT team, i.e., ESXi licenses +
management, iLO access to the server, replacement of faulty server or
server parts, that sort of thing.

The server is treated like a router, i.e., it is not part of the vSphere
cloud, each device is an island, it does not run any other VM's,
CSR1000v VM upgrades are decided by the Network team, ESXi version
upgrades are decided by the Network team, the server is hosted in the
same rack as other core routing/switching devices rather than being in a
server farm rack, e.t.c.


> But I guess my argument stands in this area as well, once you develop a successful OEM HW and NOS match that gets traction within the company you're no longer a network engineer as you've been promoted to full-time vendor of this product (dealing with bugs, new features, the overall support of this inhouse built platform).

Not in our experience.

We just recently replaced all our 2014 HP servers running CSR1000v with
a bunch of new Dell numbers at the end of 2019. 6 years was a good run.
Until then, these things sat there humming. Not much needed to manage
them on a day-to-day basis. Our biggest issue is the servers are a lot
more sensitive to ambient temperature increases than IP/MPLS or
Transport gear.


> So I stay by my MDE mantra -I'd rather stay as a SME for the networking side of things on the project and let devops/sysadmins do what they are best at.

I don't disagree with this at all. In the start, network engineers
should figure out what they are doing, so that when they want to scale
it up, they know exactly what to ask the software heads. Reversing that
process is where the pain starts, and the reason you have so many people
falling over themselves clamoring to get hired for a DevOps skill, or to
get trained in it.

Mark.