[j-nsp] Service Activation Testing

[James Bensley]

> On 22/09/2016 14:41, Joe Freeman wrote:
>> I've been asked to put together a solution that allows us to do SAT on
>> every new turnup. These are all Ethernet services.
>>
>> I've been trying to figure out how to do it in the MX platform since that's
>> what we predominately have in our CO's, but JTAC has recently told me that
>> RFC2544 or Y.1564 service testing won't be available until 17.1 at the
>> earliest, contrary to all the published documentation for 16.1.
>>
>> What solutions have others used?

Sadly, the highest overhead but most reliable option which is to get a
field engineer on site and have them perform testing as part of the
service activation.


On 26 September 2016 at 00:29, James Harrison <james at talkunafraid.co.uk> wrote:
> RFC2544 isn't great for non-lab testing - Y.1564 is the way to go
> (unless you want to validate TCP throughput rather than or in addition
> to L2/L3, in which case RFC6349 is the right tool). It's worth digging
> into what you're actually looking to test and assure - CIR? EIR? Do you
> need to prove burst characteristics, QoS/CoS etc? VLANs, multicast/IGMP?
> Some of this will determine how you'll have to test.

+1 for this, as mentioned RFC2544 and Y.1564 aren't the same. Also if
you are running MPLS down to the CPE you can activate OAM between PE
and CE, this works more or less in the little testing I've done with
it, but scaling to every MPLS enabled CPE rules it out for us.

There are multiple things to test. In the case of jitter and latency
for example we have lots of small Cisco routers dotted around that are
configured in a full mesh of IP SLA test and we poll from the NMS for
the test results and create a matrix. SAT for a new PE would mean
adding it as a client to the mesh of IP SLA probes, not for CPEs
though. My point is, OP might need to break this down into multiple
tools or platforms etc.

> Realistically there's a lot of kit out there for multiprotocol Ethernet
> service assurance, and while the MXes can do TWAMP and such just fine
> I'd be looking at dedicated hardware for Y.1564 et al. Apart from
> anything else, having performance measurement endpoints/devices
> dedicated to just that makes isolation of variables easier when
> diagnosing performance faults and gives you a bit more flexibility in
> how you deploy test endpoints.

If OP is offering a wires only service (we usually do managed CPE but
we do wires only from time to time) a hardware tester is the way
forward. Get a field engineer on each end of the link/DCI/whatever and
give it a thorough testing, save the results somewhere safe and get
the customer to approve the results before handover (assuming there
are no issues).


> For 1G and up, EXFO, Viavi and VeEX all have products worth looking at,
> though only the former two have "service assurance" platforms
> specifically aimed at turn-up testing (AFAIK - VeEX has a lot of stuff
> in the HFC world, less so on pure Ethernet). Below 500M or so there's
> more scope for cheap and cheerful options like perfSonar/iperf/bwctl and
> friends, but I'd really avoid those if you can - we see a huge amount of
> variation in test performance above even a few hundred megs.

On 26 September 2016 at 18:04, Joe Freeman <joe at netbyjoe.com> wrote:
> Primarily, I need to valid throughput and frame loss at the moment. having
> the ability to do L2/L3 with CoS/QoS is icing on the cake.

A shameless self plug here, I have written an tool that has been used
by me and a few others at other ISPs to test layer 2 services and
pseudowires and VPLS services https://github.com/jwbensley/Etherate

Similar to above when I said we send an FE to the each end of a link
for testing with hardware testers, sending two guys with laptops with
1G NICs, Etherate will easy saturate the link with 1Gbps of traffic
and let you calculate the delay, frame loss, detect out-of-order
frames etc. Stuff like iPerf is great but I've had problems with
varying iPerf results when all I wanted is a basic speed test in UDP
mode. Etherate is a very simple and light weight program so it can
easily generate 1Gbps and 10Gbps but most laptops don't have 10Gbps
NICs (yet!).

We do also have iPerf servers dotted around in a few PoPs, customers
and engineers can run tests back to them anytime after the service
goes live which is helpful as it allows customers to perform some
diagnostics before they come to our support desk.

Cheers,
James.