[j-nsp] DSCP-marked traffic mysteriously being dropped by MX960

[John Neiberger]

We've been troubleshooting a strange problem for a few days. JTAC is
on the case, too, but we have not found any resolution. I thought
maybe picking some minds here would be helpful. Here is a simplified
diagram:

[Device A] -------   [Router A] -------  [Router B] ------- [Router C]
----- [Device B]

The problem is that packets from Device B to Device A are being
dropped at Router A. Routers A and C are MX960s. Router B is a CRS.
Router C has an ingress firewall filter that does nothing but mark
traffic as cs2. Router A has an egress firewall filter toward Device
A, but it specifically allows the source IP address of Device B as
well as any traffic marked as cs2.

Here is where it really gets weird. If we remove the filter on Router
C that marks the traffic, everything starts working. Put the filter
back in place and the traffic stops. We've been looking at this for a
couple of days and JTAC has spent a few hours looking at it and we're
still no closer to figuring out why cs2 traffic is being dropped. With
the filter in place, traceroutes from Device B to A stop at Router A.
Remove the marking filter and traceroutes complete and pings start
succeeding.

Can any of you think of a potential culprit that we're not seeing? I
would hope that if this were something obvious, JTAC would have caught
it by now. We're all stumped.

Thanks!
John