[j-nsp] Strange behavior on directly connected interfaces?

[Hannes Gredler]

frances,

question 1: what is the MAC adress of the device that
             generates the 10MBit/s worth of traffic.

question 2: is your juniper router the only exit for your traffic

question 3: could it be that there are hidden backdoor(s)

question 4: what traffic is being looped / unicast / broadcast

question 5: what is the destination MAC adress of the looped traffic
             (broadcast address / unicast address of the router)

/hannes

Frances Albemuth wrote:
>  The issue can thus far be mitigated (believe it or not) by filtering
> ICMP to and from the "mystery node", or by filtering ICMP to and from
> every network on interface "A".  I'm in possession of the MAC of the
> "mystery node" and I know exactly where it lives on the network, but
> it doesn't seem to correspond oddly with anything and I haven't
> identified anything quirky about the network configuration.  What else
> should I be keeping an eye out for?
> 
>  -FC
> 
> On 5/17/06, Hannes Gredler <hannes at juniper.net> wrote:
> 
>> frances,
>>
>> to mitigate the problem while diagnosing
>> you could configure a firewall that discards
>> traffic from non-local-subnet sources.
>>
>> but lets focus on the loop:
>>    what is the mac-adress of the mystery node ?
>>
>> /hannes
>>
>> Frances Albemuth wrote:
>> >  Hi Hannes,
>> >
>> >  Thanks for your response.  When I'm sniffing on the segment I see a
>> > massive stream of ICMP TTL exceeded messages being returned by the
>> > "mystery node".  The topology is definitely loop-free and the
>> > "loop-ish" behavior that we see only seems to occur when data is
>> > transmitted to unreachable destinations.
>> >
>> > I assume by forwarding loop you mean an Ethernet loop? I would agree
>> > that it behaves this way in some respects.  Of course, if I had a
>> > genuine loop the problems would be more serious and would occur
>> > regardless of routed traffic (the Ethernet topology with a handful of
>> > hosts would cripple itself).
>> >
>> >  Also interesting: the node returning the TTL exceeded "storm" lives
>> > behind a link with a maximum synchronous capacity of 10M.  The "storm"
>> > itself results in 10M of traffic pushing consistently over all ports
>> > where the VLAN lives.  It thusly only cripples other devices with a
>> > 10M maximum synchronous bandwidth.
>> >
>> > Thanks!
>> >
>> >  -FC
>> >
>> > On 5/16/06, Hannes Gredler <hannes at juniper.net> wrote:
>> >
>> >> frances,
>> >>
>> >> looks like you have a forwarding loop in your setup;
>> >>
>> >> for further troubleshooting attach a packet-sniffer to
>> >> the subnet in question and spot for the source MAC-adress
>> >> that is bouncing back your traffic.
>> >>
>> >> /hannes
>> >>
>> >>
>> >> Frances Albemuth wrote:
>> >> >  Hi,
>> >> >
>> >> >  This is my first post to the list and I would like to preface 
>> this by
>> >> > stating that I doubt this problem is actually related 
>> specifically to
>> >> > Juniper equipment (perhaps a configuration error involving Juniper
>> >> > equipment, however). I'm hoping the issue I'm working on right now
>> >> > might ring bells in the heads of others, and in any case I figure 
>> this
>> >> > is as good a place as any to find yourself beaten by the clue stick.
>> >> >
>> >> >   I have a directly connected interface facing a large, flat 
>> Ethernet
>> >> > infrastructure.  There are dozens of IP's mapped to the interface in
>> >> > question (this is a legacy aspect of the design, but migration to a
>> >> > more hierarchical infrastructure is a long process).  Periodically,
>> >> > when packets are transmitted with an unreachable destination IP
>> >> > residing on the directly connected interface,  a massive series of
>> >> > ICMP TTL exceeded packets is returned by a different host 
>> residing on
>> >> > a different logical interface.  Traceroutes to the unreachable IP
>> >> > similarly show a one-node loop (the same IP responds until the 
>> TTL=0).
>> >> >  The node is always the same, but if unmitigated ICMP traffic is
>> >> > permitted to and from addresses on the logical interface, 
>> sniffing the
>> >> > wire shows this behavior occurring to and from a number of nodes.  I
>> >> > haven't managed to duplicate the multi-node behavior in a
>> >> > semi-controlled environment.
>> >> >
>> >> >   When sniffing the segment in question, the ICMP is clearly 
>> visible,
>> >> > so for whatever reason it is universally broadcast, even though both
>> >> > nodes involved in the ICMP communication are legitimate unicast
>> >> > destinations.  If a ping is left running, these TTL exceeded 
>> messages
>> >> > will continue an accelerate ad nauseum until a de facto pseudo
>> >> > broadcast storm occurs, crippling access on every switching node 
>> where
>> >> > the VLAN in question is mapped.  Usually (but not always) the
>> >> > anomalies halt when the ping is killed.  The issue is largely
>> >> > mitigated by denying all ICMP to and from addresses mapped to the
>> >> > logical interface.
>> >> >
>> >> >   That's all I'm comfortable asserting about the issue at this time.
>> >> > What I'm really digging for here is an explanation as to why when 
>> the
>> >> > Juniper tries to transmit to an unreachable node, it doesn't 
>> discover
>> >> > the node is unreachable due to a lack of response from an ARP 
>> request
>> >> > and return ICMP unreachables on it's own.  I may have missed 
>> something
>> >> > obvious here (I'm sort of hoping so) and would appreciate any
>> >> > suggestions or experience from others.  If I've sent this message 
>> to a
>> >> > woefully inappropriate list I would greatly appreciate a 
>> suggestion as
>> >> > to a better place to bring my question(s).
>> >> >
>> >> >  Thanks,
>> >> >
>> >> >   -FC
>> >> >
>> >> > _______________________________________________
>> >> > juniper-nsp mailing list juniper-nsp at puck.nether.net
>> >> > http://puck.nether.net/mailman/listinfo/juniper-nsp
>> >>
>>