> Talking to the technician, I found that they believe someone hooked up a
> set-top box to a router "backwards", sending a stray signal back to the
> system. It interfered with customers on opposite sides of town. They have
> not yet located the source, but took down a bunch of customers in their
> efforts to track it down. I concluded that the reason I got quick response
> to my call was that they suspected I might be at the source of their problem.
> Once they found everything at my end in proper shape, they went on their way.
>
> Have you ever encountered this sort of thing?
This can happen if somebody attaches a router to the network with the
routing tables screwed up (usually with an incorrect gateway setting)
and the private side hooked up to the public side. This is much more
common when the provider is using NAT (network address translation)
which effectively gives you a private network hanging off of a private
network with no true global IP address. I've seen it where the ISP
provided addresses in the common 172.x.x.x pool which can provide
confusion because those same address numbers are used on the private
side of your router. If you have both sides set to the same network
and you reverse the connections, ugly things can happen. To one extent
or another, almost all ISPs have gone to NAT because of IPV6.
However, this also points to a technical flaw in the implementation of
the FTTH deployment there. In a traditional DSL deployment, we use a
form of ATM technology (not the banking machines) that is what we call
a "Layer-2" Technology, which keeps your Ethernet connection isolated
from everybody else's all the way to the router. Each stream is
terminated at a common point right where the brains are. Almost every
other form of broadband to the home works in what I lovingly call
"Layer 2.5" or even "Layer-3". In this environment, you are basically
in a community-wide local-area-network. There are even deployments
where you can actually see your neighbors' computers! not so much any
more, but these types of networks still exist.
As is usual, the technology itself is fine and 99.999% reliable.
Unfortunately, all it takes is for the human element to bring
everything crashing down. You saw it happen in your neighborhood. It
happens with pretty high frequency on the NNI (network to network
interface) level too. When that happens with incorrectly programmed
border routers, it can affect hundreds of thousands of users or
websites. It doesn't take much to bring an entire server farm down. My
conservative guessimate is that we average every user or website
around the world being affected by this up to four times a year. I
subscribe to a couple monitoring services for Zone-10 and can see how
often it is affected. You know it's this kind of error because one
monitor will catch it, but another will be fine. Meanwhile, I've had
people email me saying my website was down when I can get to it fine
and so forth.
After all these years, it's pretty easy to identify the problem and
also pretty quick to isolate and repair it. When it happens on the NNI
level, there are software tools that monitor for this and generally
squelch it in a hurry. But when it happens within an ISP's own
network, it can take a little while for all the routers to get their
routing tables sorted back out again. The biggest impact is when DNS
is messed up. Between routing tables and DNS databases, it can take 12
hours to purge the misroutes out of the system.
When all else fails, turn everything off and back on.
--
Ken Norton
ken@xxxxxxxxxxx
http://www.zone-10.com
--
_________________________________________________________________
Options: http://lists.thomasclausen.net/mailman/listinfo/olympus
Archives: http://lists.thomasclausen.net/mailman/private/olympus/
Themed Olympus Photo Exhibition: http://www.tope.nl/
|