Improved myNetWatchman Notification Routing Policy
Effective March 1, 2004 we have changed our approach to identifying and notifying the responsible party associated
with myNetWatchman (mNW) detected security incidents.
Previously, we used a combination of reverse DNS, DNS Start-of-Authority,
and IP Whois information to identify the responsible party.
Our goal was to route notices to the most-specific contact
possible, easing the notification burden on Tier 1 and 2 network
operators. However, after three years of trying we have concluded
that this approach is unscalable and ineffective for all but
the largest network providers.
Problems with the old approach
The specific challenges with the old approach were: IP Whois data is very poorly maintained and is often out-of-date, invalid,
or otherwise bogus
It is the responsibility of each network owner to keep their Whois data updated. However, because there are few
negative consequences to letting it lapse, and the Internet Registries have extremely complex procedures to perform
updates, many don't bother.
The number of distinct contacts is unmanageable
Since inception, we have attributed incidents to over
300,000 distinct domains. Identifying and maintaining appropriate abuse contact addresses for this quantity
of domains is nearly impossible, especially since over 90% of these domains do not follow Internet standard mailbox
conventions (e.g. abuse@, security@, etc.).
IP Whois data is often NOT the best contact
Our intial goal was to route our notices to the most-specific
contact possible in order to minimize the workload on higher-tier
network operators. Although this would seem to be the most
effective approach, we have found that in the majority of
cases the most-specific contact (even if you can find a valid
one) is often not the *best* contact because the smaller entities
often lack formal procedures for abuse handling and don't
understand the significance of the incidents we have attempted
to alert them to.
iP Whois data can't be easily cached
When you query a Whois database by IP Address you can easily obtain the most specific contact and the range of IPs that contact
is applicable to. Unfortunately you can only assume you now know the most-specific contact for that ONE IP. If you later
need to find the contact for another IP, even if that IP is contained within an IP range you already received information
on, you still need to issue a full Whois query on that IP as their could always be more-specific information available.
This makes Whois caching very difficult and really necessitates a full query for each distinct IP address you need to evaluate.
The myNetWatchman system receives reports regarding 500,000+ distinct IP addresses per day. Performing this
volume of Whois lookups is both inefficient and not appreciated by the Whois operators.
We had compromised by caching Whois data for 30 days and
only doing new lookups for IPs for which didn't match any
IP ranges of cached Whois data. This provided acceptable lookup
rates, but also meant that the stale data would often be used
and it would never necessarily be the most-specific.
No reliable way to associate an allocation with a responsible
*domain*
Yes, there are Point-of-Contact (POC) records that provide contact email addresses, however,
we feel it is inappropriate to send security notices to Administrative and Technical POC contacts as these are
often people that have nothing to do with managing abuse issues. Some Whois databases provide for optional Abuse
contacts to be specified, however, less than 10% of IP allocations contain an Abuse POC. The RIPE Whois database
supports abuse contacts, but only through comment tags, which are then very difficult to parse
automatically.
We attempted to identify the responsible domain by extracting
the domain names listed in POC email contacts, but they often have no association with the owner of the IP allocation.
For example, XYZ Corporation has an IP Allocation with a Technical POC of fred@yahoo.com. Obviously, yahoo.com
is NOT the resposible party. More frequently, the POC email contacts are consultants or network integrators that
setup the customer network with absolutely no relation to the actual responsible party. IMHO, a major flaw in
the Whois database is the lack of a Owner Domain field.
Even in cases where we were able to identify the responsible
domain, identifying an abuse contact within that domain is
difficult. ARIN Whois has the facility for Abuse POC records,
however, few utilize them. You could hope that most organizations
would create an abuse@ mailbox as recommended by standards,
but few do....a surprising number don't even bother having
a postmaster@ mailbox (or at least don't bother monitoring
it). Third-party databases exist (e.g. http://www.abuse.net
) to provide abuse addresses, but again, only a minority of
domains bother to register their mailbox preferences.
Given this scenario, the only real way you can reliably map
IP Whois allocations to a responsible domain is by reviewing
all netblocks *by hand*. Identifying the proper abuse reporting
mailbox often requires 30-60 minutes of research and several
phone calls. As labor intensive as this is, we actually tried
to do it. For the last three years, a half dozen faithful
mNW volunteers have hand-processed over 37,000 IP Whois records
and associated them to a responsible domain. Unfortunately,
this only represents a fraction of the 100,000 of thousands
of allocations that exist.
Developing a new methodology
Throughout 2003 we began considering a new backtracing methodology,
with the following objectives:
- Reduce the number of domain contacts
- Utilize data that could be queried in real-time
- Reduce the number of Whois lookups
- Identify better contacts
Our conclusion was to direct our notices to the owner of the Autonomous System Number (ASN) that is
originating the route for the affected IP via Border Gateway Protocol (BGP). To enable this we
integrated mNW with a live BGP router and produced scripts which perform real-time route lookups to determine the
originating ASN. We then utilize AS Whois information to determine the responsible party. AS Whois information is just as poorly maintains as IP Whois, but at least there a whole lot less of it.
There are only about 30,000 actively used ASNs, still a lot of data to manually validate, but achieveable.
In October 2003, we proposed this new backtracing approach to members of the
NSP-SEC mailing-list (composed of the top
network security contacts representing over 400 major network operators and backbone carriers worldwide).
The overwhelming consensus was this was a good idea. Even the large NSPs that would see a big increase in notifications
relating to their downstream customers supported it, as most have already automated their abuse handling systems
to route notices to the appropriate downstream using their own internal IP to customer mapping databases.
You can now view the ASn and the responsible domain we have mapped that ASn to in the mNW web Incident report under the following
fields:
- Orig Autonomous Sys (AS)
- AS Responsible Party
If you are receiving notices for ASNs for which you are NOT the responsible party, please email support@mynetwatchman.com and I will
correct mis-associated ASNs immediately.
Conclusion
In conclusion, I realize this new approach is not ideal,
unfortunately, I'm one person attempting to distribute over
500,000 new infection notifications/month forcing me to compromise
between what is ideal and what is practical given my limited
resources. In the future, I may add functionally to allow
more granular routing of notices on a case by case basis,
but for now we will only be routing based on ASN information.
My best suggestion if you receive a large volume of notices
for downstream customers is to automate the downstream forwarding
process using whatever mapping data you know to be reliable.
I can tell you that many other security researchers are utilizing
this same approach when identifying spam trojans, IRC bots
, etc. I predict that in coming years you will see more sources
of security, spam, and DMCA notices being routed this way.
Lawrence Baldwin
Chief Forensics Officer
myNetWatchman.com
Atlanta, GA
+1.678.624.0924
|
