Over the last 6 months, the administrators of hundreds of systems that have been blacklisted for sending spam have run our forensic scanner (SecCheck). After analyzing these submissions we have concluded that more than 75% are from networks where there are dozens (if not hundreds) of PCs sharing a single IP address behind a NAT router for firewall. Moreover, in the majority of those cases, the administrator is not even running the scanner on the infected system, rather they presume if they have a spam blacklist problem, then the problem must be with their mail server so they submit a scan from that system. When they see no problems indicated on the server, they request blacklist removal and assume the problem has somehow gone away on its own.
Of course, the problem often has absolutely nothing to do with the mail server. Rather the problem is that one or more end-user PCs sharing a common IP address is infected with spam malware. So it's ironic, a firewall/router drastically improves network security by preventing inbound attacks, however, at the same time it significantly decreases security since when an infection slips behind the firewall (e.g. due to end-users specifically inviting malware in) its far more difficult to isolate than if every user had their own public IP address.
Surf to the following web site from a few sample desktops and your mail server: IP Chicken IP Chicken will display your outside public IP address, for example:
Current IP Address: 24.126.X.YIf you see all the systems showing the same IP, you know your mail server is sharing the same IP as your user desktops. As an asside, if outgoing email is critical to your business you should assign a dedicated IP address to your mail server, otherwise any future infection problems on your user desktops (something you can NOT really control) will again put your email in jeopardy.
OK so we are NOT going to give everyone their own public IP address and rip out our firewall, so what's the alternative? We need better monitoring!
Unfortunately that $50 Linksys router that makes it so inexpensive to allow a bunch of systems to share a single Internet connection doesn't exactly give you an elaborate traffic monitoring and management interface. (Actually Linksys does have logging capabilities but others have absolutely none). How can we achieve granular monitoring, independent of the capabilities of our router/firewall devices?
The absolute key to getting control over any network is deploying a packet analyzer. Ten years ago this was cost prohibitive for most as the cost of a commercial analyzer was $5,000 - $10,000. However, with the advent of open source development, specifically Wireshark (previously Ethereal), sophisticated packet analysis is affordable (free) to all. Your only cost is the 1 hour time investment to set it up and learn a few things about capturing, filtering and reading packets. Don't get me wrong, becoming a protocol analyst takes YEARS of time...I've been doing it for nearly 20 years and I often feel like I have just scratched the surface...however, what you need to know to isolate infected systems can be learned in just a few minutes.
The basic approach is as follows: Install Wireshark on your favorite spare PC or laptop (Windows or *nix versions are both available). Inject your new analyzer physically into the network in such a way that you can see ALL Internet traffic BEFORE any network address translation is performed. Start a packet capture using a capture filter that will match the traffic that you are NOT expecting (e.g. spam, port scanning, etc.), while at the same time ignores traffic that you ARE expecting.
For example, in the case of a spam malware infection, you would take a capture using a filter that would capture all SMTP traffic (e.g. port 25) while excluding traffic coming to and from your mail server. The filter would look something like this:
port 25 and not ip host 192.168.1.200
If there is still an active spam zombie on your network, you will see the traffic immediately. Now you can isolate the infected systems in SECONDS instead just running around running virus scans on dozens of systems....which probably won't even detect it anyway (sigh!).
Additionally, the source of the problem is often a transient system...a sales guy with a laptop which is connecting and disconnecting the infected system from your network...so the spam may not even be flowing all the time...it might only occur the one day a week the sales guy is not on the road. Using a packet analyzer, you can setup monitoring and let captures run 24 hours a day and just periodically check once or twice a day.
Lastly, you can use the analyzer to verify your removal attempts. Obviously you haven't successfully removed the malware if your packet analyzer still shows massive outgoing email from the infected system.
For more details on specifically how to setup and use a packet analyzer see:
Idiot's Guide to Packet AnalysisPay special attention to the injection process...if you do not inject the analyzer correctly you aren't even going to have visibility to your Internet traffic and you will see nothing but background broadcast traffic which is completely useless for the task at hand.