myNetWatchman   KnowledgeBase

Pooling knowledge to
secure the internet.


mNW Reports  FAQ: mNW Reports




(Registered Users Only)


Look Up Incidents by IP Address
 

 

How can I tell if I've been hacked?

Modified: 03-Jul-2001
By: Lawrence Baldwin

* DRAFT * *DRAFT* *DRAFT*

DISCLAIMER:

If your system has indeed been compromised, any step you take to investigate could alert an attacker that they have been detected and could lead them to destroy data. If your system contains high-value data and you do not have extensive security experience, we suggest that you immediately engage an experienced incident handler in your area before proceeding with any investigation. If you choose to use these procedures, you do so AT YOUR OWN RISK!

Also, a proper incident handling approach is highly dependent on your specific environment, your priorities, and the tools you have available to you. You may need to adjust this process to suit your specific needs/priorities.

The first version of this document is targeted to Linux/Unix systems...future documents will address Microsoft Windows systems.

---------------

If you're reading this because you received an alert from myNetWatchman, there is a very high probability that your system has been compromised by a hacker. However, your first step should be to analyze your system in order to confirm or deny our suspicions.

 

Step 1: Adopt an incident handling mindset

  • Your system, applications, data, and operating system files are now UNTRUSTED
  • You can not trust the output of ANY command you might use to analyze your system
  • Attackers routinely replace ps, netstat, ls, ifconfig, etc.. with modified versions that hide their presence
  • The operating system kernel itself could be untrusted if a loadable kernel module (LKM) rootkit has been installed
  • Your system may be being monitored by the hacker, either directly or via automated monitoring/recording tools
  • Any attempt to interrogate the system risks alerting the attacker that they have been detected, prompting swift deletion of evidence...and perhaps your data
  • A natural reaction is to immediately disconnect the suspect system from the network. Although rare, be advised that there are root kit tools that poll the Internet to detect loss of network connectivity, again, prompting deletion of evidence....and/or your data
  • Ideally, your first analysis steps should be completely passive

Step 2: Perform a full image backup

Using trusted versions of your backup software, perform a full image (NOT file) backup of your system.

Step 3: Passively monitor for port scan activity and disconnect sensor

If we sent you an escalation it was because we detected port scan activity originating from your host. Port scans are indicative of a compromise as hackers routinely install and activate automated port scanning scripts in order to use your system to anonymously locate more vulnerable hosts.

You can verify port scan activity passively, by:

  • Check the current bandwidth utilization of your Internet connection. If activity is significantly above normal, this could be an easily detectable warning sign
  • If your Internet router supports it, enable accounting statistics and/or packet debugging so that you can get a detailed breakout of the source/destination of traffic. WARNING: enabling these features often incurs a very high performance penalty, use them sparingly and for short time intervals.
  • Install a protocol analyzer on a separate PC or workstation and connect it to the same hub that the suspect system is connected to. A great (free) Windows based analyzer is Ethereal
  • Set up the analyzer to filter all traffic from the suspected host
  • When analyzing the trace watch for packets being sent that target the same destination port numbers that were reported in our escalation, although it could certainly be scanning on other ports as well. See our Top Port Targets report for a list of other hostile ports.
  • Watch carefully for any kind of Internet polling (e.g. of a well known site) which may be a "disconnect sensor".
  • Determine if there are any active connections from an attacker(s)
  • Save all packet traces!

Step 4: Rootkit detection

If your monitoring confirms port scan activity, this is very strong evidence of compromise and you should then try to detect if a rootkit has been installed. A rootkit is a set of tools (port scan scripts, password sniffers, log file sanitizers, backdoor listeners, etc.. ).

Many rootkits can be detected and identified by analyzing the output of ps, netstat, and/or ifconfig. However, you need to be sure you can trust the output of these commands BEFORE you try to use them. And before we even analyze whether we have trusted executables, we need to determine whether the operating system kernel itself has been modified.

For a great guide to detecting LKM rootkits, see:

Loadable Kernel Module Detection
by Toby Miller, GIAC Certified Analyst, MCP

If you are unable to find any LKM rootkits, the next step is ensure we have trusted versions of our the system utilities we need to use. There are several approaches to this:

  • If you are running Redhat Linux and had the foresight to create an RPM (Redhat Package Manager) baseline of your system, then you can use RPM to verify that the MD5 checksums of your executables match your baseline
    (if you didn't have the foresight, then this should be the first thing you do after you restore your system)
  • Alternately, you'll need to build fresh executables using your original operating system distribution (from read-only CD-ROM would be preferred, of course).

Now capture the output from the following commands (saving all output to floppy disk):

  • ps -ef
  • netstat -an
  • ifconfig

Step 5: Disconnect system from network

Once you have collected this data, and you are confident that there isn't some kind of disconnect sensor, then it is probably time to disconnect the host from the network for safer analysis.

Step 6: Analyze evidence

Review the ps output for suspicious processes, especially ones running with root privs.

Review the 'netstat' output and look for connections on suspicious ports or ports you know are not required by the applications supported. Also watch for port listeners on known rootkit backdoors.

Review output from 'ifconfig', look for the characters 'PROM' indicating that an interface is currently in promiscuous mode. Normally a network interface will only see the traffic that is specifically destined to it. Promiscuous mode allows an interface to see ALL traffic on the network it is connected to and is indicative that a packet sniffer has been installed/activated.

If you need assistance analyzing ps, netstat, and ifconfig output, please e-mail your output to mNW Support for assistance.

Note: We do NOT provide security consulting services, however, we will try to provide limited assistance as time permits.