How to Analyze Network Traffic to Detect Advanced Persistent Threats

Many IT workers feel more like firemen than like networking professionals. They detect network attacks, respond to them and move on to the next problem. However, putting out fires is an ineffective approach because research has shown that most cyber attacks are not one-time events. They are a series of ongoing attacks designed to probe a network’s weakness and compromise it.

While knowing that one’s network is constantly under attack may seem discouraging, the good news is that, the ongoing nature of attacks gives a chance to learn from the mistakes of the attackers. Much is made of zero-day attacks, but many advanced persistent threats (APT) actually use some sort of familiar malware and older exploits. Advanced network security companies can monitor your network at a deeper level, analyze attack patterns and help to recognize and filter new attacks before they can start running the exploit. Although the very newly created malicious files may not have any detectable signatures, these tools can often learn to recognize patterns including communication between a malware program and its command-and-control (C&C) server.

Recognizing Suspicious Network Traffic Patterns

The malware campaigns of the past were thwarted when antivirus software picked up a virus signature pattern. However, many of the today’s attacks occur before a signature can be detected and communicated to the world at large. While attackers are constantly adapting, many types of malware follow consistent communication patterns that enable security solutions to identify them as threats. Security tools can recognize attacks in progress by using these four main generic detection methods:

1. Protocol-aware detection : Many types of malware, including remote access Trojans (RAT) use HTTP/HTTPS ports for communication. For example, RATs often target Port 80 and Port 443 because these ports are open at the firewall level. If a security program detects non-HTTP activity at Port 80 or non-HTTPS activity at Port 443, then it will flag the communication as a part of a potential attack.
2. Compressed archives : When a network has been compromised, attackers may use .RAR files, which are password-protected and compressed archives, in order to steal data. Isolating .RAR files as they leave the network does generate false positive results, but it also prevents serious data breaches.
3. HTTP headers : While a malware campaign may use HTTP headers to communicate with its C&C server, it may also send requests by utilizing an application programming interface (API) that looks different from network traffic. By analyzing the HTTP headers, an advanced security solution can detect malware even without a signature.
4. Pattern detection : Many campaigns send beacons to their C&C servers at regular intervals. A periodic DNS request or request to the same URL can signal an attack in progress. Other potential attack patterns may include consistent transferred data volume, packet size or timing.

Protection Tools for Recognizing Suspicious Network Traffic Patterns

In addition to detecting general network traffic patterns, here are four other specific detection methods to thwart APTs:

1. Network content inspection : Performs file extractions, decompression, decoding and port-agnostic protocol detection.
2. Advanced threat scan : This technique scans for functionality that might not be found in a typical network file. Heuristic scanning may detect the typical distribution technique employed by a worm, the payload of a Trojan horse or the replication pattern of a computer virus.
3. Sandbox analysis : Suspected malware is isolated from the network and run in a sandbox. The sandbox filters out suspicious programs based on parameters set by the customer and keeps them from executing certain functions on the network.
4. Threat intelligence : Using a solution from a major security company gives the access to a wide network of threat assessments and detection tools. For instance, a major security company can correlate billions of URL, email and file queries every day, sorting out suspicious items based on known threats and on the source or site’s reputation.

Unfortunately, no security tool can protect the network from every threat. However, when you choose a network security solution, choose a program from a reliable company with a wide range of resources at their disposal.