Detecting APT Activity with Network Traffic Analysis

The advanced persistent threat (APT) continues expanding the geography of conquest and targets not only large-scale organizations but also small companies. As an indicator of that phenomenon, APT protection market continues to grow. According to The Radicati Group report, the annual revenue from APT protection solutions will reach a $7.5 bn point by 2021.

For timely APT detection, a company should employ a combination of methods, such as SIEM solutions, IPS/IDS systems, antiviruses, firewalls and network traffic analysis. The latter is considered a mandatory element of APT protection with its basic mission to differentiate between legitimate and illegitimate traffic. This can be done with an array of methods, which vary in their application spectrum.

This article introduces a series of issues concerning the role of network traffic analysis in the APT resistance crusade.

Malware communication with a C&C server: how to identify?

Check IP address

Having infiltrated a network, malware communicates with a Command and Control (C&C) server. It is possible to identify this communication with the help of IP address blacklists. Many information security resources (for example, IBM X-Force) publish the ranges of botnet IP addresses on the internet. Check if there are any suspicious communications with IP addresses from C&C blacklists in your network.

Pay attention to IRC and P2P signatures

Botnets communicate using certain protocols. The protocol type serves as a criterion for botnet classification:

• IRC (Internet Relay Chat) protocol-based – 1^st generation.
• P2P (Peer to Peer) protocol-based – 2^nd generation.
• HTTPS (Hyper Text Transfer Protocol Secure) protocol-based – 3^rd, the most advanced and hardly identifiable botnet generation due to traffic encryption.

Some SIEM systems have integrated components (for example IBM® QRadar® QFlow Collector) that analyze network packets and identify IRC and P2P signatures.

Use behavioral analysis

Three-vector network behavioral analysis comprises traffic pattern analysis, system activities analysis and sandboxing. In this article we will concentrate on traffic pattern analysis.

SIEM systems build a baseline of network traffic that reflects normal communicative patterns of primary network servers. Any deviation from the traffic pattern is a warning sign, and the SIEM system reacts by generating an offense.

Opt for traffic pattern analysis

Traffic pattern analysis is widely applicable for the following purposes:

To detect unknown threats

Among the variety of threat detection techniques, traffic pattern analysis proves to be the most effective tool, which can be employed to detect unknown threats (a.k.a. zero-day exploits). Usually, security administrators configure the range of IPs for every server within their local network. When a server starts communications outside this range, it may be a signal that the unknown IP address belongs to a botnet.

The success in zero-day exploit detection largely depends on the ability of security software to extrapolate the identification methods of known APTs to the unknown ones. Typically, malware is able to do at least one of the following: spread in the network, infect files, conceal itself, transmit data out of the affected environment, communicate to C&C, and make use of polymorphic techniques. Thus, malicious software varies in functions, but it has common traits that can be traced in zero-day malware.

To detect internal C&C server

Botnets can be organized right inside a private network. Placed at the perimeter, network defenses are unable to detect suspicious traffic, as communication occurs internally. In such a case, we can’t resort to external IP address identification, because there is no communication with external servers.

SIEM systems get information about internal traffic from network switches. To notice deviations from the baseline of internal network traffic, security administrators employ traffic pattern analysis. Another way to identify a connection to the internal C&C server is layer 7 application traffic analysis. It includes signature and payload analysis of a network packet. If IRC or P2P communications forbidden by the private network policies are detected, it may indicate that someone has organized an internal botnet.

To detect a malware communication with trusted sources

A classic example is Trojan malware communications, when malicious data from what is believed to be a trusted source disguises as a legitimate communication and penetrates into your network. In the case of Trojan.Gmail targeted attack, the email from a spoofed e-mail address had a malicious PDF attachment. This PDF file exploited Adobe Reader vulnerability, injected malware into the targeted system and modified Internet Explorer browser. Every time the web browser was opened, the malware logged into a Gmail account. In this case, with the help of traffic pattern analysis, security administrators can observe a spike in network traffic to e-mail hosts, which serves as a starting point for further network inspection.

To detect malware communications via HTTPS

As mentioned above, HTTPS generation of botnets is the most advanced due to the protocol encryption. Today, to decrypt HTTPS without a decryption key is hardly possible. Nevertheless, even in the case of HTTPS communication, we can resort to traffic pattern analysis. Based on traffic patterns, it allows identifying traffic deviations within a certain timeframe as well as estimating traffic volume and the number of connections from one IP address.

No universal cure

As there is no universal cure for all diseases, there is no one-size-fits-all method to combat advanced persistent threats. Network traffic analysis (in particular traffic pattern analysis) is a useful technique, but it doesn’t guarantee 100% malware detection. Yet, it is possible to maximize your APT protection with the help of information security consultants who will configure custom traffic analysis rules for a particular environment.