Narus Blog

by Supranamaya Ranjan

In my last blog post, I had provided a preliminary analysis showing how the DNS resolvers had caused an uptick in the amount of DNS traffic they were sending towards a set of DNS servers that NarusInsight Secure Suite (NSS) monitors. Since that time, the DNS exploit was released in the wild and hence I got curious to see how fast are organizations patching their systems.

Figure 1: Aggregate volume of anomalous traffic patterns seen on DNS traffic

The figure above shows the aggregate volume of anomalous traffic patterns that NSS saw on DNS traffic. We define an anomalous activity as one where either the volume of traffic going towards an ip-address increases suddenly or the behavior of traffic (e.g., in terms of number of sources contacting a destination ip-address, etc) changes suddenly. Thus, since most of the resolvers had patches applied in a co-ordinated manner, they sent traffic towards destination DNS servers at the same time- and NSS classifies them as anomalous traffic patterns as compared to the historical past baselines, the traffic volume as well as behavior to these DNS servers had changed.

From the above figure, it seems that the biggest amount of patches were applied in the June 9 - 10 timeframe and then sporadically on June 18, 21, 23 and 26. So the biggest amount of cache resets definitely occurred at the very beginning of the announcement and hence from our view via NSS, we believe the vast majority of patches were applied at the earliest announcement on June 9th.

Interestingly, the total amount of DNS traffic (as measured on a per-day basis) passing the NSS probes didn’t exhibit any significant changes (see the figure below). Though the total amount of DNS traffic on a day-by-day basis didn’t change, there were surges that occurred on a minute-level granularity (which is what the first figure refers to). In other words, the patched resolvers which had in co-ordination sent query traffic towards certain DNS servers, didn’t need to send more traffic afterwards as the responses were already cached.

by Supranamaya Ranjan

Two days ago, on July 8th 2008, security researcher Dan Kaminsky of IO Active disclosed to the public a flaw in the DNS implementations for both client-side resolvers and servers. This flaw would allow a clever adversary to launch a cache poisoning attack and hence take over the mapping that a user’s DNS cache could have for say, BankXYZ.com to point to an ip-address that the adversary controls. In short this could lead to someone impersonating your bank or your ISP and hence be able to eavesdrop on all your traffic. A smarter adversary could even set up a look-alike site for your bank and hence lure victims via a smart phishing attack.

US CERT released a note for this vulnerability on July 8th at 02:46:15 PM (most likely EDT). And the ISPs started deploying the fixes that were being released by Microsoft, Cisco and other vendors. There was widespread concern that malicious entities may try to take advantage of this newly published vulnerability to trap ISPs or users that may still be un-patched. Exact details of the attack methodology haven’t yet been disclosed by Dan Kaminsky in order to allow ISPs and users valuable time to patch their systems.

I verified two interesting hypotheses using Narus Insight Secure Suite (NSS): First, whether someone has already begun using this exploit to launch attacks against the DNS infrastructure. Fortunately, I didn’t find any attacks yet. Second, what kinds of changes in the Internet traffic did this global patching cause. The answer for this turns out to be quite interesting as the amount of DNS traffic was definitely found to be increased in our “periscope” to the Internet. NSS found a sudden increase in the number of anomalies in DNS traffic going towards the ISPs being protected.

Look at the figure below, which shows the aggregate volume (in Mbits/hour) over time for the DNS anomalies seen between July 7th and 11th. Clearly, before the CERT announcement and release of the patches, there were no anomalies. But after the announcement on July 8th, NSS saw a 1000x increase in aggregate volume of anomalous DNS traffic. NSS defines a traffic event as an anomaly if the amount or behavior of traffic heading to an ip-address exhibits sudden changes. A further analysis of the sources of these queries shows that they were being originated from open DNS proxies on the Internet and from DNS clients from well-reputed institutions from around the world. The reputation of the anomaly sources leads to the conclusion that these anomalies were not really attacks, but a side-effect of the synchronized patching.

Evidently, the DNS servers within the monitored networks were receiving a significantly larger than normal amount of queries after July 8th. These queries were being sent from all over the world by the DNS clients and resolvers whose DNS cache (which contains a mapping of domain names and ip-addresses) had been cleared out as a result of the patching. Since the clients don’t know which mappings were already “poisoned”, this complete drop of the cache would make sense. Thus, the patched DNS clients and resolvers had to send queries to almost every domain name to start re-building their cache from scratch. And since the patching itself was co-ordinated, we saw clients from all over the world trying in a synchronized manner to resolve the domains within the networks that NSS protects. Infact, in one anomaly, we saw 3500+ clients sending queries to a DNS server within an hour when historically this server used to be contacted by 100 unique clients on average.

by Supranamaya Ranjan

The intermittent outages at Amazon on Friday (June 6) and Monday (June 9) along with the fact that we have no official word from Amazon about the reasons, are leading up to an exciting Internet potboiler of sorts. While we still don’t know the reasons for these outages, we can eliminate the following reasons:

• Using NarusInsight Secure Suite (NSS), we haven’t found any instances of a large scale network-initiated attack so far that could have led to these outages.

We did detect a Denial-of-Service (DoS) attack against the Internet Movie Database (IMDB), which is owned by Amazon. This attack lasted for about 2 hours starting at 9:52 am PST, which coincided with the downtime of Amazon and its affiliate sites. The attack volume averaged 3 Mbits/sec and it was a sophisticated layer-7 DoS attack where the attacker opened 500+ HTTP sessions in an attempt to stress out the CPU resources or clobber the bandwidth around IMDB. However, the attack volume as seen by NSS probes does not seem large enough to warrant an outage as big as this on IMDB or Amazon. At this point, the attack looks coincidental with the outage and not the cause for it.

• Similarly, NSS shows that Amazon prefixes weren’t hijacked by anyone and so we can eliminate Prefix Hijacking as the cause (read Prefix Hijacking of YouTube that happened earlier this year).
• A traceroute to Amazon’s prefixes doesn’t reveal any malicious Autonomous Systems (ASes) as trying to re-route the traffic in order to steal it, in what is known as a Path Hijacking attack. Note that in such an attack, an attacker injects himself in to the BGP AS PATH.
• Amazon’s DNS entries are also pointing to the right ip-addresses for their web servers, so we can eliminate DNS cache poisoning or other DNS related attacks.

The most plausible cause appears to be errors in their Content Distribution Network or with their load balancing. In their normal operation, CDNs are supposed to direct a user to the “best” web server that could serve them the content the fastest. Best could be defined as either the server that is the closest to the user (using metrics of either smallest number of hops or least round trip-times) or one which has the smallest workload currently.

However, yesterday I was always returned the slowest web server’s ip address while trying to resolve amazon.com via DNS. For instance, yesterday (June 9th), I was always returned the following address (http://72.21.210.11) which either took forever to load on my browser or when it came back, the page had no images. On the other hand, the other two ip-addresses that most likely point to different web servers (http://72.21.206.5/ and http://72.21.203.1/) would load much faster and normally.

So the most plausible cause for these outages seems to be either (i) CDN related where users are being returned sub-optimal web server ip-addresses or; (ii) An internal load balancing issue affecting their data centers that is affecting the rendering of response pages from the fragments such as images, text and dynamic queries or; (iii) Amazon is re-architecting their data centers to accomodate new service offerings.

What the truth is, only time or Amazon can tell.

by Supranamaya Ranjan

Amazon suffered an outage today starting 10:30 am PST. For a few hours the main page of Amazon seemed inaccessible and users would get an error message ‘HTTP/1.1 Service Not Available’. There are reports though that users are now able to access the site since 1:30 pm PST.

Using NarusInsight Secure Suite, we are continuing to investigate whether this outage was a result of a network-initiated attack against Amazon. Preliminary analysis doesn’t suggest any Distributed Denial-of-Service (DDoS) attack or any other foul play against the main web site.

Contrary to emerging reports that sites that use Amazon Web Services (AWS) do seem to be running well, we’ve seen that IMDB (Internet Movie DataBase) does appear to have been affected by the outage. My preliminary analysis using NarusInsight Secure Suite shows that at least one of the ip-addresses used to host IMDB was under a sustained denial-of-service attack. My attempt to load the IMDB page via a direct connection to the web server under attack (http://72.21.206.70/) doesn’t load the images at all. It becomes interesting when you realize that IMDB seems to be hosted using Amazon Web Service (AWS) since this ip-address is registered as belonging to Amazon.

Stay tuned, as we would continue with the forensics and be back with more details.

[Update on the IMDB attack at 3:30 pm PST June 06 2008]

This attack coincided with the downtime for Amazon, beginning at 10:30 PDT and continued for about 1 hour and 10 minutes. The attack itself was interesting in that the attacker seemed to open multiple connections with the IMDB’s web server (port 80) while incrementing his source port for every new connection. The attack’s average rate was 3 Mbits/sec, certainly not large enough to cause a complete meltdown but probably good enough to delay the legit users. However, there might have been other attacks launched at the same time on IMDB which weren’t in the path of our probes. If any one else has heard anything else about IMDB’s outage, please comment.

[Update on the Amazon outage at 10:30 am PST June 09 2008]

Seems that Amazon.com suffered another outage Monday June 9th morning. This time the access problem is intermittent. I was unable to access one of the  web server ip-addresses at all directly (http://72.21.210.11) while the other two (http://72.21.206.5/  and http://72.21.203.1/) do display content normally.

by Supranamaya Ranjan

Today, we issued a press release titled ‘Narus Drives Real-Time Traffic Intelligence‘. Before you even begin dreaming of circumventing the notorious silicon valley traffic with fancy real-time traffic trend gizmos, let me forewarn you - we mean ‘network traffic’.

Service providers around the world have been using the NarusInsight traffic intelligence system for a variety of applications, ranging from thwarting Distributed Denial-of-Service attacks (read Weekend of Olympic Flame and CNN attacks) to detecting illegal VoIP gateways (read PTA busts unauthorized traffic gateway). All of these applications have a common theme, to provide an accurate view of the traffic traversing a service provider’s pipes so that the provider may act up on it in real-time using the appropriate mitigation strategy.

This has become ever more important given the way user demands on bandwidth are changing. We watch more videos online than we ever did before and now the technological innovation is happening on the front of HD quality videos online. Think of the bandwidth strain that would pose on the Internet at that time. On the other hand, with the advent of Software-as-a-Service model where applications would be hosted in the cloud and users’ desktops would become thin-clients (strangely reminiscent of the old dumb terminal days), low-latency access to the applications can only be guaranteed if the network pipes are kept clean. So coming back to my point, that service providers had better be ready to prevent a complete Internet meltdown by guaranteeing that every single ‘bit’ in their network is useful. Fortunately, NarusInsight platform has been architected with this very goal in mind, where the provider can apply their own definition of unwanted traffic and clean the same using ACLs, blackholing and third party scrubbers and start seeing higher overall throughputs as well as average revenue per user.

By Supranamaya Ranjan

A recent article by Kelly Jackson Higgins, Senior Editor at Dark Reading, brings forth an interesting theory about the continued proliferation of botnets. She propounds the view that may be the owners of machines infected with Bots don’t care or don’t know enough to do anything about it. If they knew enough, then may be they would have gotten rid of the malaise.

While until now most botnets have been used for launching spam campaigns, we are starting to see them being used for geo-political attacks against an organization or a country. The recent DDoS attacks (Estonia ‘07, Radio Freedom Europe or RFE/RL ‘08) may just be a precursor of things to come. We can only hope that users become more aware about the botnet phenomena in time before we see an all-out cyber war.

Patching your machine with the latest update from your Anti-Virus company may only go so far in preventing your machine from being hijacked by a Botnet. We have seen time and again how during zero-day worm attacks (Code Red), signatures couldn’t be extracted in time. That gap can fortunately be filled by Service Providers, who can see the traffic entering their customers’ network and can hence alert them via a Managed Security Services (MSS) offering. Potentially, a Service Provider could keep track of presence of botnets within their customer periphery and then pro-actively alert them to get rid of the same.

The NarusInsight Secure Suite provides this capability to Service Providers at multiple levels. First, a Service Provider can detect and bounce off all attacks that are entering its paying customers’ network periphery. Second, the provider can dynamically deploy new layer-7 parsers (or, signatures) to detect botnet Command-and-Control traffic (or, traffic exchanged between a bot and its commanding server). Thus, as soon as a human analyst reverse-engineers the language for communication between bots and their command-and-control server, it can be quickly pushed out via this dynamic parser to the NarusInsight platform, whereby future traffic that matches this pattern would automatically be classified as botnet traffic.

By Supranamaya Ranjan

This must be the summer of geo-politically motivated Distributed Denial of Service (DDoS) attacks. We are close to the one year anniversary of the first cyber-war against the Baltic republic of Estonia. Last week CNN was DDoS’ed by supporters of a pro-China group called, Revenge of the Flame. And in the course of all this, a bystander sports site, Sports Network Management, which only had the misfortune of sharing the word ‘cnn’ in its website while not being directly affiliated to CNN at all, also got DDoS’ed. To top it all, in apparent retaliation, one of the sites behind the patriotic drive sweeping China, Red Heart China Signature (www.5sai.com), got its site DDoS’ed by a group of ip-addresses located in Europe.

Close at the heels of all these intense attacks and counter-attacks, we hear of yet another. Various news sources are reporting that Radio Freedom Europe’s Belarus site was DDoS’ed this weekend starting from April 26. The radio station was going to cover mass protests in Minsk, Belarus dedicated to the anniversary of the Chernobyl disaster. The radio station had plans to direct people to their website to check out pictures, videos of the coverage, etc. However, much to their dismay their site was totally inaccessible for 2 days and 2 nights under a massive DDoS storm. According to the RFE/RL Belarus Service Director:

“There was not much we could do because at this moment we also lost e-mail communication and Skype communication with Belarus. As we found out later, the attack was so massive that the firewall that protects Radio Free Europe went down. And a number of other [RFE/RL] sites went down as well.”

These attacks set a scary precedent. That political agendas can very easily find their way in the form of online hacks and attacks. Even more scarier is the fact that the exact perpetrator of a DDoS attack can be very difficult to pin-point, especially if the attack was launched by a Botnet.

Let alone find the identity of the attacker, it can sometimes be incredibly difficult to even find the location of the attacker. In these geo-political attacks, a lot of times, claims are made that the attack was launched from a certain country or region. A case in point are reports that suggested that the DDoS attack on Red Heart China Signature was launched in Europe. However, there could be a completely different picture, where those 4 European ip-addresses were only used as frontmen for the attack, while for all practical purposes, the actual attacker could be somewhere in Antarctica, controlling those 4 machines.

So the point being that to perform detailed and accurate forensics of these sorts, Managed Security Services as provided by the ISPs is better off than an edge-security solution installed at the web site itself. Web server logs of the victim site can only provide an idea of the frontmen of the attack. While, ISPs can not only detect that a DDoS attack is underway and hence alert the web site, they can also correlate and identify who else did the attackers talk to - and potentially determine the botnet controller or the real perpetrator of the attack.

By Supranamaya Ranjan

Over the last 24 hours, several security researchers have analyzed the tools that were distributed to launch the DDoS attacks against CNN. The interested reader can get an analysis of the 3 tools at Dancho Danchev’s blog.

The attack tool that caught my attention is the one in which the ‘hacktivists’ via mailing lists and forums urged users to do one of the two following things. The more tech savvy users or the generals were urged to install an executable after which they would be able to serve a web page on their web site (e.g., hackerhf.com/cnn.html, etc.). This web page would then load www.cnn.com continuously (almost once per second) in a frame. The lesser tech savvy ones or the soldiers were asked to simply point their browser towards the web pages set up by the generals. As long as they kept their browsers up and running, requests would be sent towards CNN’s homepage continuously, thereby throttling either the bandwidth around CNN or Akamai, its Content Distribution Network or bringing CNN’s web server infrastructure to its knees.

Interestingly, just this geo-political activism was able to generate enough traffic at CNN so that legitimate users’ requests were delayed from 1 seconds on a usual day to 4 seconds times on Sunday morning for 3 hours (source: Netcraft) . Note however that requests were never dropped and everyone was able to browse CNN albeit slowly.
A packet trace analysis of what happens when a soldier logs on to one of the generals’ sites reveals the following sequence of HTTP requests that are generated:

Sequence of http requests

Most interestingly, note that the next group of HTTP requests to CNN start after 1 minute (time gap between request IDs 124 and 4910). In other words, this tool allows a Firefox client with default settings to generate attacks at a highly modest rate of once per minute. Interestingly, the frame embedding CNN definitely appeared to be refreshing itself much faster than that, at 20 times per minute. Every browser has a parameter where the browser sends requests to check for new content on an existing site at a particular rate. For instance, for Firefox:

“browser.cache.check_doc_frequency [Integer] (3) - This setting determines how often Firefox checks for newer versions of the page you are viewing. This setting is similar to Internet Explorer’s ‘Check for newer versions of stored pages’ setting. If set to 0 Firefox only checks once per browser session; if set to 1 Firefox checks every time a page is viewed; if set to 2 Firefox never checks (i.e. it always uses the version stored locally in your browser cached); and if set to 3 (the default) Firefox checks at automatically determined intervals. If you browse mostly pages which update their content extremely often (i.e. a few times a day) you may wish to set this to 1 though it will slow down browsing speed. The default of 3 is best for fastest browsing on most connections. You can experiment to see if 0 suits your needs, but don’t use a value of 2.”

One does wonder whether the hacktivists instructed their users to change the default Firefox setting from 3 to 1, where requests would be generated to CNN as fast as the frame refreshes. And assuming that 100% of the users had done this browser setting change, then would CNN’s response time have increased exponentially? May be then it would have crumbled to the attack!

It also makes one wonder which problem is easier: detecting attacks launched by human armies or that launched by botnets?

By Supranamaya Ranjan

Throughout this weekend, CNN’s website was under threat of a DDoS attack purportedly being planned by a group called Revenge of the Flame (source: DarkVisitor blog). Fortunately, there were no large scale attacks and CNN.com was very much up and running. The weekend plot involved dramatic twists and turns that Hitchcock would have been proud of. First, the hacker group postponed the attack since the news had leaked far and wide. Later for reasons unbeknownest to us, the group called off the attack completely and even disbanded.

Despite calls by the group for halting the attack, there were relatively smaller scale attacks that did happen over the weekend. May be the calls to stop didn’t propagate to the participants as far and wide. Multiple sites of CNN (www.cnn.com, www4.cnn.com, edition.cnn.com) were the target of these attacks. NarusInsight Secure Suite (NSS) reported 2 different kinds of attacks going towards CNN - ICMP flood attacks and TCP SYN flood attacks. Interestingly the attacks had very similar signatures, e.g. an instance of a SYN flood involved the attacker distributing his packets across multiple source ports while sending exactly the same number of packets per source port). This can be expected given that the hacker group had made it easy for the novice who could download a script to launch the attack.

The highest bandwidth attack seen by NSS was an 80 Mbps SYN flood attack, while the others were much less than that. Regardless, the attacks were never big enough to bring down CNN and much to our joy we could continue reading about the Pennsylvania primary, the olympic torch being relayed around the world and all the other stuff that gets us up in the morning.

By Supranamaya Ranjan

Since early yesterday morning (18th April), we have been following ‘The Dark Visitor‘ which has leads about a large-scale DDoS attack being planned on CNN.com. The attack was planned for 5 am PST, but seems to have been called off, rather postponed since too many people found out about it. Nevertheless, I took a look at the traffic heading towards CNN and NarusInsight Secure Suite (NSS) didn’t report any attacks for today morning.

NSS did report 2 separate TCP SYN flood attacks yesterday morning though, one was targeted towards cnn.com and the other towards edition.cnn.com. These attacks lasted very briefly (2 minutes) and had the uniformity typically only seen in attack traffic, e.g. each of the attackers generated the same amount of traffic. We will keep a close eye on this in case the attack does happen. Since CNN is very much up and running, I am off to get my daily dose of news now!