Rotophonic

On Monday, the Friends of Joe Lieberman website, http://joe2006.com/, disappeared from the Internet. Accusations have flown from many sides of the Net pointing figures at liberal bloggers like Daily Kos and even members of the Ned Lamont campaign.

I thought I’d walk through some of the theories out there about how the site got taken down and discuss the merits of each one. The list appears after the fold. There are some surprises in the top two.

5. Failed to pay the bill

This is a quick way to get your site taken down, but it’s an unlikely one. Dan Geary, Lieberman’s Internet consultant, seems to have a very cozy relationship with the hosting provider that was selected to host the site, MyHostCamp.com. Geary told TPMmuckraker that all of his company’s websites are on that server and that it’s owned by someone he works with all the time.

This hosting reseller, MyHostCamp.com, is a shared server, meaning that a single computer has many virtual sites running on it. Lieberman’s server had over 70 sites running on it. MyHostCamp.com definitely hosts its server at and probably rents a managed server from ThePlanet.com, which sells managed and semi-managed servers to resellers. Even if MyHostCamp had the most expensive server offered by ThePlanet.com, the total bill would likely be around $600 per month, divided by 70, equals less than $10/month per site in costs.

Given that Geary probably pays the bill for this server and gets paid by the 70 plus sites using it, it’s definitely not a billing issue.

4. Ran out of bandwidth and got shutdown

Very unlikely as ThePlanet.com offers a number of bandwidth options, including an unmetered service at the high end. Plus, other sites running on that server continue to run now while joe2006.com has been taken down. This is also easy to fix with your provider. If this had happened, the site would have been back up with one quick phone call. Running out of bandwidth didn’t crash the site.

3. Denial of Service Attack took down the server

Based on statements by Geary (“When we take the site down, the server is fine.“), content within the server was corrupted and thus the site needed to be taken down, but other sites running on the same hardware continued to run. Any denial-of-service attack would have affected these sites equally and would have subsided once steps were taken to avoid the attack.

Further, the datacenter where Lieberman’s site was hosted offers a number of specific protections that help mitigate any DoS attack, including Arbor Peakflow DDoS Detection and Cisco Guard DDoS Mitigation.

Lieberman’s site was not downed by a real DoS attack.

2. Joe2006.com’s Content Management System Hacked

Joe2006.com used a content managment system called Joomla!, a widely used system for setting up websites. As with most pieces of software, Joomla! is frequently targeted by hackers who try to exploit weaknesses in the system to deface or take control of sites running that software.

It’s difficult to determine which version of Joomla! Joe2006.com was running or the versions of specific components that were in use on the site, but examining Google’s cache of Lieberman’s site shows that parts of Lieberman’s site definitely used a component called com_extcalendar, which allowed the site to display a calendar with events.

The most recent serious problem with the Com_ExtCalendar component was discovered very recently–on July 7th, 2006. This issue would allow a hacker to deface or even overwrite the entire configuration file for the site. Others have written about “script kiddies” spending their summer vacations attacking Joomla sites, including those with this component. Geary told TPMmuckraker that, “We have nobody with a security background helping with this. It’s just us, what we know, how we work with our server network.” I read this as, “We just use the webserver control panel and know how to upload stuff via FTP.”

Given that Dan Geary indicated that he was flying blind with all the technical issues, it seems very unlikely that their server had been updated with the latest and greatest version of the Com_ExtCalendar component that fixes the security issues.

Geary told MSNBC that the hack on Monday August 7th involved the site being defaced and that later massive amounts of traffic to the site were linked to the failure of the site. A hack by someone taking advantage of the weakness in components of Joomla! would not have generated traffic, but could definitely have been responsible for the original damage to the site on Monday. This is the likely cause of Lieberman’s site being defaced.

1. Massive amounts of traffic to the site caused things to slow to a crawl on the shared server.

As the election approached, numerous websites, blogs and news programs were actively referring traffic to Lieberman’s web site. The amount of traffic being referred here is significantly greater than the traffic that Joe2006.com typically generated (and massively greater than the traffic experienced by any one site on the shared server like Azul Pool and Spa Services).

The immediate effect of this traffic would have been that all the services being provided to Lieberman’s campaign would have been massively slowed down. Web pages would load slowly. Email would get delivered slowly and outgoing email might have come to a stop. For a site not accustomed to this level of traffic, this might have seemed just like a Denial of Service (DoS) attack.

Without more information from Dan Geary or Lieberman’s campaign, it’s extemely difficult to determine what specifically caused the site to go offline (though it seems clear that Geary, himself, took the site offline to mitigate the massive traffic hitting the site).

At this point, this seems like the most likely scenario to come forth. Very likely.

While I’m very technically savvy, there are many others out there who might have other ideas on what could have caused this situation. Feel free to add your comments and let’s discuss.

[UPDATE] There’s a great article on KOS that has recently been posted that I completely agree with. It does a great job outlining why no webmaster worth their paycheck would ever or could ever leave Lieberman’s site down so long.

[UPDATE 2 - August 10th 8:30AM] On Wednesday, August 9th, Justin Rood of TPMmuckraker posted a story with an interview with Sam Hubbell, the owner of MyHostCamp.com. In his interview Hubbell describes the situation as a denial-of-service problem that originated from within the Joe2006.com hosting account itself, which is less of a denial-of-service and more like a software issue. If the site had indeed been compromised through an insecure module of Joomla! and then loaded up with some nasty software that began to overload the server with emails to itself. Hubbell described it this way, “It seemed like it was internally spamming itself, and there was also potentially an outside source that was hitting it.”

Given that the problem seemed to be localized to the software loaded on Joe2006.com, this is a very damning admission since this means they could have loaded up a fresh server with software, put Lieberman’s content on there, and the site would have been back up almost immediately.

So I continue to believe that causes #1, traffic overload, and #2, a hack to Joomla!, are the likely cause and while Hubbell claims to have installed all the latest patches to the software, it’s beginning to seem like Joomla! may not have been the best choice for Lieberman’s site given the significant security holes patched each month and the number of hackers out there trying to bring down Joomla! sites.

[UPDATE 3 - 8/11/2006] Lieberman Campaign lied about its ability to “get out the vote” via email!