Really? I accused another campaign of something it didn’t do?

Only 612 days after Joe Lieberman’s Joe2006.com web site crashed and burned, causing his campaign to claim the vicious bloggers at DailyKos.com and Ned Lamont’s campaign had hacked into poor Joe’s server and broken it, we now learn through a Freedom of Information Act request that the FBI long ago determined that his server simply crashed. It was never hacked. It was never subject to a Denial of Service attack. The server administrators had a mis-configured server on their hands. The Joomla! hacker never defaced the site.

Where’s the apology, Joe? Obviously those in the Lieberman campaign were aware of this finding a long time ago (the FBI email obtained by the AP was sent in October 2006) but details on the outcome of the investigation were never released. Seems like some political hackery was up. This information might have been useful to the voters in November 2006.

“Senator Lieberman’s campaign team accused an awful lot of good people of breaking the law on the eve of the primary, and they did it for political purposes,” Lamont told the AP in a telephone interview. “If he does the right thing, he’ll stand up and say, ‘I was wrong.”

We’ll never know how much this incident influenced the voters of Connecticut when they all got “hepped-up” on Joementum and put this idiot back in the Senate, but it certainly could come back to haunt him if he tries shacking up with McBush for the general election.

bloggers have added another dimension of vituperation toxicity

Well… I’m not exactly sure what you mean by putting those two nouns together–vituperation toxicity–but if that means that bloggers are an extra set of eyes calling out your terrible record of supporting Bush’s war in Iraq or the false charge that Ned Lamont’s campaign hacked your web site…then I’m happy about it.

Get the whole transcript and video at [Think Progress]

Zone-h.org spoke on August 10th with a Turkish hacker who admitted to defacing Lieberman’s site. Was this Turkish hacker working for Ned Lamont? Had he been enlisted by legions of bloggers to ruin Lieberman’s chances? Not even close. Here’s what Roberto Preatoni discovered:

We tried to contact the attacker who disclosed that he indeed attacked Senator Joe Lieberman’s website and defaced it, but being a Turkish guy, he really didn’t have a clue about who Senator Joe Lieberman was. Being Muslim we asked him if his attack was anyway politically motivated and the answer was: “I did it just for fun”.

Being also asked if he was the coordinator of the Denial of Service attacks which have been effecting Senator Joe Lieberman’s site he declared that what he did was just to deface the site, then moving to the next target. ”

Did you get that last bit? The hacker admitted to defacing the site, but simply moved on to the next vulnerable site in his list. As I reported before, this was no Denial of Service attack.

I can vouch for the risks that unpatched sites running Joomla! run. After an automated scan of my server’s pages on August 20th looking for mentions of Joomla! and the ext_calendar module (both terms would be found in that article), a computer based in Turkey attempted to load a hack onto my system. Fortunately, I’m not running Joomla!, and it certainly backs up the story that this was simply a random incident and Lieberman’s site was defaced along with probably hundreds or thousands of others that day.

As Rotophonic.com and TPMmuckraker reported last week, the Lieberman campaign’s ability to reach its supporters had not been completely disrupted, as they originally claimed. And the questions (here and here) about the robustness of their web hosting and the wisdom of their technical staff will surely color the investigation. I definitely know the FBI won’t appreciate being pulled off counter-terrorism detail to investigate the failure of Lieberman’s staff to adequately build, operate and manage its own website.

Rotophonic.com and David Sirota at The Huffington Post haven’t forgotten about the story and eagerly await the publication of any new details about Lieberman’s website failure and an explanation for what really happened.

So this morning in my daily perusal of TPMmuckraker, I was not surprised to read this story pointing to a MSNBC report that the Bush Administration rushed the arrest of suspects in the British terror plot. In particular was this quote from the original MSNBC article,

the suspects had not yet purchased any airline tickets. In fact, some did not even have passports

Strangely, the MSNBC reporters attribute America’s eagerness to arrest the subjects with this rationale:

Analysts say that in recent years, American security officials have become edgier than the British in such cases because of missed opportunities leading up to 9/11.

Edgier than the British? Really? Do the reporters writing this story actually believe that some egghead analyst in the CIA actually convinced the White House to lean on Tony Blair to arrest the terror cell? The nerds in CIA live in the reality-based world and the White House lives in its ideological dream world where Iraq continues to go according to plan, the people of the Middle East see us as glorious liberators, the weapons of mass destruction were found and the American public sees Bush’s condescending smirk as a cute mannerism.

Should we believe that a very complex, passionate argument had been going on between various departments in the U.S. Government over the last few weeks, resulting in the suggestion that the British make arrests now versus later? Or should we use Ockham’s Razor cut through the crap and determine that Bush/Rove thought the timing would be better on Wednesday to blunt the effect of Lieberman’s embarrassing loss?

Don’t forget the Bush aide who said, “We’re an empire now, and when we act, we create our own reality.”

When I was doing research for my earlier story on the reasons for Lieberman’s ‘hacked’ site woes, I noticed, in reviewing the cached pages of Joe2006.com in Google for references to Joomla! modules, that the web forms, like the “Newsletter Signup Form”, included on the home page of the Joe2006.com website were not being submitted to Joe2006.com, but to another company…Blue Hornet. Blue Hornet clearly handled all the heavy lifting involved with the collection of email addresses for Lieberman and the management of their mass-email communications.

When the email was delivered to Lieberman supporters early on Election Day, those emails originated from Blue Hornet. As such, the Lieberman campaign had mass-emailing capabilities not only in the morning of Election Day, but also continued to have this capability throughout the day. And they still do!

This statement by Sean Smith, in public as well as on the web page posted on Joe2006.com for days, was demonstrably false:

email has been totally disrupted and disabled” – Sean Smith

I guess he shouldn’t have used the word totally.

Did a member of Lieberman’s staff specifically prohibit the use of Blue Hornet’s services after their website was taken offline? It’s puzzling that Sean Smith was speaking to reporters about how the downing of their web server had suppressed “voter participation and undermine[d] the voting process on Election Day”, while the Lieberman campaign continued to have all its capability to “get out the vote” through email blasts. Discovering this, I’m actually happy the Feds are investigating.

“We call on Sean Smith and Joe Lieberman to make an unqualified statement denouncing their campaign’s moronic smear campaign on bloggers and his winning opponent and we demand whoever is responsible to continue to bring us more comedy fodder for three more months. All Joe’s attempts to suppress voter participation and undermine the voting process on Election Day were deplorable and have no place in our democracy.”

Satirically yours,

Rotophonic

I thought I’d walk through some of the theories out there about how the site got taken down and discuss the merits of each one. The list appears after the fold. There are some surprises in the top two.

5. Failed to pay the bill

This is a quick way to get your site taken down, but it’s an unlikely one. Dan Geary, Lieberman’s Internet consultant, seems to have a very cozy relationship with the hosting provider that was selected to host the site, MyHostCamp.com. Geary told TPMmuckraker that all of his company’s websites are on that server and that it’s owned by someone he works with all the time.

This hosting reseller, MyHostCamp.com, is a shared server, meaning that a single computer has many virtual sites running on it. Lieberman’s server had over 70 sites running on it. MyHostCamp.com definitely hosts its server at and probably rents a managed server from ThePlanet.com, which sells managed and semi-managed servers to resellers. Even if MyHostCamp had the most expensive server offered by ThePlanet.com, the total bill would likely be around $600 per month, divided by 70, equals less than $10/month per site in costs.

Given that Geary probably pays the bill for this server and gets paid by the 70 plus sites using it, it’s definitely not a billing issue.

4. Ran out of bandwidth and got shutdown

Very unlikely as ThePlanet.com offers a number of bandwidth options, including an unmetered service at the high end. Plus, other sites running on that server continue to run now while joe2006.com has been taken down. This is also easy to fix with your provider. If this had happened, the site would have been back up with one quick phone call. Running out of bandwidth didn’t crash the site.

3. Denial of Service Attack took down the server

Based on statements by Geary (“When we take the site down, the server is fine.“), content within the server was corrupted and thus the site needed to be taken down, but other sites running on the same hardware continued to run. Any denial-of-service attack would have affected these sites equally and would have subsided once steps were taken to avoid the attack.

Further, the datacenter where Lieberman’s site was hosted offers a number of specific protections that help mitigate any DoS attack, including Arbor Peakflow DDoS Detection and Cisco Guard DDoS Mitigation.

Lieberman’s site was not downed by a real DoS attack.

2. Joe2006.com’s Content Management System Hacked

Joe2006.com used a content managment system called Joomla!, a widely used system for setting up websites. As with most pieces of software, Joomla! is frequently targeted by hackers who try to exploit weaknesses in the system to deface or take control of sites running that software.

It’s difficult to determine which version of Joomla! Joe2006.com was running or the versions of specific components that were in use on the site, but examining Google’s cache of Lieberman’s site shows that parts of Lieberman’s site definitely used a component called com_extcalendar, which allowed the site to display a calendar with events.

The most recent serious problem with the Com_ExtCalendar component was discovered very recently–on July 7th, 2006. This issue would allow a hacker to deface or even overwrite the entire configuration file for the site. Others have written about “script kiddies” spending their summer vacations attacking Joomla sites, including those with this component. Geary told TPMmuckraker that, “We have nobody with a security background helping with this. It’s just us, what we know, how we work with our server network.” I read this as, “We just use the webserver control panel and know how to upload stuff via FTP.”

Given that Dan Geary indicated that he was flying blind with all the technical issues, it seems very unlikely that their server had been updated with the latest and greatest version of the Com_ExtCalendar component that fixes the security issues.

Geary told MSNBC that the hack on Monday August 7th involved the site being defaced and that later massive amounts of traffic to the site were linked to the failure of the site. A hack by someone taking advantage of the weakness in components of Joomla! would not have generated traffic, but could definitely have been responsible for the original damage to the site on Monday. This is the likely cause of Lieberman’s site being defaced.

1. Massive amounts of traffic to the site caused things to slow to a crawl on the shared server.

As the election approached, numerous websites, blogs and news programs were actively referring traffic to Lieberman’s web site. The amount of traffic being referred here is significantly greater than the traffic that Joe2006.com typically generated (and massively greater than the traffic experienced by any one site on the shared server like Azul Pool and Spa Services).

The immediate effect of this traffic would have been that all the services being provided to Lieberman’s campaign would have been massively slowed down. Web pages would load slowly. Email would get delivered slowly and outgoing email might have come to a stop. For a site not accustomed to this level of traffic, this might have seemed just like a Denial of Service (DoS) attack.

Without more information from Dan Geary or Lieberman’s campaign, it’s extemely difficult to determine what specifically caused the site to go offline (though it seems clear that Geary, himself, took the site offline to mitigate the massive traffic hitting the site).

At this point, this seems like the most likely scenario to come forth. Very likely.

While I’m very technically savvy, there are many others out there who might have other ideas on what could have caused this situation. Feel free to add your comments and let’s discuss.

[UPDATE] There’s a great article on KOS that has recently been posted that I completely agree with. It does a great job outlining why no webmaster worth their paycheck would ever or could ever leave Lieberman’s site down so long.

[UPDATE 2 - August 10th 8:30AM] On Wednesday, August 9th, Justin Rood of TPMmuckraker posted a story with an interview with Sam Hubbell, the owner of MyHostCamp.com. In his interview Hubbell describes the situation as a denial-of-service problem that originated from within the Joe2006.com hosting account itself, which is less of a denial-of-service and more like a software issue. If the site had indeed been compromised through an insecure module of Joomla! and then loaded up with some nasty software that began to overload the server with emails to itself. Hubbell described it this way, “It seemed like it was internally spamming itself, and there was also potentially an outside source that was hitting it.”

Given that the problem seemed to be localized to the software loaded on Joe2006.com, this is a very damning admission since this means they could have loaded up a fresh server with software, put Lieberman’s content on there, and the site would have been back up almost immediately.

So I continue to believe that causes #1, traffic overload, and #2, a hack to Joomla!, are the likely cause and while Hubbell claims to have installed all the latest patches to the software, it’s beginning to seem like Joomla! may not have been the best choice for Lieberman’s site given the significant security holes patched each month and the number of hackers out there trying to bring down Joomla! sites.

[UPDATE 3 - 8/11/2006] Lieberman Campaign lied about its ability to “get out the vote” via email!