The Bluehost massive networking failure was kind of like this from the customer’s perspective.
Back in April, we decided to move website hosting for this site to Bluehost. We had outgrown our old hosting service and needed a more robust solution. On recommendation from some other Bluehost customers, we moved all of our commercial sites to Bluehost.
Over the past 11 months, we have had intermittent problems with Bluehost. My webmaster indicated that Bluehost would go offline more often that she was comfortable with, which apparently affects our rankings with the major search engines. There were other times that our site would come to a grind halt, no doubt as a result of too much traffic from our ‘neighbors’ (we had shared hosting).
Not satisfied with the anecdotal downtime data I was receiving, I setup free monitoring on one of our Bluehost sites using Pingdom, so that I could measure Bluehost’s reliability. After all, what gets measured gets managed (or replaced). Pingdom checks every 5 minutes to see if my sites are up and sends me a text message when it is down and again when it comes up again. I set this up Monday, July 29th, 2013.
So 4 am on Friday, August 2nd, I started receiving text messages: site down, site up, site down. Over the next 13 hours, it must have been hell at Bluehost, and our sites were down for at least 8 hours, and probably closer to 10 hours. In the end, it appeared to be a network issue that cascaded throughout their data center and affected Bluehost, Host Gator and Just Host, all of which are owned by Endurance International Group (EIG).
Fortunately, we had already been researching options other than Bluehost. We moved to another hosting company, upgrading to a virtual private server (VPS) along the way, and one not owned by (EIG).
What are the lessons learned? There are several:
- Keep your registrar different from your host. If your host goes down, like Bluehost did, and you need to change nameservers (where your host is), you can point traffic to your new host. I was able to migrate the matricellc.com site within a few hours.
- Backup often and store the backups somewhere other than your host. Some of our sites are backed up daily, others weekly. Worst case scenario is that we have to restore to an older backup, but it is sufficient if it keeps the sites up.
- Monitor your hosting company with a third-party tool or service. I have created a spreadsheet for monitoring uptime, based on down/up alerts I receive from Pingdom.
- Follow Twitter feeds when issues arise. This may seem obvious to some, but some of us are not heavy Twitter users, so this idea only came to us after the issue was nearly resolved.
- Be ready for disaster.
I am guessing that a lot of people will be moving away from Bluehost, Host Gator and Just Host as a result of this outage. I’m sure the losses for their customers is in the millions of dollars. The costs that come to my mind are:
- Lost sales from e-commerce sites;
- Lost advertising costs for those people who clicked ads on Google, Bing, Yahoo, and other advertisers;
- Lost productivity from those who manage and use the sites that are hosted there;
- Opportunity costs.
This may appear on the surface to be a knee-jerk reaction, but it was really just the event that pushed us over the edge. We had three live sites (one of which is heavily visited), and one development site that were affected. I find it ironic that their hosting went down in the same week that I started monitoring. So we migrated this site to a VPS and we hope that you enjoy the speed and reliability. I can tell you that we do.