The website hosting dilemma
The bitterness of poor quality remains long after the sweetness of low price is forgotten.
— Ben Franklin
A few months ago, I had the unfortunate experience of helping a client through a website hosting crisis. This client wanted to support local businesses and chose a website hosting company located locally. The company they chose offered shared hosting. They were with that company for a couple of years and in that time the hosting experience was tolerable. The PHP versions were reasonably up to date. The response time for the server was acceptable most of the time. Even still, there were continual downtimes for a few minutes and then the site would be up again. Response to support requests were very slow. It wasn’t uncommon for us, after submitting a support ticket, to not hear anything back until 5 hours later. There was a live chat feature but it was rare that the first person who responded to my chat was able to answer my question, without handing the question off to someone else.
Servermaggedon
Well, that brings us to the dreaded weekend when we woke up to the site being down. My first approach when a site goes down is to make sure it’s not a programming error. I look at the response codes on the page. If there is a problem with the programming, that affects the server, it will usually either create a 503 server error or a blank page. But this time, it timed out.
Next I check the error logs. They can usually be found in the website hosting panel (cPanel). When I went to login there, the hosting panel site also timed out!
I pinged the IP address of the server. The IP address is the internet address of the computer server where your website files are stored. I basically knocked on the door. No one answered. The request timed out. I ran a tracert which follows the path through the internet that the connection is trying to take. I quickly received a series of asterisks as a response. All of this points to the server being completely down.
Is it just us?
At this point I submitted a support ticket to the website host (whose own site was unaffected), in order to find out what was going on. When I first logged in, their Network Status page said everything was fine!
Since they are slow to reply, while I waited for a response, I searched for some more information. I came across this old but incredibly helpful article from Smashing Magazine: What To Do When Your Website Goes Down. Ah, yes! Ping neighbouring IP addresses.
If your power goes out, it can be helpful to know if your neighbours have power. Is it just your house or a wider issue?
I went through a series of neighbouring IP addresses and they were all down as well.
So far this took me all of about 10-15 minutes.
I contacted the client and let them in on what was happening.
And then — the website hosts’ Network Status page changed to “major incident in progress.”
I immediately checked the backups I had been taking, that are hosted on a separate server. The latest site backup had finished right before the outage happened. Thank goodness!! I downloaded the backup to get it ready to restore, if need be.
Data wiped on ten servers
We finally got a reply to our support ticket almost 4 hours later that included a link to a “Live Updates” page.
The data on 10 separate servers were wiped clean. The website host was working on recovering sites from backups but some of the hosts’ backups were irrevocably damaged.
By 11am the next day, the server that my client’s site was on was only 20% restored.
By 4pm, we hadn’t received any further emails from the website host. The live updates page still said 20% complete. But, we noticed that if we browsed to the site, there was now a notice up that said that the IP address might have changed.
I tried logging into the website hosting panel and sure enough, I could log in and the IP address had changed. It looked like the site was moved temporarily to a new shared IP address instead of the dedicated one we used to have and the space the site was moved to was too small to support the site. There was a warning that we were at 120% of our file usage.
I updated our DNS (domain name) to point to the new IP address and, once the DNS propagated, the site was back up — sort of.
Over the next two days, my inbox was getting continual emails from my uptime monitors. The site was up and now it’s down and now it’s up again. My client’s staff couldn’t update any content and couldn’t upload any files.
I submitted another support ticket inquiring about the issues and asking for an update.
The site went down again
The next day, over 24 hours since I had submitted the support ticket, I still hadn’t received any reply at all. And the site was down again. They had changed the IP address back to the dedicated one that we had originally, in the middle of the day, without telling us anything about it. I repointed the DNS and, once it had propagated again, the site seemed much more stable and functional. In total, that took 5 days to get the site back to it’s original condition, with almost no instructions, contact or feedback from the website host.
I should add that I did eventually finally get a reply to that support ticket 4 days later, explaining information we had already assumed — that they had temporarily moved the site to a new IP address and then moved it back — without telling us to update our DNS.
They posted a blog explaining the incident. Some phrases really stood out like (the emphasis is my own), “Within only hours our incident response team had identified the issue …” Hours!
An attacker can do a massive amount of damage in an hour.
Lessons learned
The website host was dealing with a crisis. I do get that. But the fact that it happened in the first place, that they took hours to notice and respond to the attack, the way that they handled fixing the issue, and the lack of planning ahead for a crisis, are the real issues here.
At least they were fairly upfront about it.
At least.
This was the worst server outage I have ever seen. Many years ago I had a site on shared hosting that was down for 8 hours and the website host never said what happened. But immediately after the 8 hours, the site was back and fully functional. I changed website hosts after that.
I’ve seen a lot of very random website hosting panels. A disconcerting amount are not using SSL certificates, don’t require strong passwords, do not use the latest software versions, and don’t support sFTP or SSH for file transfers. All of your website files and your databases reside on that hosting panel. If your hosting is hacked, far more damage can be done than if your WordPress admin dashboard is hacked.
Also, depending on the quality and security of the website host, if a hacker gains entry to your hosting account, it might affect other sites stored on the same server.
What if?
I can’t even imagine how much worse the situation would have been if my clients had also hosted their domain name and email accounts on that server. Or what if I had reseller hosting and hosted all of my clients there? I could have lost everything.
I have a policy where I always recommend that my clients host their domain names with a separate domain host and email with a separate email host. In an event like this, that empowers them to switch to a different host, as soon as the server goes down, then restore their site from their own backups and repoint the domain name. The problem can be resolved in a couple of hours without being at the mercy of a third party.
The client this happened to didn’t want to switch hosts under that much pressure and decided to wait for the site to be restored. I don’t think anyone could have anticipated that the restoration would have taken that long or would have been that rocky.
It’s 2022!
You might not be aware of this but these days, so much of this is preventable. There are strategies that can be used to ensure consistent uptime, good security and excellent performance. Both for very large and very small websites.
Having a website host that monitors your server 24/7, makes changes and tweaks to optimize performance, replies to support requests immediately, has full security protocols in place and follows them and that supports you also following security best practices — that exists too.
Big promises; little returns
When you’re looking for great website hosting, please do not get drawn in by offers of “unlimited” space or bandwidth. All websites are hosted on actual, physical computers with hard drives or SSDs, similar to every other computer. That computer server simply doesn’t have unlimited space or bandwidth. When a company says “unlimited”, what they mean is that when your website outgrows the space, they’ll fix it. That said, they are very much counting on not having to fix anything.
Let’s say website A and website B share a server. If website A is large and website B is small, website A will be allocated more space even if both websites have signed up for the same plan and are paying the same amounts. When a website host sells you an unlimited plan, you are left simply not knowing what you’re actually paying for and whether or not space will be available when you need it.
What if website A goes viral or has a DDOS attack? That will affect your website too.
Often times “unlimited plans” are a clue that the website host cares more about profits than quality of hosting. If your site requires more space, what incentive do they have to provide you with the space and resources you need? You have an unlimited plan. They can’t charge more, so they will avoid upgrading your server resources at all costs and wait until the last possible minute. Or they may try to blame you or your website for the poor performance.
Whereas if a website host sells you 20GB of space. Then you can rest assured that that amount of space will be there for you when you need it. And you can plan ahead. Are you running out of room? Great! Upgrade now before your site performance goes down. When you know exactly what you have, it makes planning ahead possible. And when you need to upgrade, the website host is happy to oblige because they are charging you more for that, as they should. It’s also more fair. Why should your small site be paying the same for hosting as a corporate website? Especially if that huge site is using a lot more server resources.
Website hosting is worth more than a cup of coffee
Keep in mind that providing excellent website hosting services requires a team of knowledgeable experts working around the clock. That costs money. That is not achievable for $10/month. If your hosting costs that, the website host is definitely cutting some corners. So that means that space, performance, support or security will be compromised. Or maybe all of those things. That also means that they are not actually monitoring your server around the clock, or that they aren’t paying their staff enough or that they didn’t hire enough staff and the staff is over-worked.
Minimum wage for hiring a teenager is $15/hour or so. At $10/month you only covering 45 mins of their time; let alone a well-educated expert. The website host can’t afford to give you proper support, because you simply aren’t paying them enough.
If your website is hacked or your site goes down for an extended period of time, it can wreak havoc on your SEO. Your site can get pulled from Google and may need be resubmitted. Trust, that you spent time building with your customers, can be lost. And think about the cost of the sales you aren’t getting while your site is down. Also fixing a hack, or even the time I spent troubleshooting the outage and downloading backups — that all costs money as well.
A little extra per month on hosting can prevent that entire headache and can help you maintain your visitor’s trust.