Short version.  At approximately 6:30 pm on June 30th, the CVR website completely stopped working. Our current hosting company was unable to help fix the problem, so over the holiday weekend, the website was migrated to a new hosting company. The migration was successful, and the CVR website came back online Monday evening at around 6:30 pm and is performing better than ever.

Long version:

Images were slow to upload to Challenge. The process of saving an article or photo gallery didn’t complete and you have to hit the save button a second time or reload the page. Members would click a link on the website only to have the web browser display an error message that there was a bad database connection, or the page took too long to load.

The signs were there.

Assuming the problem was with our basic hosting package, your intrepid web team added a CDN (Content Delivery Network) to the website.  For the non-geeks out there, a CDN distributes versions of the website to servers all over the world so that if one server fails, another server will take over and the website should never (in theory) be unavailable. CDNs are cheap to use and 100% reliable when set up properly. In our case, because we were using a CDN to compensate for problems with the hosting platform itself, it was like holding the exhaust on your car in place with scotch tape and baling wire. It works, but you know it ain’t gonna last.

The plan was always to launch Challenge as a digital property built on WordPress in 2019 and then rebuild the main CVR webpage (also using WordPress) in 2020. Then COVID happened and Phase 2 was postponed. The redesign of the main CVR site was planned as a 2021 project and everything was going as planned until March when the main CVR website died. It seemed our hosting company was no longer able to support the version of PHP the main website used to operate, and one day the old girl simply stopped working. (Crash #1 if you’re keeping score.) Shelley, Mike and I quickly put together a new CVR website that was built on top of the Challenge framework and loaded it onto the cvrpca.org domain. To make that work, we had to do some rather unorthodox programming.  More scotch tape and baling wire.

On a good day, the new CVR / Challenge / Club Race website was slow. And on a bad day, of which there were more than I was comfortable with, the website was downright unstable. There wasn’t a day that went by when someone out there didn’t complain about the problems.

It wasn’t like we were unaware of the problems or that we weren’t working to fix them.  I personally spent 8-10 hours a week on the phone with Bluehost. Eventually, the diagnosis from their technical support team was the problems were our fault. We (CVR) were running our website on the least expensive of the hosting packages that Bluehost offered (true) and that our website would benefit from an upgrade to a more powerful hosting package. One that came with a dedicated IP address, more memory, a larger hard drive, and more PHP helper apps.

Websites, like engines, can always use more power.  If adding a turbocharger to a lawn tractor was good enough for Tim Allen, then adding additional memory and storage space to CVR’s website couldn’t be a bad thing. This, we were promised, would solve the problems. So at the end of March, we purchased a “much better” hosting package and on April 5th, Bluehost began the migration.

That’s when the real problems started.

At first, the addition of more power and a dedicated IP address seemed to improve things. We had also taken the additional step of building a brand new version of the website on the cvrpca.com domain, and as part of the migration process, we turned off the cvrpca.org domain, activated the cvrpca.com domain and redirected all of the traffic that went to the .org domain to the .com domain. The efforts seemed to work: The .com website loaded faster and there were fewer Gateway or Page Unavailable errors. Perfect? No.  But measurably better? Yes.

Then on Tuesday, May 18th at 6:59 pm, nearly 6 weeks after the process was started, Mike Keller (in his capacity as Webmeister) received an email from the hosting company notifying us that the migration to our upgraded account had been completed. Mike immediately forwarded that email to me, but I didn’t get it for several days. That’s because, shortly after Bluehost finished the migration, all the CVR email accounts stopped working.

The first I knew of the problem was about 12 hours later when some members told me (via my business email) they were having problems logging into their CVR email accounts. I tried mine and had the same problem. So I called Bluehost. I called them once on Wednesday, twice on Thursday, and three times on Friday. By Saturday and Sunday, I was calling 4 to 5 times a day.

It was the following Tuesday – a week later – when the email problem was diagnosed and fixed. Apparently, as part of the migration, our email server was also moved to new hardware and given a new IP address, and somehow, that caused all of our logins to fail. (Bluehost said, We’ve never had that happen before.) Once diagnosed though, it was a fairly simple process to fix, and instructions were sent out to everyone who was impacted.

But the website was still slow. Slower than it should be for a relatively small website and slower than we’d been promised it would be. In addition, we had some stability issues that showed up around the time of Club Race that began to happen more and more frequently.  Something was still wrong.

After Club Race, things got worse, and the number of times web pages were unavailable grew exponentially. I’d long suspected that part of the problem was the .htaccess file that handles redirects.  (Redirects are used to send users clicking on a link from the .org version of the website to the .com version of the website.) When redirects are set up properly they are seamless to the user and work instantly, but as I started looking at our problems, that wasn’t the case. The redirects Bluehost had put in place for the .org version of the website weren’t working, and some of the members who were writing articles for Challenge were uploading them to the old version of the website without realizing it. I didn’t want to touch the .htaccess file, and after discussions with Allen and Mike, we decided to solicit proposals to address the redirects problem and migrate the website to a new hosting company.

The first developer we reached out to didn’t want to touch the problem. The second submitted a proposal with a price tag of $8,000. Mike and I thought that was a little high and decided to hold off on accepting that proposal while we looked for additional options.

In the meantime, the website was getting worse by the day.  I decided to tackle the .htaccess file myself and, for a few weeks, those changes sped things up.  And then on June 30th, at about 6:30 pm, the website crashed (#2 for you scorekeepers). The chart below shows that crash in stark terms:  The white space represents “uptime” – that is the time when the website is available for use.  The red blocks represent “downtime” which is the time when the website is not available. And as you can see, from June 30th until July 5th, the website was, for all intents and purposes, offline.

The timing of the crash meant that the July Challenge that was scheduled to go out at 7:30 am on July 1st couldn’t be sent. Not only would members not be able to read any of the articles, but the email itself couldn’t be built because the RSS feed that contains the articles relies on a working website, and we didn’t have one.

Something had to be done.

Allen, Mike, Shelley and I had been in constant contact about this situation, and Allen asked me to please not work on this over the 4th of July weekend. I agreed, but then it rained, and I was stuck inside with not a lot else to do.  I decided to tackle the website issue once and for all.

I reached out to a hosting company called Kinsta.  Kinsta specializes in WordPress hosting, they have a global footprint, are a little cheaper than WPEngine (the “gold” standard), and came highly recommended by some of my peers.  So I reached out to them on Saturday morning. After some discussion of our needs, I purchased an annual hosting package from them. The selected package came with “premium migration” which meant that they’d handle everything involved in transitioning the website from Bluehost to their environment. The timing was 1 -2 business days, so I assumed the new website would be live by Wednesday of this week.

Two hours later, I was contacted by a Kinsta technician.  He told me that he was starting the process and that he’d reach out to me periodically with any updates. By 4:00 pm on Saturday, the transfer was 99% done but Jim told me that he was going off shift and would finish the process Sunday. At 3:00 Sunday afternoon, I got an email from Jim stating that the migration was done and that the site was available for testing.

I spent hours testing the website. Everything worked. Everything was faster. Best of all, as you can see from the chart above, there have been no downtimes since the move was initiated (the green line).  Not one.

As the chart above shows, CVR now has a faster website than PCA National has.

Now we’re on to the next step. Migrating the email from Bluehost accounts to Google Workspace accounts.