It’s been an interesting week. I’ve italicized interesting because the problems (and solutions) combine some expected and predictable issues, and some surprises, largely related to information technology and our network of websites, including this blog.
The challenge: We were experiencing extremely slow loading time on our sites, plus experiencing severe email delivery issues — even though it seems the standard spamblock systems (on the surface) seemed to be giving an “all clear”. Nevertheless, I knew there were problems, and believe they originated with some bugs/flaws in our eletter delivery program that caused “bounces” (undeliverable emails) not to be properly recorded — possibly causing enough complaints or indicators to turn off some email reputation/managing services. As well, much to my surprise, I experienced the “white screen of death” on one of our sites — who our offshore consultant told me, on review, had been hacked.
The solution: Redesign the back-end of our systems, build in more safeguards and management tools, and swap everything off of our existing server to a new one with fresh IP (Internet Protocol) addresses. But while in theory this process isn’t that complicated, it certainly took a lot longer to complete than I had expected, and not without some serious snags.
In the first place, the back-end reprovisioning, where a variety of domains were moved from a single account to different “cpanel” accounts took much longer than expected. Then we needed to copy the files — a process that took more than a day. During that time, the internal server load was so great that the sites were effectively “down” (and I couldn’t blog here). Finally, after things were moved, I switched the DNS (Domain Name Server) on all of the websites and thought things were done. But there was a flaw in some PHP script elements that caused the sites to go to another white screen with an obscure error message. That problem took another day to correct.
Things are mostly back to normal now. Our entry/information request forms are working okay, and the email management software indeed is generating proper blacklist entries. However, there are still some challenges with email delivery, and I expect I’ll find more glitches and bugs as we go along.
Yes, it has been a pain, and costly for momentum and other projects/marketing objectives. I realize our IT problems aren’t quite as bad as some I’ve heard that either brought businesses to the ground. (One of our printers, a major commercial operation, is still recovering from the flawed introduction of a new integrated IT system that was supposed to put everything together — instead it virtually decimated the business when it delivered false results, or simply didn’t work — forcing staff to revert to totally manual processes.)
There are also the rewarding moments, when I found shortcuts and enhancements. The thought of switching DNS records individually for 30 domains scared me, until I searched Google how to handle bulk swaps, and completed that task in 30 minutes.
In some respects, I’ll be happy now to take a break from IT issues. But I realize that if everything is going smoothly, it is probably time to break things and update/enhance them — because this story never really ends.
