Sometimes the best-laid plans must be laid to rest.
Events of the past week proved the validity of that phrase.
This time a week ago, we embarked on a major upgrade of our computer server to reduce downtime and improve speed/reliability. I set the timing for the work for the Canadian Victoria Day long weekend (yes, named after the old Queen of England) — one week before the current US Memorial Day holiday. The extra time would give us some margin for error, and in fact, everything seemed to be going okay — at least until we noticed some irregularities with the Domain Name Server.
Turns out, our DNS address (the web address that actually points to our sites/server) should have been okay, but our Internet Service Provider was concerned about duplicate/caching issues, and so we set a new DNS. No problem, I thought — we have an extra day for a margin of error, and indeed by Monday evening, everything seemed fine — but it wasn’t.
There was an issue with DNS Propagation — it seems about half the world (including about half the ISPs in North America) didn’t see the new DNS address — and so were coming up with blank screens on our sites. As well, other functions weren’t working right.
Finally, by Wednesday, the DNS resolved — just a few hours before I discovered we had lost interconnectivity at my home office.
Nothing can create a stark reminder about technology dependency as powerful as the loss of Internet access.? Of course I called our service provider, and after some diagnosis, the carrier decided that we would need a telephone company technician to visit the home — and after further delays, Bell set the technician visit time for Friday evening.
For the next two days, hindered without reliable access, I used my cell phone data plan and borrowed public WIFI access as best I could to maintain essential activities, but virtually everything not immediately urgent needed to be deferred. Today is “catch up day”.
It turns out that a technician trying to fix someone else’s problem, switched the Internet links to our address to another location — cutting us off completely. At least I won’t have to pay for the service call, and I’m thankful that our company’s servers are hosted off-site (in the US, in fact.)
None of these problems were killers — we were able to hold the fort and carry on, and there were backup processes and systems in place to minimize any real harm.
Yet I’m reminded about how easy it is to take things for granted, and how little unplanned things can cause big problems. It is good to get back to work as usual.
