It happens every year about this time. Even before Thanksgiving, and certainly by the time Christmas rolls around, computer systems start crashing.
Most corporations institute what is called a “change freeze” for the last couple of weeks of the year. One of the largest reasons is financial. Or, more accurately, financials. Companies have to close out their books at the end of the year. They have to get out W2 forms in the first thirty-one days of January. They don’t want to risk anything delaying their ability to get those financials posted by the end of the year.
Okay, great, a change freeze sounds like it should help prevent outages at the end of the year, not cause them. So, what gives?
Project Managers. Many projects have the end of the year as either a formal or informal deadline. Also, many companies do their reviews based on a calendar year. That means, if you are a PM and you want to get that big bonus (ha ha, actually, I’ve never heard of a project manager getting an end of year performance bonus. But, I digress) you want to get your project rolled out by the end of the year.
That doesn’t mean December 31. That means before your change freeze date. Many projects get rolled out in November and the first half of December.
It’s said that the greatest period of instabiility for any system is the week after your team gets back from a tech conference. They just can’t wait to try out the new tips and techniques they learned.
The same thing happens with end of year changes. Modern computer systems are terribly complex. And sometimes just terrible. But, even the best architected and maintained system has an incredible number of moving parts. In fact, most systems are too complex for any one person to understand all of it.
And new systems and programs have to integrate smoothly into the existing architecture. You do that by testing. But, testing, no matter how good, can only get you so far. It’s typically impossible to truly test a complex system without actually having a test system that is as big as your existing system. Even if you could do that (and virtually no one can,) you still couldn’t replicate all the users on the system at once.
What happens is that much of the testing is actually done in a “live” system. And the whole point of testing is to find weaknesses. Those weaknesses, when exposed in a production environment results in problems and outages.
Every year we see it happen. It’s like clockwork, or the turning of the seasons. We’ll spend the next couple of weeks “testing,” putting out fires, testing some more, and finally, the change freeze date will arrive to relieve the stress. Almost like Christmas. . .Just a couple weeks early.
Rodney M Bliss is an author, columnist and IT Consultant. His blog updates every weekday. He lives in Pleasant Grove, UT with his lovely wife, thirteen children and grandchildren.
(c) 2019 Rodney M Bliss, all rights reserved