I don’t know why IT projects always have to kick off in the middle of the night. I mean, after 30 years in the industry I do understand why we kick off our projects in the middle of the night. I’m just not sure why we weren’t smarter about picking a start time.
1:00AM
Of course, that was East Coast time. It was only Midnight in Central time and an early 11:00PM for the guys in Colorado in Mountain time. I think we might have had an engineer on from India and I’m pretty sure it was the middle of the day for him.
Anyway, I was with our client representative at our new facility in South Carolina. We were on a crazy rush schedule. Seems like we were always on a crazy rush schedule when we brought up a new center. But, this one was crazier than the last one. And the last one was crazier than the one before that.
It was a simple step. We needed the client to leak network routes to our center so we could bring up our network. You could also say they had to advertise the routes. Or even publish the routes. Yeah, it’s computer jargon. But, in this case, it’s a short cut word that takes a longer explanation.
Leaking, advertising or exposing a network route means that the client has to communicate with our routers and let them know that they can route data to the client’s network. This is an important step because our IP addresses are being managed by the client. And the tools our agents will use are sitting on the client’s network.
Once they leak the routes, we still have to advertise our own internal routes so that our agents can get to our tools, domain controllers, Active Directory servers and so on.
It’s probably just as technical as it sounds, but fortunately we had engineers on the call to make it all happen. Well, almost all happen.
As the call started off, it quickly became clear that we had a problem. Our engineers had a route set up to our Denver data center. That was all well and good, but the client engineer wanted to know why we didn’t also have a route set up to our Cleveland data center.
The Cleveland MPLS circuit isn’t scheduled to go in until next Thursday.
That’s a problem.
Why?
Because our corporate policy is that we cannot advertise routes to a single data center. If we cannot advertise to both data centers we are going to have to back out this change and reschedule.
You never told us we had to have both data centers connected.
Sorry, you should have known. That’s how we’ve always done it.
Harry Truman once said, “It’s amazing what you can accomplish if you don’t care who gets the credit.” The same cannot be said for deciding who gets the blame.
Our entire project schedule came to a screeching halt. Our training lauch and our production launch dates were both now in jeapordy. All because we didn’t have that route to Cleveland setup.
I was the lucky person who got to share the news with our vice president,
This is unacceptable, Rodney. Who screwed up?
Well, it wasn’t really anyone’s fault.
We’ve done three sites up until now and none of them had this issue. How can it not be someone’s fault?
I’m not opposed to accepting blame. If it helps my team, I’ll even take the blame myself. But, I was having a hard time assigning blame for this one.
If someone offers to give you a ride to work do you just assume they have a car? If they show up with a motorcycle, who’s fault is it that they don’t have room to take your presentation board? I had a difficult time blaming my network engineer. He had verified that the route to our Denver data center was setup and working.
But, I also had trouble blaming the client network engineer. Every time they set up a new center there are redundant routes. The project manager wouldn’t have known. Sure, I knew, but no one was asking me. And this is a site we set up a connection from a couple of years ago.
There was plenty of blame to go around, but I was having a tough time figuring out where to place it.
The good news is that we were all techy project manager, engineer types. We are good at crisis management. Once we figured out the problem, and realized we were not going to be successful, we immediately came up with a new deployment schedule. We have a change order to get the connection to Cleveland setup. We also have a new time set up to bring the network online. It cuts our window from 10 days to two, so we have much less margin of error to still make our dates.
I really hope it works because if we have another setback, I’m really not sure who to blame.
Rodney M Bliss is an author, columnist and IT Consultant. His blog updates every weekday. He lives in Pleasant Grove, UT with his lovely wife, thirteen children and grandchildren.
Follow him on
Twitter (@rodneymbliss)
Facebook (www.facebook.com/rbliss)
LinkedIn (www.LinkedIn.com/in/rbliss)
or email him at rbliss at msn dot com(c) 2019 Rodney M Bliss, all rights reserved