The Biggest Team I Never Led

“Hello? This is Paul in Accounting. Is this the checkin number?”

“Yeah, Paul. Thanks for getting up to help us. How do your servers look?”

“Everything I can see looks good. . .Do you need anything else from me?”

“Nope. If you don’t have any alerts, that’s all we need.”

It’s 3:00 AM on the third Saturday of November. I’m the Change Manager for data center maintenance for a large non profit. We have just finished 7 hours of maintenance work representing 18 major tasks and over 300 subtasks across multiple data centers. I’ve been planning this work for 6 months. The engineers tell me everything is done. We’ve completed our internal checks. And now we are waiting on our customers to call in and tell us if all of their systems are up and running.

“So, how many does that make?”

“Let me see. . .that’s 23 portfolios checked in so far.”

“How many total?”

“27 portfolios, but some have multiple people. If we get 30 out of our 35 I’ll be happy.”

“Yeah, hello? This is Frank in shipping.”

“Hello, Frank. Thanks for waking up to help out. How do your servers look?”

“Everything looks good except for L4250. I can ping it, but I can’t connect.”

“Hang on, Frank. We’ll get someone to check that and get right back to you. Stay on the line with us. Is Network Operation Center still on the line?”

“Yes, this is Jose.”

“Jose, would you have someone check L4250? And also, we haven’t heard from Daniel Carmichael in video productions. I know they have a broadcast tomorrow and I really want a status from them. Would you call their duty phone and wake them up?”

“Sure, we’ll get right on those.”

The maintenance goes from 8:00 pm to 3:00 am. The verification goes from 3:00 am to 6:00 am. In addition to the 35 customers who have signed up to verify their systems, and the NOC folks, I have a large team tonight of close to 50 engineers, vendors and managers. In addition there are program managers who provided the food and most importantly the billing codes. There is also the occasional senior manager, or HR representative who wants to come down to the data center on a Friday night to be part of our production.

As we moved through each phase of the maintenance over the past weeks; pre-planning, approvals, planning, communication, change control, and the actual work tonight, everyone has looked to me for direction. I sometimes feel like a director in a play. I don’t do any actual work on the system, but I coordinate everything.

The surprising thing is that none of the people involved in this process actually work for me. This group that I’ve affectionately named “Red Shirts,” in honor of Star Trek’s engineering teams, is made up of people from every department across our entire enterprise. Each team has willingly donated people to work on the monthly maintenance because I was able to convince them that by contributing some of their teams budget and resources would not only help the organization, but more importantly would see that their own projects got seamlessly integrated into our maintenance schedule.

All this planning, all these people, and Monday we start the entire process all over again for December.

I counted up the number of people involved on a busy month like November and it came to over 70 people. It is by far the biggest team I never led.