Who Let That Guy In Here?

Are you sure?

Yes, Rodney. Our reservation agents cannot get into the GES application.

But, that’s just one tool within the portal application, right?

Yes.

And all the other tools are working?

Seem to be

It was 3:00AM and I was overseeing maintenance some of the backend servers. Specifically we were upgrading the phone switch. We run two of them, a primary and a backup. We’d patched the backup with the latest release and we had just moved all traffic to the backup. (Called “failing over” even though it’s not a “fail.”) The problem was that when my agents logged back in they had an error in their GES application. The client owns GES. I honestly couldn’t see how switching my phone servers could break client tools.

But, we were going through our validation process and GES was broken. After the agents checked it over a second time, I got back on the phone with engineers.

Guys, I’m really sorry guys. I need you to switch us back to the other system.

You realize that system isn’t patched, right?

Yeah, that’s why we need to switch back. I don’t think we broke this, but we need to switch back and see if that fixes it.

Switching back didn’t fix it. But, after we switched one of my agents said, “Members mentioned they can’t access GES from the client’s website either.” At that point I knew two things: First, I didn’t break it and I needed to have my engineers switch us back to the patched system and start patching the other server, and second, I needed to call my contact at the client and inform him that he had a system outage.

Jared? Yeah, this is Rodney. Sorry to wake you up. No, no, our maintenance is going forward just fine. But, it appears that the GES application is broken. I’m positive it’s not a result of my maintenance.

Thanks, Rodney. I’ll check into it.

And about a half hour later he called me back.

Okay, the issue seems to be the GES team decided to do some unscheduled maintenance last night.

. . .ah. . .

Yeah, I know. I’m not happy about it.

Our maintenace had another couple of hours to run and we failed back and forth several times. The engineers did a great job and we didn’t generate any errors.

It always helps to know who else is playing in your sandbox.

Rodney M Bliss is an author, columnist and IT Consultant. His blog updates every weekday. He lives in Pleasant Grove, UT with his lovely wife, thirteen children and grandchildren. 

Follow him on
Twitter (@rodneymbliss) 
Facebook (www.facebook.com/rbliss) 
LinkedIn (www.LinkedIn.com/in/rbliss)
or email him at rbliss at msn dot com

(c) 2017 Rodney M Bliss, all rights reserved