Anatomy Of A Technical Error

3:30 AM – Jamie, how does our file look this morning?

This is a message I’ve sent off every morning for the past week. And yes, I actually get up at 3:30 AM to send it. Bleary-eyed I stared at my monitor and considered how I managed my project so poorly that I’m stuck getting up in the middle of the night to babysit a file upload process. The life of a Project Manager is just full of excitement.

Last week, I talked about the challenges we had getting a file uploaded to our customer’s FTP site. (Not My First Rodeo.) I thought I’d walk you through how that process came about. Sometimes the simplest things turn out to be the hardest.

I’ve known for several weeks that one of my deliverables (yes, that is actually a term we use in Project Management) was to send my client a list of licensed agents every day. This wasn’t a noteworthy task. In fact it was one of about two dozen that I had to complete to be able to successfully launch our new site on July 15th. Getting an automated process set up to send this type of file isn’t hard. The client sent us the information about what the file was supposed to look like, the format. I passed that on to our database team and didn’t think about it too much.

Around July 1st, we started looking at testing our file. This is where the first red flag showed up. I didn’t realize it was a red flag at the time. In fact, it sort of looked like a pale yellow flag.

Rodney, we didn’t get an actual schema file from the client.

Okay. . .

That means I don’t know exactly how they want the file to look. I’m just going to make my best guess.

That’s fine, we’ll figure it out during testing.

I didn’t realize what a mess I had just walked into. Not only walked into it, but I brought my database team along with me. We started our testing and the first step was to upload our test file to our clients FTP site. FTP stands for File Transfer Protocol. It’s a special web site that acts like a file folder. We had to put our files onto the client FTP site. We immediately ran into issues. Just like you have multiple folders on your hard drive, or in your email, an FTP site can have multiple locations. And you don’t want to store a document in your My Music folder. We worked through which folder to store our file in. It was a process that took several days before we got it right.

Rodney, it looks like we have successful received your file.

So, is there anything else we need to do?

Nope. You are golden.

Falser words were never spoken, but none of us realized it at the time. After the testing, we had to start sending real data, the actual names of our agents, three days before we went live. So, we needed to start sending real data to the client no later than the 10th. Oh, and these files had to arrive by 5:00 AM Central Time. The day before our deadline we uploaded the file at 4:30 AM just like we had done during our testing, except this time it had real customer data. I wasn’t awake for it. I hadn’t yet started my babysitting duties.

Rodney, your file failed.

What was wrong with it?

It’s all about spelling. . .check your spelling.

Huh?

That was the actual feedback I got. We were about to miss a key milestone date. Project Managers absolutely HATE missing dates. We’ll do almost anything to avoid being late. I huddled with my team early Wednesday morning. We looked at our test files that had blessed as “golden.” We compared them to our production files. They appeared exactly the same. Here’s a hint, although we didn’t know it at the time. One of our files was named:

Daily_Licence_File_.xml

You might think that file has a misspelling. Well, if you live in the US you might think that. Readers in the UK, or India (where the developers who created the file worked) know that there are actually two spellings of what we in America refer to as license.

License – A verb meaning to grant someone a certification allowing them to do something
Licence – A noun. The actual certification that the person received

Because we failed Wednesday morning, I was out of time. My next file upload was Thursday and that one had to be right. What does “had to” mean? If I missed that date, then we couldn’t launch our site. Meaning that everyone involved in the months long project would look at me and ask why I screwed up. More importantly there was a $50,000 per day fine for everyday that we were late launching our new site. I HAD to make the Thursday file work.

Yeah, Jamie since we failed this morning, I’d like to schedule some time with your tech teams today. I want to upload a copy of our file every two hours. You tell us if we’ve fixed all the problems.

I’m sorry Rodney, we really don’t have that kind of time during work hours to devote to this.

Okay, how about a meeting this evening around 8:00 PM?

No, we’ll be in our maintenance tasks by then. Just fix the issues and upload the file tomorrow like normal.

Right.

The team scrutinized the file. We read through all the documentation we’d received. We looked at the sample file formats the client had sent us. And I set my alarm for 3:30 AM Mountain Time. (Actually, I didn’t set my alarm. From the time I was a kid I’ve been able to wake up at a specific time. So, I set mental alarm clock to 3:30.)

Whoever came up with the phrase “bright and early”? Not someone who had to get up at 3:30 AM and try to find their way around their house in the dark so as not to wake up my family. I finally got to my office, closed the door, turned on the light. Turned OFF the light. Ow, that light was bright.

How’s our file look this morning?

It failed again. Rally your team. We are now in crisis mode.

I started waking people up. Fortunately half the team is in India where it was 3:00 pm in the afternoon, but the Utah team was asleep.

We spent the rest of the day on it. And then the next day. And then a good portion of the weekend. The problems we eventually discovered:

– The file had the wrong name. We wanted the American spelling, not the British
– The file was encoded wrong. We were using Unicode(16bit) and they wanted UTF-8. (This means that we were sending an extra space character after each of our characters
– We had the file capitalization wrong because our client was using Unix. On Windows, you can name a file using any combination of uPpEr or LoWeR case, and the receiving system just figures it out. On a Unix system, you must match the filename exactly, including capitalization. It’s like your password in that sense.

So, what about all that testing we had done early on? The “golden” test? Turns out we were only testing the ability to upload a file, not the filename or any of the content. If you’ve read this far, and if you’ve paid attention, you have to be wondering about that $50K/day fine. Was Rodney going to get socked with that?

The client decided that they really, really wanted us to open our new site on time. We sent them the names that we weren’t able to upload in the file and they entered them manually. Finally, Tuesday morning, July 15th we successfully uploaded a file. The team was hugely relieved. It shouldn’t have been this tough.

Turns out we lowered our guard too soon.

I was on site at our new location (As I described in When Cops Interrupt Your Conference Call.) The client came in Wednesday morning. And she didn’t look happy.

What’s wrong?

Can you guys really not upload a file successfully two days in a row?

You’re kidding.

Nope. You’re trying my patience here Rodney. This shouldn’t be that hard.

Okay, I guarantee the file will be there tomorrow and it will be correct. I’ll see to it personally. If it’s not there you can yell at me.

Oh, I’m going to yell!

She was smiling when she said it, but she was also serious. So, for the past week, I’ve gotten up every morning and checked in with the India team.

Did the file upload successfully?

Yep. Looks good.

I’ll let you know if the client has any issues with it.

Today marks a full week of uploading our file. And tomorrow I’m going to sleep in. . .until at least 7:00am!

Rodney M Bliss is an author, columnist and IT Consultant. He lives in Pleasant Grove, UT with his lovely wife and thirteen children and one grandchild.

Follow him on
Twitter (@rodneymbliss)
Facebook (www.facebook.com/rbliss)
LinkedIn (www.LinkedIn.com/in/rbliss)
or email him at rbliss at msn dot com