I Survived Bedlam3

(Photo Credit: baxiabhishek via flickflu.com)

Is anyone else having a slow email problem?

Seriously, Rodney? You’re kidding right?

No. I sent Bill an email 45 minutes ago and it’s still not there. What’s the deal?

Last week I talked about the Message Recall feature as the Ultimate Save Your Job Feature, And Why It Took Microsoft So Long To Implement It. In addition to the ability to recall a potentially career ending message, the other benefit of Message Recall is to prevent Reply All storms. If you accidentally send to the wrong distribution list, use Message Recall to get the message back. . .before it’s too late!

When Exchange was first released, it had the ability like all email systems to send email to more than one person at a time. What it lacked was the ability to restrict the ability to send to more than one person.

Why would anyone want to restrict the ability to send to multiple people?

Bedlam3
Technically it was called Bedlam DL3.

I was working for Microsoft, and Exchange 5.0 had just been rolled out internally. Employees were getting familiar with the new features. I was writing training courses about Exchange at the time, so I was also diving in and learning all the ins and outs of Exchange Server and Outlook.

On the morning of October 14th, 1997 a Microsoft employee looked at their entry in the company Global Address List. They noticed that in addition to distribution lists for their team, their department and some social email lists, they were on a distribution list called “Bedlam DL3.” The employee was as mystified to the purpose of Bedlam DL3, or Bedlam3 as it became known internally as you would be.

It’s a catchy name though, don’t you think? It sparked the curiosity of this particular employee. They decided that they were not comfortable being on someone else distribution list. And they made a fatal mistake. If you have a question about a distribution list, the best thing to do is look up the owner and email that person and ask. Or, you could email your IT rep and ask her. But, the absolute worst possible action is to email the distribution list. That’s the course of action the employee chose.

To: Bedlam DL3
From:
Subject: Why am I on this mailing list? Please remove me from it.

That doesn’t sound so bad does it? It wasn’t just bad, it was awful. Everyone on Bedlam3 got a copy of the “Please remove me” email. Some of them hit Reply All.

Yeah, take me off too.

And that message went to everyone. And soon others were joining in.

Me too!

This is what’s known as a Reply All Storm. With Exchange 5.0 there was no way to stop it once it started. Of course, the easiest way for it to stop it is if everyone will stop Replying All to the thread. So, several people started saying that.

Stop using REPLY ALL. You're just making it worse.

And of course, these messages also went to everyone.

I’ve seen Reply All storms that filled your mailbox with dozens of Reply All messages. Bedlam3 quickly outstripped all of those. It generated a storm of epic proportions. See, Bedlam DL3 was one of four Bedlam distribution lists. Each Bedlam DL included the names of one quarter of all Microsoft employees. The Bedlam DLs were never intended to be used for email. They were set up by the IT department to map employees to Windows Server security groups, and as general catch-all groups. Bedlam3 had 13,000 names on it.

The storm quickly flooded the network and completely shut down the mail servers. Before it was done, it’s estimated that over 15 million emails were generated in the space of about an hour. The storm pushed 195 GB of data around the network during that hour. Most of it was messages saying “Me too” and “Stop using Reply All. I wasn’t one of the 13,000 people on Bedlam3. So, I didn’t see my mailbox fill up with Replay All messages, but since I had to use the same email servers as everyone else, none of my mail was being routed either.

It took 2 days to clean up the mess. And, of course, this being Microsoft they had t-shirts made. The front said “I survived Bedlam3.” The back said “Me too!”

Microsoft has implemented many enhancements to ensure this never happens again.

– Hidden distribution lists. Users no longer know they are on a “Bedlam DL3” list
– Maximum number of senders. Users can be restricted to a maximum number of people on a single message
– Message Transfer Agent rules. The IT guys can kill a storm once it starts.
– A bunch of other enhancements that prompt you if you are sending to a large group

Microsoft, like many tech companies is filled with super smart programmers, testers, program managers, etc. The employees should have known better. Some of them (almost) knew better since they told people to stop using Reply All.

Even in a smart company, people can make silly mistakes.

The most ironic thing from that day so many years ago was the fact the name of the distribution list. Maybe they were simply tempting fate too much naming a distribution list Bedlam.

You can read more about the Bedlam DL3 episode here.

Rodney M Bliss is an author, columnist and IT Consultant. He lives in Pleasant Grove, UT with his lovely wife and thirteen children.

Follow him on
Twitter (@rodneymbliss)
Facebook (www.facebook.com/rbliss)
LinkedIn (www.LinkedIn.com/in/rbliss)
or contact him at (rbliss at msn dot com)