Skip to content

I Survived Bedlam3

October 17, 2013

(Photo Credit: baxiabhishek via flickflu.com)

Is anyone else having a slow email problem?

Seriously, Rodney? You’re kidding right?

No. I sent Bill an email 45 minutes ago and it’s still not there. What’s the deal?

Last week I talked about the Message Recall feature as the Ultimate Save Your Job Feature, And Why It Took Microsoft So Long To Implement It. In addition to the ability to recall a potentially career ending message, the other benefit of Message Recall is to prevent Reply All storms. If you accidentally send to the wrong distribution list, use Message Recall to get the message back. . .before it’s too late!

When Exchange was first released, it had the ability like all email systems to send email to more than one person at a time. What it lacked was the ability to restrict the ability to send to more than one person.

Why would anyone want to restrict the ability to send to multiple people?

Bedlam3
Technically it was called Bedlam DL3.

I was working for Microsoft, and Exchange 5.0 had just been rolled out internally. Employees were getting familiar with the new features. I was writing training courses about Exchange at the time, so I was also diving in and learning all the ins and outs of Exchange Server and Outlook.

On the morning of October 14th, 1997 a Microsoft employee looked at their entry in the company Global Address List. They noticed that in addition to distribution lists for their team, their department and some social email lists, they were on a distribution list called “Bedlam DL3.” The employee was as mystified to the purpose of Bedlam DL3, or Bedlam3 as it became known internally as you would be.

It’s a catchy name though, don’t you think? It sparked the curiosity of this particular employee. They decided that they were not comfortable being on someone else distribution list. And they made a fatal mistake. If you have a question about a distribution list, the best thing to do is look up the owner and email that person and ask. Or, you could email your IT rep and ask her. But, the absolute worst possible action is to email the distribution list. That’s the course of action the employee chose.

To: Bedlam DL3
From:
Subject: Why am I on this mailing list? Please remove me from it.

That doesn’t sound so bad does it? It wasn’t just bad, it was awful. Everyone on Bedlam3 got a copy of the “Please remove me” email. Some of them hit Reply All.

Yeah, take me off too.

And that message went to everyone. And soon others were joining in.

Me too!

This is what’s known as a Reply All Storm. With Exchange 5.0 there was no way to stop it once it started. Of course, the easiest way for it to stop it is if everyone will stop Replying All to the thread. So, several people started saying that.

Stop using REPLY ALL. You're just making it worse.

And of course, these messages also went to everyone.

I’ve seen Reply All storms that filled your mailbox with dozens of Reply All messages. Bedlam3 quickly outstripped all of those. It generated a storm of epic proportions. See, Bedlam DL3 was one of four Bedlam distribution lists. Each Bedlam DL included the names of one quarter of all Microsoft employees. The Bedlam DLs were never intended to be used for email. They were set up by the IT department to map employees to Windows Server security groups, and as general catch-all groups. Bedlam3 had 13,000 names on it.

The storm quickly flooded the network and completely shut down the mail servers. Before it was done, it’s estimated that over 15 million emails were generated in the space of about an hour. The storm pushed 195 GB of data around the network during that hour. Most of it was messages saying “Me too” and “Stop using Reply All. I wasn’t one of the 13,000 people on Bedlam3. So, I didn’t see my mailbox fill up with Replay All messages, but since I had to use the same email servers as everyone else, none of my mail was being routed either.

It took 2 days to clean up the mess. And, of course, this being Microsoft they had t-shirts made. The front said “I survived Bedlam3.” The back said “Me too!”

Microsoft has implemented many enhancements to ensure this never happens again.

– Hidden distribution lists. Users no longer know they are on a “Bedlam DL3” list
– Maximum number of senders. Users can be restricted to a maximum number of people on a single message
– Message Transfer Agent rules. The IT guys can kill a storm once it starts.
– A bunch of other enhancements that prompt you if you are sending to a large group

Microsoft, like many tech companies is filled with super smart programmers, testers, program managers, etc. The employees should have known better. Some of them (almost) knew better since they told people to stop using Reply All.

Even in a smart company, people can make silly mistakes.

The most ironic thing from that day so many years ago was the fact the name of the distribution list. Maybe they were simply tempting fate too much naming a distribution list Bedlam.

You can read more about the Bedlam DL3 episode here.

Rodney M Bliss is an author, columnist and IT Consultant. He lives in Pleasant Grove, UT with his lovely wife and thirteen children.

Follow him on
Twitter (@rodneymbliss)
Facebook (www.facebook.com/rbliss)
LinkedIn (www.LinkedIn.com/in/rbliss)
or contact him at (rbliss at msn dot com)

Advertisements
3 Comments
  1. This lives on and on and on. I do a search every now and again to see if it still lives, and it does.

    I’m that fateful employee…I sent that email…and I didn’t make a mistake…it was intentional. Here’s why.

    Background: I was a tester on the Picture It!/PhotoDraw team. At that point, I was a 4 year veteran…but hadn’t done the “oops, just sent to a large distribution list” email. I was savvier than that…or so I thought.

    I had a routine of checking to see what aliases/DLs I belonged to…cleaning up old ones, etc. When I saw Bedlam DL3, I was perplexed…I hadn’t subscribed to it…asked a few of my team if they were familiar…no dice with anyone.

    1) I was running Windows NT at the time, that was the only reason I could even open up the DL and see owner/members (yes, I did look).
    2) There was no owner listed (mostly because I wasn’t supposed to even see it or its membership).
    3) There were no members listed in the DL (again, hidden, because Exchange/Outlook couldn’t enumerate the users).

    So, with a low risk, empty DL, I sent an email to it, hoping the owner (that I couldn’t see) could enlighten me. Sadly, as we all found out (and I spent that entire day with a growing sense of dread), it had a LOT of people on it. I thought…I’m fired….they’re going to fire me….I’ve crippled the entire email system at the biggest software company on Earth. Well…that didn’t happen…and I lived on.

    I learned later on that if I had been on Windows 95, trying to look at the DL would have crashed Outlook. That would have been a different email to a different audience (IT). I actually put the Bedlam DL3 event in my review that year…Stress tested the Exchange email system.

    • Brian, that is one of the coolest stories I’ve ever heard. It’s like your the original typhoid Mary. (But, not nearly as deadly.)

      I’d like to write a follow up blog that included the information you’ve put here. If you’ll send me an email (rbliss at msn dot com) I’ll be happy to send you a draft before it posts.

      After all this time, it would be great to have at least one entry that sets the story straight.

      If for some reason you object, let me know, but I think it would make a cool entry.

Trackbacks & Pingbacks

  1. Best of 2015 #5: A Good Muse is Hard To Find | Rodney M Bliss

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: