This past weekend, I spent time setting up four new laptops for one of my business clients. Out of habit, one of the things I do when I get to the point of setting up e-mail is send test messages. I didn’t have any reason to believe that e-mail wouldn’t work, but I’m glad I stuck to my habits.
Sunday night, I completed loading Office 2007 on the new laptops and configured Outlook to each of the new users’ hosted Exchange mailboxes. It was then I noted something strange was afoot. I could send from these newly minted mailboxes just fine, but inbound SMTP messages weren’t arriving (messages sent within the Exchange environment were fine). I thought maybe the SMTP server in the Outlook I was using to send test messages could be the cause. So I jumped onto webmail interfaces for a couple more of my other e-mail accounts and sent messages from them to these mailboxes. None of those test messages arrived.
I then ran a full DNS Report from DNSStuff.com and verified the MX records for the domains were OK. They were. Everything looked fine. I logged onto the control panel for this hosted Exchange account and temporarily disabled SPAM filtering, thinking that it was somehow flagging my test messages. I sent another round of test messages from multiple accounts and still nothing arrived. Then I started to think there was something bigger going on, so I sent test messages to an address of another business customer that I also have setup with the same hosted Exchange service (they were on another server with the Exchange Provider). That test message also failed to arrive.
Being Sunday night, I didn’t think I’d be able to catch anybody on the support line for the hosted Exchange provider (I’m usually not shy to name names, but I’m not going to in this case), but I called anyway. I was using their Small Business Hosted Exchange offering, rather than their Enterprise Hosted Exchange service, so I was not entitled to 24×7 support. Still, I was alarmed at the situation, so I logged into their support portal and logged a ticket. I also sent an urgent message to their support e-mail address informing them of the situation.
Much to my surprise, I got some live responses (you can typically tell when a person replies vs. a bot’s reply) in a little over an hour. Within two hours, I got another response that they had indeed discovered a widespread problem with incoming mail. Shortly after that one, another message arrived that they found the cause — an anti-virus database glitch that was identifying a large percentage of incoming SMTP e-mail traffic as dangerous and deleting it with no notification to sender or recipient. Yikes!
Thankfully, once they pinpointed the problem, it was fixed pretty quickly. After the “all clear” was sounded, I sent a new round of test messages which got through immediately. Still, I think both myself and the hosted Exchange provider were lucky I just happened to be testing new mailboxes. This could have easily gone unnoticed for a much longer period of time, causing major hassles for my customers. It was also fortunate that it was a Sunday evening that this happened. If it were in the middle of a workday, it could have been uglier.
I pressed the support reps at the Exchange provider to see if they had any clue as to the duration of this problem. They said they had no way to tell. I just wanted to know so I could inform my customers. By my estimates, the issue existed for several hours at minimum. On Monday, I drafted and sent an e-mail to key contacts at impacted customers to notify them of the issue. It was the right thing to do.
In the end, I still have faith in this Exchange provider. I’ve called on their support team several times in the past, and have almost always gotten very quick (and helpful) responses. Hopefully, it will be a long time before having to call on them again.