Mail problems and mail problems fixed..

About an hour and a half ago mail started failing for pretty much ALL customers..

After some research, it seemed like dns was the problem. However, things seemed fine on the dns servers themselves. Something was wonky with the network it looked like.

Poking around for a while finally revealed the problem, the load balancer our mail machines were behind was out of (65,000) ports! Everytime our mail servers send mail they have to do a dns lookup, and to do so they needed to go out through the load balancer to our dns servers, which would then use up a random port waiting for the return packets. Now that we’ve got nearly all our mail servers behind the load balancers though, it was apparently using up all the ports concurrently, causing time-outs and failures.

To fix this, we set up another dns server just for mail and put it behind the load balancer! Things are working again now and this problem shouldn’t rear its ugly head anymore.

Our apologies for this.. hopefully this is close to the last wrinkle to iron out with our new load balanced email setup.