LAX data center connection issues
We are currently experiencing connectivity issues to our LAX data center.
All servers in the following clusters would be involved:
Blingy, Frisky, Looney, Spacey, Swarthy
Updates to come.
Update: All Servers seem to actually be up, but there is heavy packet loss at the distribution switch making them seem down to the outside.
Update: Packet loss has stopped. Our admins do have significant network upgrades planned shortly as a long term fix connectivity issues like this. If you have any further downtime , let support know via our panel form.
Update: Spoke too soon, Packet Loss is back . Our admin team is working feverishly on this.
Update - Wed Jan 23 21:34:52 PST 2008 - After a reload the switch is working properly again. We are already in the process of a major upgrade to this particular part of the network to resolve this issue. The actual swap-out of equipment will happen as soon as possible. Until that time we will be taking steps to keep the current switch operating properly.
.
January 23rd, 2008 at 8:09 pm
Popinski is down
January 23rd, 2008 at 8:10 pm
Zelda is also down. No response.
January 23rd, 2008 at 8:11 pm
Medium. Medium? I am unable to connect to the mail server. I’d call it pretty severe.
January 23rd, 2008 at 8:11 pm
pierce is down.
January 23rd, 2008 at 8:11 pm
renault is down . No response
January 23rd, 2008 at 8:12 pm
Cayman is also slow to respond on HTTP. Ssh connections time out.
January 23rd, 2008 at 8:12 pm
I think Brazil is also effected. Can’t get at websites there.
January 23rd, 2008 at 8:13 pm
I can’t connect to my site at all. I’d be happy if it were just the mail server that I couldn’t connect to
January 23rd, 2008 at 8:13 pm
Did some Juniper layer 2 gear fail you?
January 23rd, 2008 at 8:13 pm
Hug is also down! I’d say totaly severe. Can’t connect to mail either.
January 23rd, 2008 at 8:13 pm
prowler is down.
January 23rd, 2008 at 8:14 pm
impala seems to be down also. No Response.
January 23rd, 2008 at 8:14 pm
Spyro is so slow a simple webpage can not load in several minutes. My sites on Kroner are fine, so I am puzzled.
January 23rd, 2008 at 8:14 pm
(Ping to brazil times out too)
January 23rd, 2008 at 8:14 pm
Another day, another massive failure by Dreamhost…
January 23rd, 2008 at 8:14 pm
Things look really good right up until the border router in LAX where tails sits, missing 73% of its packets. I am looking forward to a stable network
-Eric
Host Loss% Last Avg Best Wrst StDev
1. 172.16.36.1 0.0% 86.3 108.3 85.9 463.2 73.3
2. gw.newdig.com 0.0% 86.6 108.5 86.1 468.6 74.1
3. 206-169-234-53.static.twtelecom.net 0.5% 86.8 118.2 86.0 550.7 79.9
4. peer-02-so-0-0-0-0.palo.twtelecom.net 0.0% 102.3 124.7 101.2 484.9 74.0
5. ae-1.r20.snjsca04.us.bb.gin.ntt.net 6.5% 103.5 126.2 102.8 488.1 73.8
6. ae-0.r21.snjsca04.us.bb.gin.ntt.net 0.5% 104.2 125.6 102.7 482.9 72.3
7. as-0.r20.lsanca03.us.bb.gin.ntt.net 0.5% 112.2 133.5 110.8 471.7 72.8
8. po-1.r00.lsanca03.us.bb.gin.ntt.net 6.5% 112.2 145.2 110.7 457.8 75.6
9. po-1.r00.lsanca17.us.bb.gin.ntt.net 0.0% 111.6 145.4 110.4 483.4 77.4
10. xe-1-3.r00.lsanca17.us.ce.gin.ntt.net 0.5% 112.9 144.1 112.1 487.0 76.0
11. border1.po1-bbnet1.ext1a.lax.pnap.net 1.1% 111.2 139.0 110.6 481.7 73.6
12. newdream-1.border1.ext1a.lax.pnap.net 0.5% 113.9 162.5 112.6 507.0 89.9
13. ip-66-33-201-69.dreamhost.com 0.0% 139.0 160.4 120.9 540.6 72.9
14. apache2-udder.tails.dreamhost.com 73.8% 142.8 161.0 127.0 528.2 71.0
January 23rd, 2008 at 8:14 pm
ditto dimarco….
January 23rd, 2008 at 8:14 pm
Wow, although it sucks that we have an outage, they are quick. I just sent a support email a few seconds ago and before that nothing was listed on this blog. I agree with the others that anytime you have zero connection, it shouldn’t be medium but rather severe and in the red!
January 23rd, 2008 at 8:15 pm
Just to make it complete, Swarthy is down. No FTP or HTTP access
January 23rd, 2008 at 8:16 pm
can’t connect to database on “rover” server.
January 23rd, 2008 at 8:16 pm
my Site is up, but my SQL server is down. Arrgghhhh!!!!!
January 23rd, 2008 at 8:16 pm
luigi is down too
January 23rd, 2008 at 8:17 pm
Glassjoe is down, no ping, no mail, no ftp, nada.
The funny part is they resolved my ticket and said “No server-wide issues found” and then I come back to the status page and voila, there IS a problem. The right hand doesn’t know what the left hand is doing eh?
January 23rd, 2008 at 8:17 pm
hope after they fix this the horrible lag will go away as well. the hosting for a while was superb I was having no lag issues at all. but things have gone down hill recently hope its not a constant for the rest of this year.
January 23rd, 2008 at 8:18 pm
titan and esprit are down.
why do I give you my business again???
January 23rd, 2008 at 8:18 pm
No mail or blogging for the wicked. I think I’m on Swarthy.
January 23rd, 2008 at 8:18 pm
low = single server, medium = cluster, high = everything is wtfpwn’d
I guess I’m lucky because this is the first time my site has been down since I got hosting here a year ago. Well… first time it was down & it wasn’t because I screwed something up.
January 23rd, 2008 at 8:18 pm
We’ve lost all of our ability to retrieve e-mail here on the east coast in Virginia via Nehi @ Dreamhost. Has been down for roughly 30 minutes. Website okay.
January 23rd, 2008 at 8:19 pm
vidar is not responding… nor my emal. This sucks.
January 23rd, 2008 at 8:19 pm
Some traffic to my SQL Server seems to be getting through…
January 23rd, 2008 at 8:19 pm
Japan is down ! I need my website up and running !!!can’t read my mails, I hope they fix it today
January 23rd, 2008 at 8:19 pm
Wrangler is also intermittent.
It’s dropping packets, but sometimes responds to pings. I’m not irritated though, these things happen, I’m just relieved to know you guys are aware of it already.
-Kirk
January 23rd, 2008 at 8:20 pm
what ISP hosts dreamhoststatus.com
they have demonstrably good up time as I am here all the time and have a chance to observe ITS reliability.
January 23rd, 2008 at 8:20 pm
if my mail/sql/http down are medium, i’d hate to see severe.
January 23rd, 2008 at 8:20 pm
Zelda appears to be back up now.
January 23rd, 2008 at 8:20 pm
I am so sick and tired of these outages! If it’s not an outage, we’re getting charged twice! I’ve had enough, Dreamhost!!!
January 23rd, 2008 at 8:20 pm
Simple question - how do I know what server I am on?
This is just for curiosity - whichever one it is, is down.
Mike
January 23rd, 2008 at 8:20 pm
Hornet is down as well.
January 23rd, 2008 at 8:20 pm
vega is down also
January 23rd, 2008 at 8:20 pm
Seems to be working now for me.
January 23rd, 2008 at 8:21 pm
We’re back up here on Shadrach.
January 23rd, 2008 at 8:21 pm
zebes is 90% down.
grunt mySQL server is 100% down.
January 23rd, 2008 at 8:21 pm
Hope connectivity is restored soon, Wonder what could have failed. My home network seems to be more reliable lol.
January 23rd, 2008 at 8:21 pm
Swarthy seems back up. I can get mail again.
January 23rd, 2008 at 8:21 pm
strange is down as well.
was wondering why my stats sucked today
and yeah when your sites don’t work, it’s not medium..it’s HIGH!
January 23rd, 2008 at 8:22 pm
thank God my email is using Google apps as dreamhost demonstrated it can not be depended on.
January 23rd, 2008 at 8:22 pm
Are we back up? My network link looks good and ssh let me in!
Host Loss% Last Avg Best Wrst StDev
1. 172.16.36.1 0.0% 87.0 87.0 85.9 90.5 1.1
2. gw.newdig.com 0.0% 86.3 87.0 86.3 88.7 0.6
3. 206-169-234-53.static.twtelecom.net 0.0% 86.9 100.9 86.3 300.8 43.5
4. peer-02-so-0-0-0-0.palo.twtelecom.net 0.0% 102.3 102.4 101.6 106.6 0.8
5. ae-1.r20.snjsca04.us.bb.gin.ntt.net 5.6% 105.9 104.0 102.9 106.9 0.8
6. ae-0.r21.snjsca04.us.bb.gin.ntt.net 0.0% 103.4 104.5 103.0 119.2 3.1
7. as-0.r20.lsanca03.us.bb.gin.ntt.net 0.0% 111.6 111.8 110.8 116.6 1.2
8. po-1.r00.lsanca03.us.bb.gin.ntt.net 8.6% 144.3 125.6 110.9 246.4 32.8
9. po-1.r00.lsanca17.us.bb.gin.ntt.net 0.0% 117.9 133.6 110.8 292.6 53.2
10. xe-1-3.r00.lsanca17.us.ce.gin.ntt.net 0.0% 113.4 113.2 112.1 120.2 1.4
11. border1.po1-bbnet1.ext1a.lax.pnap.net 0.0% 113.4 115.3 111.1 149.1 10.2
12. newdream-1.border1.ext1a.lax.pnap.net 0.0% 114.1 142.6 112.8 320.4 55.2
13. ip-66-33-201-69.dreamhost.com 0.0% 118.0 116.5 113.9 131.1 3.8
14. apache2-udder.tails.dreamhost.com 0.0% 121.1 115.4 113.5 121.1 2.1
January 23rd, 2008 at 8:22 pm
prowler is up - nice!
January 23rd, 2008 at 8:22 pm
Jax is so slow it might as well be down. I’ve only been with this host 3 weeks and there have been far more “hiccups” than I ever would have dreamed.
January 23rd, 2008 at 8:23 pm
titan and esprit are back up. I guess they plugged the cable back in.