DreamHost Status » Mysql Outages

Exodus is down for the count.

Sandon — Sat, 06 Jun 2009 07:57:07 +0000

It looks like one of our web-server ‘exodus’ was unresponsive and had not came back up after we attempted to power-cycle it remotely. Right now an admin is en-route to the data center to check out the situation.

Please check here for updates on this server’s status.

Update: 02:19 AM PST. it looks like the problem was with the raid controllers BBU that was preventing it from booting. It has been replaced and should be back up shortly.

Update 02:30 AM PST. Ok the server and all its services should be restored.

riffraff MySQL server problems

danwy — Mon, 18 May 2009 20:57:28 +0000

We’re currently having some hardware issues with the riffraff MySQL server. One of our admins is currently heading down to the datacenter to recover it. We don’t have an ETA on when this will be resolved yet, but we’ll update this post as soon as we have more information.

Update - 3:57pm PST: This server has been back up and running for the last 30 minutes or so and we’ve verified that things are now running smoothly. If you’re still having problems it was not likely related to this server and you should contact support to investigate it further. We’re sorry for any inconvenience this caused you.

MySQL server maintenance April 17th@2am Pacific - Done

Ken — Wed, 15 Apr 2009 10:54:23 +0000

The following MySQL servers will be down for maintenance at 1am April 16th Pacific:

flipper, fin, nobnin, riffraff, and morris

The servers should only be down for about 10 minutes up to a maximum of 30 minutes.

Status will be updated when we start and when the services are back up.

UPDATE: Due to another issue, the mysql maintenance will be deferred until further notice. Currently, the Mysql servers that this affects are still up and fully operational.

UPDATE: Maintenance has been rescheduled to April 17th at 2am Pacific.

UPDATE: Starting maintenance. Should be finished by 2:30am Pacific. mysql machines will be down until then.

UPDATE: Maintenance is done. All mysql services have been restored on these machines.

Missy MySQL Server Downtime

Justin — Mon, 06 Apr 2009 18:18:21 +0000

The missy MySQL server is having a hardware issue, one of our admins is on his way down to the data center now to diagnose the problem. We are working to get this resolved as soon as possible. When we have further information to share we will update this post. The missy mysql server is used by “spacey” cluster customers only, no other clusters are currently effected.

– UPDATE –

The admin has diagnosed and replaced the hardcare causing the problems, and the missy MySQL server is working normally again. If you do continue to have any issues related specifically to this server please contact support and provide details about the problem. Again our apologies for the downtime.

Network issues

Terri — Sat, 04 Apr 2009 01:49:03 +0000

We are currently experiencing networking issues. This is affecting some machines, as well as mysql services and oneclick installs. We are troubleshooting the issue and will fix it as soon as possible. We apologise for the inconvenience.

Update: Thankfully the network issue was resolved in a short period of time. There was a misconfiguration in our core router, and our network administrators patched it up.

Issues with our Homie cluster

danwy — Sun, 15 Mar 2009 03:54:33 +0000

We are currently experiencing issues with one of our server clusters, homie. If your site is on this cluster and it is currently down, this is the reason why. Our admin team is aware of this and is on the phone with our datacenter right now trying to find the cause. We don’t have any additional information at this time, but we will update this post as soon as we have more specific details. We apologize for any inconvenience this causes you and we hope to have everything back up shortly.

Update: Sorry we didn’t include this earlier. Here’s a list of servers that may be experiencing issues:

barker bonanno castellano castiglia genovese mangano mcgurn rothstein scalise schultz spillane spilotro

Update: This issue was caused by a power strip failure in one of our server racks. We have the datacenter engineers on their way to the datacenter to resolve the issue. We don’t have an ETA on when things will be fixed, but we are doing all that we can to remedy this issue as soon as we possibly can.

Update: For the most part, this issue has now been resolved. The datacenter engineers were quick to respond and resolve the problem with the circuit breaker. There was a small handful of servers that weren’t booting up correctly, so we had to go through and manually reboot them. We have most of the services back up and running, but there are still a few PS servers that we’re still working on. We should have those all back up and running shortly. We’ll post one final update once we’re confident everything is resolved.

Update (1:39am): Alright, whew. Everything should be fine now. What actually happened was a circuit in the datacenter blew and it just so happened that the switches that your machines are connected to were out because of that. It took some time to get a tech from the datacenter in there to reset the breaker and then after that I had to go and manually check each machine. The DreamhostPS and PS Mysql customers had some networking issues that also slowed things down. I apologize for the downtime! If you’re still having trouble, please send a message to our support staff.

Pan fried

Leon — Mon, 09 Feb 2009 23:48:30 +0000

Looks like webserver pan has blown a gasket and needs to be moved to new hardware. I’m working on this now and it should be done in about an hour.

Update 4:43pm: All set! If you’re still having a problem with pan, please contact a dreamhost support robot!

some MySQL PS problems

danwy — Thu, 30 Oct 2008 17:02:42 +0000

One of our MySQL PS host machines is acting up right now. One of our admins is currently investigating the problem, but doesn’t have any updates at this point. If you’re on one of the following MySQL PS servers and are experiencing problems, we are investigating the issues and hope to have it resolved as soon as possible:

psmysql3163 psmysql3843 psmysql3851 psmysql3870 psmysql3882 psmysql3888 psmysql3893 psmysql3897 psmysql3914 psmysql3939 psmysql3976 psmysql4021 psmysql4044 psmysql4062 psmysql4068 psmysql4077 psmysql4137 psmysql4139 psmysql4145 psmysql4150 psmysql4155 psmysql4175 psmysql4183 psmysql4238 psmysql4250 psmysql4255 psmysql4260 psmysql4264 psmysql4276 psmysql4281 psmysql4292 psmysql4298 psmysql4310 psmysql4311 psmysql4314 psmysql4315 psmysql4410 psmysql4415 psmysql4416 psmysql4592 psmysql4613 psmysql4634 psmysql4645 psmysql4659 psmysql4712 psmysql4714 psmysql4715 psmysql4765 psmysql4772 psmysql4878 psmysql4890 psmysql4932 psmysql4934 psmysql4936 psmysql4939 psmysql4945 psmysql4992 psmysql5000 psmysql5019

Please check back on this post, as we will be making updates to it as we troubleshoot the problems.

UPDATE 10:30am PST: We’ve tracked this problem down to a bad RAID card on the MySQL PS host. We’re moving the disks over to new hardware now and should have it fixed up soon. We’ll update this post again once everything is resolved.
UPDATE 12:20pm PST: We have swapped the drives to a new chassis and after a bit of reconfiguring the guests are now back up and running. If you continue to have issues with your mysql private server please contact support.

Issues with accessing databases in phpmyadmin

andrea — Thu, 16 Oct 2008 16:24:23 +0000

Some customers on certain database machines have been having issues accessing their databases using phpmyadmin. We apologize for the inconvenience this is causing. Our admin team is looking into the cause and is working on a resolution. The symptoms of the problem are one of the following: blank page loading instead of the phpmyadmin interface, some interfaces are missing images and/or the dbs are not selectable in the drop down list. For the latter one we are suggesting a workaround for now: work in phpmyadmin without using the left frame, browse to ‘Databases’ on the right, selecting th one you want.

Command line database access remains unaffected. As a reminder, here is the generic command:
mysql -u username -ppassword -h hostname databasename

Please check back here for updates, and sorry again for the problems.

MySQL Service Downtime

Justin — Thu, 02 Oct 2008 13:35:28 +0000

We are experiencing some mysql service downtime at the moment. We are still investigating the cause and working to get this fixed up as soon as possible. It would seem a system-wide configure of the mysql services is what triggered this, with new configuration files not being pushed to the new servers. We do have our admins on the job and will do everything we can to get service restored. Sorry for the problems. When we have further info we will update this status post.

Update: I apologize for the lack of a followup but we actually resolved this Thursday morning a short time after the initial posting of this status report (the admin who did was unable to post at that time).