DreamHost Status » System Changes

Spunky and Central Move

Terri — Sat, 22 Mar 2008 03:50:10 +0000

We’re about to begin to move the spunky cluster and our central machines. We’ll be taking down servers soon, but our status site, of course, will remain up and running.

Again, we’ll try to keep this move quick and painless as we possibly can, and we apologize for the inconvenience.

For more details on this, see the previous post here:
http://www.dreamhoststatus.com/2008/03/14/central-services-and-spunky-cluster-move/

UPDATE — Sat Mar 22 00:12:05 PDT 2008 –

Most central functions are back up. Dreamhost.com and panel.dreamhost.com are working again. We are currently working on webmail. The spunky cluster move is still underway. We’ll post more here later.

UPDATE — Sat Mar 22 02:45:26 PDT 2008 –

Webmail is up! 99% of all central services are functioning now. Spunky is 80% racked, more info on that to follow.

UPDATE — Sat Mar 22 03:40:27 PDT 2008 –

Spunky is coming up. We’re spotchecking and fixing troubled file, web, and mail servers. We’ll sound an all when all is clear… We will likely be firefighting for the next few hours.

UPDATE — Sat Mar 22 06:01:28 PDT 2008 –

Spunky is still coming up, we’re dealing with a couple of machines that didn’t survive the move and some more cleanup stuff. We’ll keep you posted as things progress!

UPDATE — Sat Mar 22 07:04:35 PDT 2008 –

We have noticed a problem with some of our networks. They seemingly did not make the jump to the new router and we are looking into that. Any websites with IP addresses that look like either 64.111.xxx.xxx or 208.113.xxx.xxx are down to the outside world right now. The machines themselves are up so as soon as the networks are working again they’ll begin working immediately.

UPDATE — Sat Mar 22 09:55:54 PDT 2008 –

The networking problem has been resolved with our network provider so as of right now all services should be up and running. If anything is not working, please open a new support message from our web panel.

FastCGI Environment Changes for Rails - Updated

greg — Fri, 14 Mar 2008 22:05:59 +0000

Recently we’ve made changes to FastCGI to disallow any environment except for Production when using Rails. This was brought on by some issues with the Development environment causing the quality of service to degrade. If you need to run your application in Development mode, please use your local machine for this. If you have a live application that was running in Development mode and now it doesn’t work, this is most likely the cause. You can correct this by making sure you have setup your database.yml file to have the proper MySQL connection information for Production, and that you’ve changed your environment.rb file to reflect these changes.

Update: I’m sorry, my first post isn’t entirely accurate. We are not disallowing any environment except for Production. We’re simply defaulting Rails installations to Production instead of their old default which was Development. This doesn’t mean we allow development on our servers, and in fact, we frown upon any development on our production servers because of the risk involved. I understand some people are concerned about staging, and this should be fine, so long as you use CGI instead of FastCGI.

LAX Networking upgrade

Kelly — Mon, 10 Mar 2008 19:54:49 +0000

On Wednesday, March 12, starting at 10PM we need to do a quick network migration. We are moving from one router to a newer, larger router. This will hopefully solve a problem we are having with load balancing across our links back to our downtown datacenter. Cisco isn’t sure what is causing it, and we have no free capacity to do the troubleshooting they need to properly diagnose the problem.

The outage should be around 30 minutes, and will affect dedicated servers, looney, swarthy, spacey, frisky, and blingy. This is going to run concurrently with the Blingy file server upgrade.

Update 11:15PM Pacific: As of right now, I believe we are back in business. The blingy cluster took a bit longer to show OK due to the upgrade concurrently happening. Please let Technical Support know if after right now you are still having problems!

Blingy file server upgrade

Kelly — Mon, 10 Mar 2008 17:52:59 +0000

On Wednesday night, March 12, 2008 starting at 10:00pm Pacific time, we will be performing a major OS upgrade on the file server that handles the entire blingy cluster. This upgrade is at the direct recommendation of BlueArc to hopefully resolve periodic slowdown issues and other quirks of the system. The estimated downtime by the BlueArc technician is 1 hour.

Quick update: This is the first upgrade we couldn’t do live behind the scenes. We have already upgraded once to the latest version that could be applied live to the clustered heads.

Update 11:15PM Pacific time: The BlueArc technicians are still working on the upgrade. It seems they are nearly completed, and one of the redundant pair of heads seems to be serving data right now. Stay tuned for a final update.

In other news, while BlueArc and I were standing there next to the file server discussing the problems that have been plaguing us at 7-10pm, it happened! The unit crashed and completely stopped serving data. I left them to gather their crash data and continue the upgrade process.

update 11:58PM Pacific: Everything looks to be better. Please contact technical support if you are having any problems!

Randy cluster move

mikeh — Sat, 26 Jan 2008 17:45:34 +0000

Due to continued space and power constraints in our primary data center, we will be moving the “randy” cluster to one of our newer data centers. This move will begin Friday, February 8, at 10PM PST, and is expected to last up to 8 hours, until Saturday, February 9, 6AM PST. All web servers, mail servers, file servers, and MySQL servers in the randy cluster will be unreachable during this time.

The following web servers will be off line during this move:

alexander arkanoid attila baht bam bob bomberman breakout caesar centipede contra cornelius cortes crack cruisin cssmania deblume defender digdug dinar dinero dollar donkeykong ellipsis euro fidget forint franc gage galaga gap go grant guilder happy herod ikari ike io ivan jezebel joust karnov katamari khan klax korben kroner lamlam leeloo lei limbo-randy limbo-randy2 limbo-randy3 lira lucky mark marvin morgan napoleon nexus nimitz orbital pacman paperboy patton peso pizarro pound qbert quid rhod ruble rupee rygar scagnetti schilling scipio shekel slaughter slimy tetris ugh ultima venice vex vito wasabi yen yoda zorg

If your site is on any of these servers this status post applies to you. Also, if you check the “Account Status” button on the upper right hand of the webpanel ( https://panel.dreamhost.com ) you should see your cluster under the section “Your Email Server:”. We are also sending out an email announcement about this downtime window to affected customers.

We do apologize for this inconvenience. Rest assured we will do everything possible to keep downtime to a minimum.

If you have any questions regarding the server move, please contact our support team.

New php4 packages

jeremy — Wed, 16 Jan 2008 20:21:43 +0000

In order to fix a couple of bugs with PHPMotion and to fix a couple of security flaws, we’re slowly rolling out new PHP4 packages to all servers. The changes are fairly minor, containing only a minor version bump and a fix to the binary installed to /usr/local/bin/php (it had previously been the CGI version rather than the command-line version). It’s already been tested on half a dozen servers without any issues so hopefully nobody (except PHPMotion users) will notice the change.

Panel downtime

mikeh — Thu, 27 Dec 2007 11:11:58 +0000

Next Thursday morning, we will be performing some maintenance on one of the database servers which powers our Web Panel. This requires us take the Panel off line. The downtime window will begin at 2:00am PST Jan 3, and extend for two hours, until 4:00am PST Jan 3. Expected downtime within this window is 1 hour.\

Panel is back online as of 3:00am.

Rails upgrade addendum

Terri — Tue, 18 Dec 2007 23:32:30 +0000

As many of you noticed, there was a slight glitch when you tried to freeze your rails. You probably got an ugly error like:

no such file to load — /home/.blah/blah/blah/config/../vendor/rails/railties/lib/initializer

Not very helpful! As it turns out, there’s a bug in rails 1.2.6 that causes this! Thankfully, it was a matter of just slightly adjusting one file, and all should be well.

Go ahead and try the freeze again, just make sure to delete the rails directory under the directory called vendor, otherwise it’ll look like it’s still broken.

I apologize if this was a pain. Again, please email tech if you’re having any further problems.

Dedicated Server Mail Upgrades

jordan — Tue, 18 Dec 2007 08:36:57 +0000

If you received an email last week about the mail service on your dedicated server being upgraded then you will be affected by this upgrade. If not then your upgrade will happen later this week and we will email you regarding the date and time of the upgrade. This upgrade should not cause more then 5-10 minutes of downtime where you will not be able to check email. When the upgrade is complete on all servers I will resolve this post.

UPDATE — 1:40AM PDT

The upgrades are finished. Mail service should have returned to normal and if you were upgraded you can login via user@domain.com now instead of mXXXXX mailbox names. If you are still having troubles please contact support.

Rails: The Revenge!

Terri — Tue, 18 Dec 2007 00:02:43 +0000

Just when I thought it was safe to upgrade, another version appears!

I’m going to start upgrading our servers to Rails 2.0.2. This involves packaging, testing, and then slowly rolling it out. The jump is to a 2.x version from 1.2.6, so I’m going to take my time with this one to make sure nothing goes horribly wrong, so it might be right after the new year when it’s all done, just to be safe.

For your new year’s resolution, you will freeze your rails! Login via shell, and in your rails directory, run “rake freeze_gems” - that should do the trick!

Please contact support if you have any questions or concerns.