RadioWest.ca's Recent Journey

RadioWest.ca's Recent Journey

Postby jon » Sat Feb 11, 2012 3:20 pm

As of yesterday, this web site is now a 100% DreamHost-Free Zone. An hour ago, DreamHost "came clean" with a newsletter explaining their "perfect storm" of disasters in January. I'll share that in the posts that follow.

Our journey away from DreamHost began by figuring out where we should move to. And ended up selecting a place where you install all the software yourself. Kind of like buying a home computer with an empty hard drive. Your problem to install Windows and everything else you want and need.

Once that was up and running, it was time to move the programs, audio, video, pictures and such to the new web host. radiowest.ca could then be pointed to the new web host. But, the Catch is that it takes 48 hours for the change in where RadioWest.ca points to be "visible" worldwide. Even then, individual computers might have cached pointers to the old location.

In the middle of that process, DreamHost's normally stable shared hosting Database Server, or at least the one we used, failed repeatedly. So, we had to make an emergency move of the database that contains all the Posts, even though those outdated pointers to DreamHost left some of our users behind.

The final step, completed yesterday, was to move all of the registrations for RadioWest.ca and other web sites that share our web hosting account. From DreamHost to GoDaddy. This was not just a "go off of DreamHost" strategy, but a technical necessity if we wanted to reach the full reliability potential of the new hosting: our new hosting offers 5 ways into the site, but DreamHost only allowed us to define 4.

What follows are articles from the latest DreamHost newsletter, from earlier this afternoon. As I said only half jokingly after our 18 hour outage: I checked to see if DreamHost was publicly traded so I could "sell them short" (bet their stock would go down).
User avatar
jon
Advanced Member
 
Posts: 9256
Joined: Mon May 08, 2006 10:15 am
Location: Edmonton

Re: RadioWest.ca's Recent Journey

Postby jon » Sat Feb 11, 2012 3:23 pm

DreamHost
February 2012 Newsletter


Introduction

Holy crap guys and girls.

Holy crap.

"NightmareHost" is a term that's been thrown around by upset customers since the very beginning. Pick a name like DreamHost and you're pretty much asking for it.

Sticks and stones, right? Not this time. This is the first month where we feel like we've actually earned the title. And we're just as... nonplussed about it as you are.

January was a perfect storm of software deployment issues, critical networking hardware failures, and a security situation that prompted a large-scale password reset.

If you'd like to relive the magic of the last 30 days, January's troubles were documented in great detail as-they-happened over on our system status blog:

DreamHost System Status [ironically, the first item right now is about a 48 hour outage that just ended this morning, and may well have hit us, had we still been there - jon]

This should not come as much of a surprise - even if you didn't notice any problems with your site in January, the email we sent out to all our customers was hard to miss.

We let you down - of that we are acutely aware. All I can say is we're sorry beyond words and we're working on fixing things - a lot of things. We're hopeful that once we come out on the other side we'll again be that gleaming paragon of hosting excellence that you were supposed to be thinking of us as all along.

There's really nothing funny about January. Sorry.

The good news is we've got lots to announce this month, so let's get this show on the road.

-Brett
User avatar
jon
Advanced Member
 
Posts: 9256
Joined: Mon May 08, 2006 10:15 am
Location: Edmonton

Re: RadioWest.ca's Recent Journey

Postby jon » Sat Feb 11, 2012 3:24 pm

Simon Says... (Written by CEO Simon Anderson!)
As Brett has said, January was not a good month for our level of service. And on behalf of all of us here at DreamHost, I want to apologize and let you know what the issues were, and what we're doing to fix them.

First up, security. We're now a mid-sized global web hosting company with over 300,000 customers and 1.2 million domains under management. That means we get our fair share of hack attempts from around the world. Sometimes it's just kids looking to prove themselves, other times it's more malicious than that - people wanting to take down customer blogs or websites because they don't agree with the opinions expressed on them; people wanting to steal other folks domains, that kind of thing. We put a lot of people and dollars into our security software and systems and protect our customers from these attempts every day. But in January a hacker got through our defenses and we took the rapid action of resetting all customers' FTP/shell passwords to prevent any malicious activity. In the process we identified a whole slate of improvements to our security systems, software and data management that we've already been implementing. Security is one of those issues that we can't give too much detail on, but rest assured we're on it and have upped our game substantially so that this kind of event is prevented in future. And as outlined in the newsletter below, we're also asking you to participate by using strong passwords and other best practices.

The second issue in January was related to our network. Given our unlimited bandwidth policy for shared, VPS, and dedicated customers, we have a network that has been designed for speed using tried and tested network equipment. However, the complexity of our routing and switching systems has increased as our global customer base has grown. As a result, we experienced some unplanned network downtime in January due to network configuration errors, plus the testing of some new distributed denial of service (DDoS) equipment. We've already been in the process of a focused effort since late last year to upgrade and streamline our network for the next 5 years of customer needs. But these issues have accelerated our planned efforts in this area. We'll be investing over 1 million dollars in network upgrades in 2012, including a new US east coast data center to better serve our European and east coast customers.

Third, we had an issue with software updates in January that resulted in some service downtime. The issue arose because of the way we have approached security updates for the operating system that runs on most of our servers. We've typically tested security updates to ensure they install correctly, but our testing of whether they break anything has typically been "light" due to the fact that we haven't seen any such issues in the past. Well this has proved to be a flawed approach, because one such update in January went badly wrong, and automatically removed some critical software on production servers, causing downtime and an understandable amount of customer angst. We've immediately changed our software update process to fully test all software going into production, and we'll be acutely focused going on ensuring that any necessary software upgrades are performed efficiently and quickly in planned maintenance windows going forward, with minimal downtime.

Finally, a quick word about our team. We have very dedicated, smart and super friendly people on our support, systems administration, development, engineering and data center operations teams. They work long hours and are absolutely passionate about delivering you an awesome and flexible web hosting service at a great price. When there are service issues like we've seen in the last month, they pull out all the stops -- nights, weekends, 24/7 -- to fix the issues and plan out changes to give you a better experience for you in future.

I'm fully committed to ensuring we deliver a reliable, flexible and useful service for you going forward. And I believe we're adapting and focusing on the right priorities and investments to continue to deliver a great hosting service and experience.

Best,
Simon
@DreamHostSimon

(Now back to Brett!)


[I won't bore you with the rest of the newsletter, but the end is really odd, so I will include that]
User avatar
jon
Advanced Member
 
Posts: 9256
Joined: Mon May 08, 2006 10:15 am
Location: Edmonton

Re: RadioWest.ca's Recent Journey

Postby jon » Sat Feb 11, 2012 3:26 pm

See ya next month!
See ya next month!
Brett D!

P.S. - We apologize if you smiled reading any of this. If smiles did happen, feel free to unsubscribe.

https://panel.dreamhost.com/id/?tab=contact
User avatar
jon
Advanced Member
 
Posts: 9256
Joined: Mon May 08, 2006 10:15 am
Location: Edmonton

Re: RadioWest.ca's Recent Journey

Postby jon » Sat Feb 11, 2012 3:43 pm

Personally, since moving to DreamHost in October, my technical world has changed completely. Around 2006, our web hosting changed from just being a place to store stuff that you could see on the Internet, to a feeling of having a fully functional computer at our disposal. With that change came something called cPanel that gave a point and click Windows-like control of all things I ever needed to do, such as change e-mail addresses, add a new web site to the account, etc.

But, neither DreamHost nor our current web host (Linode.com) has cPanel. Much has to be done by typing a command line, much as we all did in the days of IBM and Microsoft DOS. Including monitoring what is going on, especially when things aren't working so well.

On our new site, I've discovered why DreamHost worked so poorly. DreamHost made a decision -- about a year ago, according to a friend who has been on DreamHost for a couple of years -- to eliminate something called a Swap File. During periods of high activity, instead of gracefully having active processes moved out to disk from memory into a file known as the Swap File whenever we ran out of memory, everything essentially froze or, worse yet, a server reboot was forced. Which explains why we experienced so much slowness and shorter outages. As my friend put it, the only way to avoid these problems at DreamHost is to pay for 4 times as much memory as you really need for your normal workload. Which, of course, gets very expensive.

So, I didn't expect to learn all about Linux and web software in my retirement -- WordPress, phpBB, php and MySQL was enough, I thought -- but that is the way it has turned out. The end result is that I hope that by creating a very simple web site environment, with very little software installed, we can finally deliver good reliable web hosting to RadioWest.ca and our other web sites.
User avatar
jon
Advanced Member
 
Posts: 9256
Joined: Mon May 08, 2006 10:15 am
Location: Edmonton

Re: RadioWest.ca's Recent Journey

Postby CubbyCam » Sat Feb 11, 2012 4:10 pm

My condolences to all at RW Tech... You did amazingly well under the circumstances. If it's any consolation, I've hosted my sites on GoDaddy for years... and never a hiccup. Keep on truckin! And thanks again for all you do. :salute:
CubbyCam
Advanced Member
 
Posts: 394
Joined: Mon May 25, 2009 11:52 am

Re: RadioWest.ca's Recent Journey

Postby jon » Sat Feb 11, 2012 4:42 pm

Just to clarify, we only do our domain registrations at GoDaddy. I'm doing the actual hosting of the web sites themselves with "roll your own" style software installs at Linode.com where, ironically, I steered reelradio.com 3-4 years ago and they couldn't be happier.

Why didn't we go there at the same time? Because, until recently, Linode did not have the step-by-step software install guides I needed to get me going there.
User avatar
jon
Advanced Member
 
Posts: 9256
Joined: Mon May 08, 2006 10:15 am
Location: Edmonton

Re: RadioWest.ca's Recent Journey

Postby Jack Bennest » Sun Feb 12, 2012 8:30 am

C panel is good

My compliments to DreamHost for admitting their failures of service. Many companies blame someone else.

That is why I moved away from the last unnamed company that bragged alot but could not explain their down time.

So far so good with my New York buddy.
User avatar
Jack Bennest
Advanced Member
 
Posts: 4471
Joined: Mon Apr 17, 2006 9:25 pm

Re: RadioWest.ca's Recent Journey

Postby TRENT310 » Thu Feb 16, 2012 2:04 am

I noticed the netblock owner changed when I logged on. What are the plans for jonradio.com?
TRENT310
Advanced Member
 
Posts: 235
Joined: Tue Aug 07, 2007 11:40 pm
Location: Whitecourt/Edmonton, Alberta, Canada

jonradio.com

Postby jon » Thu Feb 16, 2012 9:01 am

TRENT310 wrote:I noticed the netblock owner changed when I logged on. What are the plans for jonradio.com?

As most of you know, all of the software we run on this site (RadioWest.ca), EdmontonBroadcasters.com and other sites I host to help pay for our web hosting account, is Open Source = developed mostly by volunteers and licensed for unlimited use by anyone and everyone, for free. I have always felt the Need to "Give Back" to the development community that is responsible for each Open Source software product that I use. Up to now, the only practical approach has been to fully report any bugs I find, test Beta versions (but not on the public web sites you would see), and answer questions on the official Support forums.

I am very close to taking this Giving Back to the Next Level, at least for WordPress, which drives View Most Recent Posts on the home page of RadioWest.ca, and most of the other web sites. I have a Plugin nearing completion that would add full support for IPTC fields for .jpg files to use for captioning still images in WordPress (IPTC fields allow you to store text inside a digital photo file without corrupting the image itself). Plugins are the WordPress way to allow additional functionality to be added to the WordPress software by anyone.

jonradio.com will be the website associated with this and any future Plugins I create. I may also add personal material to the site that would not be relevant to the non-forum portions of RadioWest.ca. For example, I would find it convenient to list all the cuts on my Various Artists CDs on a web page, sorted by Artist.

Nothing there to see yet though. The WordPress there is Multi-Site, which I'm currently using for testing and will eventually also drive a number of smaller web sites I'm working on.
User avatar
jon
Advanced Member
 
Posts: 9256
Joined: Mon May 08, 2006 10:15 am
Location: Edmonton


Return to Welcome to RadioWest.ca

Who is online

Users browsing this forum: No registered users and 84 guests