SFdC system goes down

I guess I have to take the blame for jinxing Salesforce.com because in an earlier post about Typepad's outage I pointed out that SFdC doesn't experience significant outages. This morning Simon commented that I spoke too early because SFdC was in fact down for 5 hours on Tuesday.

A couple of things jumped out in this story, the first being that they seemed to have very poor communication about the outage. One lesson in this is that companies offering hosted services need a completely different comm strategy for dealing with product issues than their non-hosted counterparts. It's really inexcusable that SFdC would have no status messages on their website and an automated recording on their customer service line that was really no better than not having any message at all.

Secondly, any company offering hosted services needs to invest in redundant infrastructure with rapid failover, something that SFdC was apparently already investing in but not live with at the time of the outage. Earlier this year we saw the trouble that a company can get into when they consolidate datacenter operations, in this case it was Comcast consolidating their regional DNS to larger consolidated operations. Whether from natural disaster or plain old system failure, there is too much risk in not having redudant datacenter capabilities in hosted systems operations.

All-in-all they seem to be doing a pretty good job of managing their datacenter but as with anything in this business there is always room for improvement.

Comments

[john]

I think this is the start of a trend. Robert Cringley, whose articles I don't always agree with, made some good points about hosting on the internet:
http://www.pbs.org/cringely/pulpit/pulpit20051020.html
To make a long story short, uptime is going to get very expensive for internet services due to increasing demand for rackspace and energy.

[Neville Hobson]

Jeff, you mention communication:

"One lesson in this is that companies offering hosted services need a completely different comm strategy for dealing with product issues than their non-hosted counterparts."

I think any company must have a clear communication plan ready for addressing a potential crisis such as a service outage.

While there may be different tactics required in terms of the specifics of how an individual company will communicate and what they need to say, there are some essential things to do that are common to a situation no matter who the company is.

We saw with TypePad last week what happens when you have a communication vacuum - the customers talk among themselves. A lot of anger out there, amplified by the lack of meaningful and timely information from the company.

I've not seen any customer commentary anywhere re Salesforce.com's outage on the scale we saw re TypePad's.

Maybe that's because bloggers tend to be passionate and opinionated and perhaps CRM users aren't ;)

[Charlie Crystle]

Jeff,

For the past 5 months, Salesforce has consistently slowed down in performance in the last 10 days of every month. Click and wait, click and wait. This isn't the first time performance and reliability have suffered.

If their performance matched their rhetoric, it would be an acceptable platform, were it not for the terrible UI. But that's another story for another time.