RunningWithBulls.com - The curse of Sanfermin

[UPDATE:22GMT] Jack leaves a comment mentioning Greg Reinacker, founder and CEO of NewsGator has posted about the outage.

In the short of it, the MS SQL DB managed to write GBs of log files, take up 100% hardware capacity, and take the whole DB offline. Instead of failing over to another cluster node, the cluster goes taken offline fully, and DB recovery happens.
With all the new traffic from Europe and other timezones, more logs are written, and the DB gets deadlocked.
They take it offline, and start the recovery process offline.

The outcome is:

“Moving forward, we’re doing a number of things. Specifically, we still need to get to the root cause of this particular problem, but in general, we’re working a lot on our back-end systems to build additional redundancy, scalability, performance, and resiliency. We’re going to partition data in different ways, so a failure like this will only affect a portion of the user base, a portion of the content, or a portion of the functionality. Our back-end cache servers will expand to be able to serve data while underlying data store is offline. And lots more, which I won’t go into details about just now, since this post is getting a little long.”

I think a word of thanks has to go to Greg. And also, a cup of strong coffee. Sorry for the troubles Greg, but I don’t think anyone will really hold a grudge.

Servers go down, disks crash. You just try to have it happen as little as possible.

Heh, reminds me of some of our issues in the past. Lets not remember them tho’.

==

Just as I had finished writing my thank you, Jack came back to leave another comment.

“Sorry for the extra posting, but I wanted to also mention that we have a brief announcement about the outage on our forums:
http://www.newsgator.com/forum/shwmessage.aspx?ForumID=22&MessageID=16794

We will update this post later today when more detail about the outage is available.”

Don’t be sorry at all Jack. Thanks for the info.

At the moment the first post there says:

“NewsGator Online experienced system problems this morning which resulted in disruption of service. As of 10:30am MST these issues were resolved and services are now fully restored.”

Ok, so we know *that* happened, but it would be useful to know what happened, or at least some of what happened.

I made a suggestion to Jack that:

“Maybe a suggestion: you guys (the Technical Support guys) should get a technical support/maintenance blog to notify people about these kinda things. I have an ISP blog feed that keeps me up to date with technical outages from them. Useful when these kinda things happen.“

This would be handy. Although they do have the forums, they are hosted on the same site newsgator.com

[UPDATE:] Jack did create a nice OPML file of RSS feeds for all the forums as posted here, but as I said, they are hosted on Newsgator servers.

Anyway, lets hope that the forums post gets updated with some information.

And now, back to your usual programming.

Technorati Tags:
irishblogs, runningwithbulls.com, rss aggregator, newsgator, jack brewster

Technorati Tags:
irishblogs, runningwithbulls.com, rss aggregator, feed reader, newsgator, jack brewster, greg reinacker