Posted by: Jeff Timmins | July 31, 2015

RM Thought of a few weeks ago – Release failure

RM Thought of a few weeks ago – Release failure

The team responsible for making Production changes to the NYSE computer systems scheduled an update for Tuesday night the 7th of July. Unfortunately the Production change was not found to be not working correctly at the time the Market opened (9:30am ET) on Wednesday the 8th of July.

* Whoops

Per reports, the problem was not investigated until 11:00am – 1.5 hours after the opening. It is possible that reports didn’t get their Help Desk for an hour but it seems like a long delay to start their troubleshooting.

* Not trending well so far

After investigating for 30 minutes the NYSE decided to shutdown their trading systems – and suspend trading – around 11:30am.

* Seems fair to stop the bleeding but please note – the WORLD IS WATCHING! 

After 3.5 (about 3:00pm ET) the NYSE rolled-back the change by switching to backup servers in a different data center. That set of Servers had not yet been updated with the new software. This allowed the market to restart for the final hour of trading. I’m not sure of the details but per reports the outage had little impact to the trading of stocks.

* The WORLD IS WATCHING so probably doesn’t matter the impact.
* Great planning – you had a rollback plan that worked!
* Wow! That took long time to decide to rollback

The cause of the problem was … actually I don’t care. I mean it is interesting why it failed but the Release process main goal is to do no harm. It failed. Customers were impacted for about 5.5 hours with a hard down for 3.5 hours. I’m sure there was a real monetary cost associated with the outage but what about the impact to the Brand of the NYSE? What about all the news articles? Brands can take some poor press but too much can cause lack of trust & people to lose their jobs.

Don’t get me wrong, we all fail but I have a hard time accepting that large of a downtime when you actually have a nice rollback plan. If I was the Change Manager I’d make sure to ask a few hard questions the next day.

Sources & links to articles used

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: