verb. sap.

Keeping the customer satisfied July 24, 2007

The choral group I sing with, Saddleback Master Chorale, has, in recent years, been taking baby steps into the connected world.  We have a web site, which I occasionally remember to update (note to self: now would be A Good Time) and this year, we’ve started selling tickets online.  So, back in January, I was looking around for a company to work with in this space. A friend recommended Vendini, so I gave them a call, talked to Spencer Rosen, and we signed up.  Vendini does full-scale box-office solutions, but they seem to be happy with clients at the lower, less sophisticated end of the scale, which is where we sit.  In any case, we’ve run two concerts through them, and everything has worked just fine.

However, that’s not what I’m here for today.  Today is a case study of how to deal with a customer service problem.  Yesterday, I received an e-mail from Mark Tacchi, CEO of Vendini, apologising for an outage which impacted the service his company provides to its customers.  With Mark’s permission, I include the message below.

We had a network outage on Friday July 20, 2007 that affected several organizations.  I understand how important a role Vendini plays within your organization.  We’ve worked hard to build a system that you can trust will be available and responsive and I know that we’ve let you down.  I want to tell you that I am very sorry if this affected you.

Although we have taken much care in creating a redundant and fault tolerant system, the problem stemmed from three linked and cascading issues that made it difficult for our engineering team to quickly diagnose.  The first was our primary read/write database becoming unresponsive while replicating over to its slave servers and read only servers.  The second issue was an address conflict issue on the system’s private network which prevented certain servers from communicating with others.  The third issue was our primary firewall to the main application and database servers had reset its configuration.  This resulted in turning away requests to the system on that firewall for a period of time.

I was personally involved with the network engineering team at our data center in San Francisco to assist in coordinating a solution.  We had all hands on deck working to resolve the problem.

Going forward, we are taking several steps to make certain this does not happen again.  We intend to procure additional computer equipment, add additional staff, and tighten up our operations process.  We are committed to providing a world-class solution.

Please accept both my sincere gratitude for the trust you have placed in us, and my apology for this unanticipated service lapse. We take your business very seriously and your success is our success.

Please feel free to contact me should you wish to discuss further at…

Sincerely,

/Mark
Mark Tacchi
President & CEO
Vendini, Inc. – http://www.vendini.com

It’s hard to see how Mark, and Vendini, could have handled this situation any better.  Here is a CEO admitting to a problem, describing the nature of the problem, having hands-on involvement in the solution and talking about the steps they’ll take in remediation.  Mark also included his phone number and e-mail address. This is transparent, honest ownership of the issue, and, as a small customer, who wasn’t even affected by the outage, it makes me feel as though we made the right choice when we picked Vendini.

I don’t believe good customer service is rocket science, and, if we lived in a sensible commercial world, a message like Mark’s probably shouldn’t be worthy of comment.  Unfortunately, we don’t, and so it is.