Brandkit May 15th 2019 Outage Report

Last week on May 15th Brandkit was down for 95 mins - which is not a good thing at any time.

It happened to coincide with New Zealand's largest Tourism Conference, TRENZ 2019, where many of our New Zealand Tourism and Destination Marketing customers were exhibiting and demonstrating their image libraries to their customers - mainly travel buyers from offshore.

We had the problem solved and we were back to normal by 9:35am - but we still had several concerned calls from customers between 8:15am and 9:30am NZST (8:15pm to 9:30pm GMT), as they prepared for a busy day at TRENZ.

Here's the story of what happened and lessons learned.

The good news - it wasn't anything technical with our software or technology and no data was lost.

The not so good news - it was human error - my human error.

It all started in April when Westpac, our bank, called to say they had blocked some suspicious CC transactions on our company CC card.

After running through the transactions with the bank, they were clearly fraudulent. We were lucky that Westpac had the systems in place to detect and block these payments,

How it happened - we'll never know - but I was in San Francisco in February this year and used the card at local restaurants, and merchants. So that could be related.

Disaster avoided. 

However, as a software company, most of our supplier agreements are with other software providers, typically in the US. Most of our supplier payments (as is the fashion today) are via paid by regular credit card payment. That same credit card is used for those payments.

Westpac advised that we needed a new card with an entirely new number. Oh boy - we were going to have to update a lot of supplier accounts with the new details.

The worst aspect of this was that we had to wait a few days before the new card and number would be issued and couriered. Westpac could not issue the number by phone or electronic means. Meaning we were not able to make any payments for a few days - and we could not update any supplier accounts.

The is the first problem of these card fraud events. There was nothing we could do until the new card arrived. If a bill became due during this time - they would have to wait.  Not easy to negotiate when you're on an automated CC billing cycle.

A few days later we received the new card and number and proceeded to update our most important supplier accounts first.

One of these Services was Heroku - our platform as a service provider. Together with Amazon Web Services these two suppliers provide our core infrastructure.

The CC details were updated and in April Heroku took their usual monthly fee. All good.

What we didn't know at the time is the March invoice hadn't been paid, it was caught in that waiting period for the new card. I had assumed that when we updated the card, that Heroku would have taken the payment for that period - as most vendors do. They didn't. We discovered that last week. We had to make a manual payment to clear that invoice.

So despite taking payment for the later April invoice, the March invoice remained unpaid.

Nevertheless (we thought at the time) - everything was back to normal.

On the 12th and 13th May we received an email that there was a past due invoices for payment. So on Monday 13th, I clicked through to check that I had actually updated the CC details. I assumed this related to the recent period (the April invoice). So I re-entered the CC details again assuming they would take the outstanding payment (spoiler alert: they didn't).

On 15 May at 8:13AM NZST (8:13pm GMT) Heroku sent an email advising our account was suspended.

Due to unpaid invoices, your Heroku account has been suspended and is scheduled for deletion. Please visit https://dashboard.heroku.com/account/billing/pay-balance to update your credit card information and pay the outstanding balance. Once the invoices are paid your account will be automatically reinstated.

Shit. I couldn't believe this - I assumed once the card details updated they would charge the card for any outstanding invoices.

“Assumptions are dangerous things to make, and like all dangerous things to make -- bombs, for instance, or strawberry shortcake -- if you make even the tiniest mistake you can find yourself in terrible trouble."

I screwed up.

I assumed that Heroku would take the new CC details and take the payment for the March Invoice. They didn't even try. It was up to me to do it manually and I didn't.

Irrespective of how it happened, the most important thing was to get back up and running.

Once we realized what was going on it was going to be dead simple to fix. Just log in and pay the invoice and we'd be back in business.

How wrong I was.

8:30am Alerted to the outage - we were trying to figure out what was going on.

8:45am  I found the email from Heroku, realized what the problem was and logged in to Heroku to pay the offending March invoice.

Oh ooh - Failed transaction message - "there's a problem with your card".

What?

I tried again. Same thing.

What's going on? I checked with the bank that we had funds to pay in the CC account. Yes - all good.

8:50am Then a text message from our bank (Westpac) saying that a suspicious transaction was detected and a request to call the Card Team via the phone number on the back of the card.

My guess is that they had flagged the transaction as odd and blocked it - hence the transaction failure at Heroku. It was a guess - because we didn't get and still haven't received any information from Westpac about the block or why.

I thought, come on guys, this is a legitimate purchase from a regular supplier. Why block it?

Well at least there's a hotline to the cards team - so we can sort it out quickly. I just need to tell them the transaction is legitimate right?

Well no.

The phone number is not a hotline to resolve card issues - instead, it's just a phone number that gets the regular Westpac voice queue system!

After wading through number menus and listening to recorded service ads, I end up with the Westpac Cards team - on hold. Wait time 15 minutes.

9:05am After 10 minutes waiting with no response.

I dialed the same number again on another phone and this time, instead of selecting Queue options, I just waited for the general call center staff to answer - which they did after another 10 minutes.

9:20am 5 minutes later after going through the various identification checks, the operator could confirm that the transaction had been blocked, but could not explain why or do anything about it. Instead, he had to put me through to the cards team - and you guessed it - into a new wait queue.

I had two calls open and in the end, the new transferred call was answered. Another 5 minutes going through the same identification process (Grrrr) and finally I was speaking to someone in the Cards team who could actually help. Then back on hold while they checked something.

It turned out they had put a block on the card for all transactions - not just Heroku. Why? - they couldn't really tell me - although they had flagged 3 other US supplier transactions (which all from regular suppliers).

9:30am Back on hold (Grrr) while they checked some things and removed the block.

The thing that really annoys me - is that these were all regular suppliers who charge us on more or less the same day each month. It's likely that the previous hack in March has raised the sensitivity on our account in Westpac's system. I'm guessing because I've had no explanation from Westpac.

Long story short. Westpac removed the block and I was immediately able to complete the payment on Heroku's site.

9:35am and we're back

Lessons learned

This whole episode should not have happened and it was my fault.

My assumptions about Heroku's process were plain wrong and I am guilty of not paying enough attention. That said Heroku's communications could have been clearer.

Once we were aware of the issue we could/should have resolved it in 5 minutes, except for Westpac's block and subsequent, on-hold process.

1. Don't assume online providers will charge your card for invoices that previously had declined transactions when you update CC details. 

2. Use a different Credit Card for online business service purchases (especially SaaS) vs Staff expenses (assuming it's less likely to have fraud issues with cards only used for online B2B transactions).

3. The Bank has its own opaque process for blocking credit card transactions and it's not always possible to get to the horse's mouth to understand why or to get something actioned in a hurry. It's poor service from the bank - but they're all tarred with the same feather right?

4. If you get hacked or have fraud events, you are going to be inconvenienced through no fault of your own, and there is often no one to blame. Get over it. You are going to have to deal with it.

Beyond our control

While we take full responsibility for this outage, our suppliers could make changes to help avoid these situations, and help us respond faster in the future.

The following is beyond our direct control - but we'll be asking our suppliers if they can make the appropriate changes.

Heroku should be clearer in their communications about what is going on. They should be saying either the CC charge failed, Or that there is an outstanding invoice, not concatenate the two into a generic message.

In my opinion, they should also consider phoning or texting an alert that suspension is imminent. Email is just one method of communication available to them and there is a lot of noise. Emails can be missed.

Equally, we should be diverting this type of email to a priority inbox to deal with more urgent matters. The challenge is to filter effectively.

There is a larger/ongoing problem with email being used for alerts in general by many services (us included). There are just so many emails.

In fact, we get multiple emails from bot@heroku.com multiple times per week. Heroku could do a better job of separating billing emails from other updates, consider phone alerts or even better a good old fashioned phone call. If you had a client with 5 figure billing - wouldn't you at least get an account manager/rep to call to ask what's happening with the account in these situations?

Westpac (as with most banks and large customer service organizations) really need to reconsider how they deal with inbound phone calls from existing customers. Calling most call-centres is a frustrating experience. Westpac is no better and no worse than most. The biggest issue is that call-centres seem to prioritize new business over supporting existing customers. I'm not saying Westpac does that, but in general, it seems that way.

For Credit Card fraud, I strongly believe the Bank should have a hotline or be able to respond immediately - no waiting around. Banks should be giving priority to these incidents and a direct line/dedicated number should be imperative.

Secondly, the Bank should be (a) calling customers and (b) explaining to customers when and why a CC block occurs. To me, this should be a baseline service. They already detected a suspected fraud, it's a lot of money and they have a business customer doing a lot of transactions through the account on a regular basis.

A text message with call-centre queue as the only way to get in contact - is just not good enough in my books.