It’s a national sport to kick a company when something goes wrong. Despite all the quality and approvals systems in the world, there will always be a weak link in any process that can cause a major fault. Here are some examples of how it can go wrong from the construction, mining and finishing with the doozy in the communications industry, industries well known for cumbersome and often over the top control mechanisms. Minor “glitches” that can cause massive flow on effects.
One of our good customers is a professional firm involved in the mining space. When it comes to their purchasing process, they understand the benefits of getting value for money, hence spread their exposure across a broad range of domestic suppliers. Payments are always on agreed terms without the BS that many companies have of blaming person J for not having signed off before it goes to K; meaning that their processes are indicative of a reliable and efficient procurement system. It was therefore a surprise to our accounts person when the company paid a batch of invoices twice, a day apart. Upon receipt of the second payment, she called their Finance Manager to advise of the problem and offer to return the funds. First there was silence, then a somewhat panicked “I’ll call you back”.
True to her word, not long after she called back, very grateful for the contact. Turns out that despite all the tick offs etc in the system, it hadn’t stopped the entire monthly payments run from being duplicated. She owned the problem admirably, despite the huge task now faced to contact all suppliers and get them to remit the funds back, and the flow on effect of having depleted the bank account.
The second example relates to respected construction contractor Shadforths. Their cardinal sin was that when a former employee departed they missed removing his access to certain computer systems. Simplified, this person then whilst working for a competitor over a 15 month period went in through the back door to look at Shadforths project submissions before the new company he was involved with then submitted their tenders. Hard to prove that tenders were lost as a result of this, but astonishing that the court only imposed a $15,000 fine plus a conviction recorded. As only certain people had sign in access, the access probably wasn’t even in the IT departments “standard Exit” forms, but a blind side for the company with an unknown cost implication. It’ll always be hard to safeguard against all types of wrongdoing, in fact most companies (including ours) that sell something will have been exposed to departing employees leaving with things like customer lists and becoming competition, hitting the bottom line.
The third example is the failure of the Triple 0 emergency services network, a failure that Telstra is wearing the brunt of this week, including a summons to appear at a Senate inquiry. I’ll start by declaring that I am by no means a Telstra fan, in fact my dealings with them as a customer and supplier is that they have massive issues with inefficiencies etc. But the way they responded when their network went down was actually exemplary. They provided all the updates and notifications they could whilst working feverishly behind the scenes to fix the issue, then conducted a massive follow up program including a proper apology. The CEO was on a planned holiday, and the delegation of duties worked as intended. Contrary to headlines, the Communications Minister was notified of the fault before the CEO, yet did nothing other than criticise through the media.
Except the Triple-0 issue wasn’t even Telstra’s fault… The Senate inquiry will find it much easier to go after a company than to admit Governments own failures. There is no requirement on communications providers to provide 100% availability, in fact they achieve 99.9% design accuracy for an individual tower and 99.99% because towers can back each other up. What happened in this situation is that a $30,000 node in one tower failed (a routine software update upset the time clocks a-la what we were warned about way back during Y2K). The problem then cascaded.
Now for a more correct blame game. Triple-0 had an outage 8 months prior when Optus had a network outage. A different company in other words, and it was identified that the Triple-0 failover to other networks didn’t work. Lots of chest thumping by our politicians back then about Optus failure too. Seems like no actions were put in place to fix it though.
A mix of Commonwealth and State/Territory Governments run Triple-0 but rely on the combined networks of the telcos also. There is no single agency or department responsible. And that’s the problem – our governments keep growing, making things more complicated but never take responsibility for issues of their own making.
So why didn’t Telstra have a redundancy or back up for the relatively cheap part? Well, there is one of these in every tower so they have thousands of them, so not as simple as it might seem. Second, and as I can’t explain it any better than Jared Lynch (Technology Editor in The Australian) did, I’ll borrow his explanation:
“In March, the communications regulator ignored almost everyone, except the rattling of the government’s big piggy bank, to impose a $7.3bn tax grab on telcos via spectrum – that thin air that mobile networks operate on – renewals.
This was up from an earlier assessment of $5bn to $6.2bn. Telstra argued that renewal price tag for the whole sector should be capped closer to $3.9bn.
Why? That money has to come from somewhere, and telcos which are experiencing stagnant revenue growth have two choices: hike prices (which Telstra already did in May) or trim back on capital expenditure and network investment.”
In other countries like Portugal, Spain and France, telcos don’t have to pay spectrum fees at all so long as they pay for the network. Here, it’s another tax double dip. When the Government sets up the industry framework, then strips the companies operating within it of as much money as they possibly can, they can’t also complain when the required maintenance doesn’t occur. (The same thing is happening to coal in the power grid). In effect, the controlled nature of the market transfers the responsibility squarely to the Government, and the question should be asked why didn’t the Government have a better back up?
The assessment might seem harsh, but it was the Government itself that made the outage a big issue. It will happen again, as no network can possibly have 100% reliability. For starters, this is a “big city issue” – there are large portions of the country that has no network coverage (and hence no triple 0) in the first place. Listen to the remote areas a bit further, and they are having regular power supply issues: no power = no Triple 0. Or anything else that runs on electricity.
It’s the Government departments that should be appearing in the Senate inquiry, and the obvious solution is to do away with all but 1 department being responsible and then actually hold them to account. And whilst they are doing their “forensic”, suggest they look at the electricity grid reliability, as I’m sure it’s lower than the telco’s offer, and has a much bigger impact.
Words from the wise
“It’s the little details that are vital. Little things make big things happen.” — John Wooden
“Insanity is doing the same thing over and over again and expecting different results.” – possibly Albert Einstein, but others also lay claim to versions of this.
As always, Onwards and Upwards!
Fred Carlsson
General Manager
