This article was written by Darryl Veitch and Allison Kealy for The Conversation. Read the full article here.

Telstra experienced a second major network fault after yesterday’s nationwide outage, with the telco confirming late last night that some calls, including to Triple Zero, were not going through.

At a press conference this afternoon, Michael Ackland, Telstra’s Chief Financial Officer, apologised for the disruption. He said the company had completed 639 welfare checks on people who tried to call emergency services, and that seven people required assistance.

Yesterday’s outage crippled more than just people’s ability to make calls. It also brought down train services, payment systems, public transport ticketing systems and electric vehicle charging stations.

Telstra has blamed the outage on a computer timing failure in its network. But how can Telstra’s clocks have gone out of whack? Could this happen with other pieces of critical infrastructure that rely on accurate time? And what can be done to make timing systems more resilient and robust?

“Maybe you’re having a phone call, or you’re scrolling through TikTok, you’re breathing on it, and it says, ‘hey, you’ve got the markers for the flu, you should go to the doctor.’”

Perhaps the device could be used by a doctor to track their patient’s health in remote areas that do not have access to state-of-the-art facilities, or without the need for invasive procedures.

In time-sensitive situations, it could reveal the identity of an infectious disease so it could be treated immediately, or whether a sportsperson has dabbled in doping.

Scholten and her colleagues are working to make these dreams a medical reality.

A clock that ticks with the universe

All computers incorporate what’s known as a software clock.

This clock serves several functions. It allows the computer itself, and any applications running on it, to tell the time. It also enables events to be timestamped and ordered, so we can ensure they are unfolding as they should and take timely action if needed, and enables time intervals to be calculated, so we know when something is taking too long or is happening too quickly.

If a software clock is inaccurate, many things can go wrong. For example, events may be accepted when they should be ignored, or flagged as dangerously late when actually they’re fine. Response times may be assessed as adequate when they’re anything but.

For a software clock to track real time, it needs to connect to timing hardware – something that actually “ticks” with the universe.

Typically, this is provided by an electronic circuit known as an oscillator that contains a thin piece of quartz crystal, relying on it to produce a regular “tick”. But these don’t tick perfectly. Left to themselves, a clock built on them will drift off quite quickly.

To ensure all software clocks agree, they must take input from a more accurate source aligned with a standard time reference, such as UTC (Coordinated Universal Time). They can get this standard time from satellites via a GPS signal, or access it over a data network. This is why your laptop clock never runs behind, while a simple battery-operated desk clock can run behind or ahead.

Most of the world’s computer population makes use of a hierarchy of time server computers. These communicate timestamp information (via the “Network Time Protocol”) over packets sent over the internet.

At the top of this hierarchy are so-called Stratum-1 servers. These are the only servers that actually connect to reference hardware sources, such as GPS. A single Stratum-1 server communicates time to a set of Stratum-2 servers over the network, each of which communicates to a set of Stratum-3 servers, and so on. The goal of such a system is to propagate the reference accuracy down the hierarchy inexpensively.

However, such a network brings vulnerabilities, as each Stratum-1 guides (or misleads) an entire server tree below it.

More accurate timing networks rely on additional hardware and dedicated links, and can be far more expensive.

So, what went wrong with Telstra?

The telco giant has said the outage was caused by nodes that managed time synchronisation within some of its network data centres.

The Sydney Morning Herald cited two internal sources who said a faulty update caused some of Telstra’s servers to reset their clocks by almost 20 years, making them believe it was November 2006.

We do not have any details beyond this at this point.

But it’s entirely possible these were Stratum-1 NTP servers, and that their issues misled the servers and computer lying below them in the hierarchy across the data centres and perhaps beyond.

The resulting errors could then have led to a cascade of secondary effects involving multiple interconnecting software systems.

Keeping track of time

Telecommunications is only one domain in which timing faults can have serious consequences.

In fact, because of society’s now enormous dependence on GPS as a source of time as well as position, the vulnerabilities are very broad indeed. A particularly important example is the electricity grid, which is critically dependent on timing for its fundamental operation, often provided by GPS.

The war in Ukraine has highlighted the use of GPS in drone attacks. However, blocking GPS as a means of protecting against such attacks brings with it the risk of crashing other infrastructure, including the financial system, freight and the electricity grid.

Improving the resilience of timing systems requires more than simply making clocks more accurate. This has been the focus of our own research on replacing the Network Time Protocol hierarchy with a more robust alternative.

Future critical infrastructure needs to have multiple, nationally distributed timing sources, rather than a single one.

To this end, the United Kingdom has committed £180 million ($347 million) to establishing a nationally distributed, resilient timing infrastructure to reduce reliance on vulnerable GPS timing signals.

Networks of geographically separated clocks, linked by secure land-based communications, can continuously compare and validate each other, automatically detecting faults and correcting anomalies.

Diverse clocks can be used, including optical clocks that are based on optical frequency combs. These are devices that enable a very precise bridge to be built linking light to radio frequency technology.

By combining multiple independent timing sources, these systems provide redundancy, diversity and the ability to isolate compromised components before failures cascade across interconnected infrastructure. Australia would be well served by learning from the UK’s example.

Read the full article here.