FlowTT country manager on outage: Time to switch internet connectivity

On the evening of December 11, FlowTT country manager Kurleigh Prescod and two communications professionals sat for an exclusive virtual interview with technology journalist Mark Lyndersay to explain what happened to Flow’s Internet connection on the morning of December 7.

On December 1, the internet alarms were ringing in Flow’s offices. Monitoring systems in Miami and Latin America were registering an interruption in service between Curacao and Guadeloupe.

That cable break affected all islands north of Trinidad and south of Guadeloupe, including Barbados, Dominica, St Lucia, St Vincent and Grenada.

But resilience in Flow’s networks made it possible to continue delivering service. Land-based teams were quickly mobilised to investigate, and it soon became clear that the break was beneath the water, between two landings of the CWC cable EC Link, a 987-kilometre run of fibre-optic cable activated in 2007.

Land-based tests suggested that the break was closer to Guadeloupe, and divers were assigned to the repair.

Kurleigh Prescod, country manager of FlowTT, said the company has dive teams on standby in the region – but they weren’t in Guadeloupe. Then covid19 restrictions slowed the arrival of the teams to the site.

Flippers were in the water on December 6, and the damage was located during that first day. Repair splicing on the break in the line was completed two days later, drifting well into the evening hours.

But that wouldn’t be the company’s only problem.

In Curacao, a nationwide power outage shut the country down on December 7, while the EC Link cable route was still under repair.

The tiny island – just 24 per cent larger than Tobago, at 444 square km – declared a covid19-related state of emergency on December 11, and it has been reeling from a series of national power shortfalls since September.

The Curacao Chronicle reports that power company Aqualectra is down to delivering 110 megawatts – to a country with a demand for 120 megawatts.

That loss of connectivity crushed Flow’s capacity. And because Flow’s parent company Liberty Latin America is a bulk bandwidth provider in the Caribbean region, the disruption affected other ISPs which experienced varying levels of network congestion.

“Our (uninterruptible power) batteries did keep the system going, and the generators did kick in,” Prescod explained. “The facilities on the island remained powered. But there was a disruption on another aspect of the infrastructure that led to the outage.

Nevertheless, he said, “The vulnerability was in-country and was resolved between 10 and 10.30 that morning.”

The FlowTT manager declined to offer details of the nature of the Curacao failure, but pointed out that another nationwide power outage happened on December 10, providing an inadvertent, but successful, test of the infrastructure repair.

“We were able to keep the majority of services live after the first break,” Prescod said. “While we did not have the capacity to carry full network traffic, we were able to do diagnostics.”

Prescod believes that more redundancy in the internet connections to TT would be better for the country since although there are four connections, only three are capable of supporting the throughput expected in 2020.

The oldest active connection, Americas-II, is now 20 years old and approaching the end of its life.

The cable itself is likely to be capable of more throughput, but the supporting equipment, both on land and under the Caribbean Sea, would have to be upgraded.

That’s likely to be a case of the candle costing more than the funeral for a connection that went live in August 2000.

Americas-I, the first cabled link to Trinidad, in 1994, has been retired, and is now only a connection between Miami and the British Virgin Islands.

The other connections are the Suriname-Guyana Submarine Cable System (SG-SCS), a 1,249-km link opened for service in July 2010, and Digicel’s 3,000-km Southern Caribbean Fiber, active since September 2006.

“It is not a trivial investment,” Prescod said. “There are opportunities for growth that have been tabled, but a subsea cable system is not inexpensive, it costs tens of millions.”

The last subsea cable connection cost Flow US$40 million.

“You have to be able to build a return on a significant investment of that nature.

“A collaborative effort among all the players (working) with the government might be able to build a business case for the development of a new connection.”

Prescod has some ideas about where that should happen, noting that all the connections to Trinidad are in the north of the country, an unfortunate congregation of vulnerability.

He is also concerned about the fact that there’s only one data cable connecting Trinidad to Tobago.

“Ideally, we should have a cable connection that connects on the south coast of the island to ameliorate risk, and a subsea cable that linked Trinidad with Tobago and then with Barbados would make sense in a geographic line.”

If it seems absurd that a country just off the South American coast makes all its connections looking north, that’s a consequence of TT’s consumption patterns. All the country’s data comes from the north, so connecting to an internet backbone in South America wouldn’t be as helpful as it might seem.

Prescod is also a key figure in the TT Internet Exchange Point project (TTIX), a network of local servers founded by a collaboration of local ISPs to keep data traffic in-country as much as possible.

But that system didn’t work as well as it should have during the outage.

Several banking systems failed, because they required offshore connections and the Linx system, which is entirely local in focus, also collapsed.

“There were things that continued to work,” Prescod said. “If your DNS was hosted here, it would resolve correctly.

“We had all these super-fast connections, the networks were up and connected; but because the app is based off-island, users could not make a connection.”

“We have reached out to the app providers to have local instances that can continue to operate in the absence of an overseas connection.

“An example would be WhatsApp. If a (WhatsApp) server was hosted locally, it would transfer local messages.

“A lot of the app providers (including Facebook and Google) have content and servers located locally.”

The TTIXP commissioned its own domain-name root servers in 2015. DNS software and hardware manage domain names and directs URLs to unique internet addresses. The local servers host copies of a D-root server provisioned by the University of Maryland, and an E-root server, provided by NASA.

While the primary intent of the local root servers is to speed up requests, the system also makes it possible to keep internet traffic completely local when external connections are spotty or non-existent.

Making the system more locally focused demands more investment in local infrastructure and a willingness to create and deliver local content and applications locally to provide a fallback position should external internet connections fail.

But that isn’t as easy or as cheap as buying the commodity cloud services readily available in the first world.

Mark Lyndersay is the editor of technewstt.com.