A “ridiculously weak“ password causes disaster for Spain’s No. 2 mobile carrier

Orange España, Spain’s second-biggest mobile operator, suffered a major outage on Wednesday after an unknown party obtained a “ridiculously weak” password and used it to access an account for managing the global routing table that controls which networks deliver the company's Internet traffic, researchers said.

 

The hijacking began around 9:28 Coordinated Universal Time (about 2:28 Pacific time) when the party logged into Orange’s RIPE NCC account using the password “ripeadmin” (minus the quotation marks). The RIPE Network Coordination Center is one of five Regional Internet Registries, which are responsible for managing and allocating IP addresses to Internet service providers, telecommunication organizations, and companies that manage their own network infrastructure. RIPE serves 75 countries in Europe, the Middle East, and Central Asia.

“Things got ugly”

The password came to light after the party, using the moniker Snow, posted an image to social media that showed the orange.es email address associated with the RIPE account. RIPE said it's working on ways to beef up account security.

 

 

Security firm Hudson Rock plugged the email address into a database it maintains to track credentials for sale in online bazaars. In a post, the security firm said the username and “ridiculously weak” password were harvested by information-stealing malware that had been installed on an Orange computer since September. The password was then made available for sale on an infostealer marketplace.

 

 

Researcher Kevin Beaumont said thousands of credentials protecting other RIPE accounts are also available in such marketplaces.

 

Once logged into Orange’s RIPE account, Snow made changes to the global routing table the mobile operator relies on to specify what backbone providers are authorized to carry its traffic to various parts of the world. These tables are managed using the Border Gateway Protocol (BGP), which connects one regional network to the rest of the Internet. Specifically, Snow added several new ROAs, short for Route Origin Authorizations. These entries allow “autonomous systems” such as Orange’s AS12479 to designate other autonomous systems or large chunks of IP addresses to deliver its traffic to various regions of the world.

 

In the initial stage, the changes had no meaningful effect because the ROAs Snow added announcing the IP addresses—93.117.88.0/22 and 93.117.88.0/21, and 149.74.0.0/16—already originated with Orange’s AS12479. A few minutes later, Snow added ROAs to five additional routes. All but one of them also originated with the Orange AS, and once again had no effect on traffic, according to a detailed writeup of the event by Doug Madory, a BGP expert at security and networking firm Kentik.

 

The creation of the ROA for 149.74.0.0/16 was the first act by Snow to create problems, because the maximum prefix length was set to 16, rendering any smaller routes using the address range invalid

 

“It invalidated any routes that are more specific (longer prefix length) than a 16,” Madory told Ars in an online interview. “So routes like 149.74.100.0/23 became invalid and started getting filtered. Then [Snow] created more ROAs to cover those routes. Why? Not sure. I think, at first, they were just messing around. Before that ROA was created, there was no ROA to assert anything about this address range.”

 

That disruption was soon corrected. Then, beginning at 14:20 UTC, Madory wrote in Thursday’s writeup, “things got ugly.” That's when Snow published four new ROAs that contained “bogus origins,” meaning origins that had no connection to Orange. As a result, the number of routes that originated from the Orange AS dropped from roughly 9,200 to 7,400, as a relatively new BGP protection known as RPKI—short for Resource Public Key Infrastructure—prompted many backbone carriers to reject the announcements.

 

Ironically, the purpose of RPKI is to prevent route hijacking, an all-too-common occurrence in which a party either intentionally or mistakenly publishes ROAs that route traffic through unauthorized IP address ranges, sometimes posing serious threats to Internet stability and even potentially to national security.

In effect, Snow had weaponized this protection to create a denial of service for Orange subscribers.

 

Madory wrote:

 

As demonstrated in our earlier analysis, the internet’s RPKI ROV deployment has reached the point where the propagation of a route is cut in half or more when evaluated as RPKI-invalid. Normally this is desired behavior, but when an RPKI config is intentionally loaded with misconfigured data, it can render address space unreachable, effectively becoming a tool for denial of service.

 

 

Graph showing 40-50% drop in traffic from 14:20 to 18:00 UTC

Kentik

 

Using Kentik’s aggregate NetFlow, we observed the outage (illustrated above) as a large drop of the volume of inbound traffic to Orange España (AS12479) between 14:20 UTC (3:20pm local) and 18:00 UTC (7pm local). However, there were more developments prior to this window of time as well as some lingering effects, which we will dig into in the post below.

After regaining control of its RIPE account, Orange restored service by publishing the proper ROAs. Madory said, however, that many of the bogus origins remain published. “It’s not causing any problems because they published legit ones with the correct origin, but they should just revoke these: https://rpki-validator.ripe.net/ui/149.74.0.0%2F16?validate-bgp=true."

 

Snow’s unfocused and, at times, ineffective tinkering with the Orange routes makes the ultimate objective of the hijacking unclear. The party initially published ROAs that had little or no effect at all. Even when Snow published more destructive ROAs starting at 14:00 UTC, the effect was more limited than it could have been, since it deleted roughly 20 percent of the Orange’s 9,200 routes.

Going “nuclear”

Madory wrote:

 

I suspect, at first, the person was just seeing what they could do in the account. Maybe they assumed that it might be monitored and they would get kicked out after doing some innocuous things. But over time they got more daring until they decided to cause a major outage to get the provider's attention about their poor security practices. I don't think the objective was solely to cause an outage, but I'm really just speculating.

 

It seems, at first, this person was just seeing what they could do without causing harm. Then [they] tried changing the origins of a few routes [to see if it] could potentially cause a little disruption. Then at 14:20, they went nuclear with the two /12's ROAs.

Besides underscoring the continued fragility of BGP, the incident exposes a concerning lack of security hygiene at Orange. For one, an infostealer installed on an employee's computer that went undetected for four months. For another: the use of a weak password and the failure to secure the account with multi-factor authentication to protect an account on a Regional Internet Registry such as RIPE. All are amateur omissions that should never have been possible at an organization with Orange's reach. Also troubling: Madory said that until Wednesday, the Orange RIPE account was never configured to track the creation of new ROAs, a lapse that made route announcements harder to track.

 

“If RPKI wasn’t on Orange España’s radar before, it sure is now,” Madory wrote. “Hopefully this incident can serve as a wake-up call to other service providers that their RIR portal account is mission-critical and needs to be protected by more than a simple password.”