What Went Wrong At Experian (And How It Could At Lots Of Other Places, Too)

As almost everyone knows, the credit rating firm Experian closed off last week with every business’ least favorite headline: They’d been breached to the tune of 15 million T-Mobile customers’ account data going right out the virtual door.

In a blog post on the first day of October, Experian confirmed that the personal information of some 15 million T-Mobile customers (or those who were applying to become customers) was plucked from its servers by hackers.

“On Sept. 15, 2015, Experian discovered an unauthorized party accessed T-Mobile data housed in an Experian server,” the firm wrote. “Experian’s consumer credit database was not accessed in this incident, and no payment card or banking information was obtained.”

As has been the case with many of 2015’s flashier breaches, the “no payment card data taken” statement is something of a cybersecurity booby prize, since what did manage to end up in the hands of hackers is not something anyone wants going up for sale on the Dark Web. According to Experian, that data includes “name, address, Social Security number, date of birth, identification number (typically a driver’s license, military ID or passport number) and additional information used in T-Mobile’s own credit assessment.”

Makes one sort of wish that they had “just” taken credit card data.

Now, reading this, you may have found yourself experiencing déjà vu. Perhaps you wrote the feeling off to the fact that this latest Experian breach matches up with the MO of several other breaches that have happened this year already. One could also conceivably dismiss the nagging feeling that this story was a rerun given the legion of other breach stories this week — Trump Hotels got hit, so did LoopPay (but not Samsung) and a Lyft executive might have hacked arch-nemesis Uber.

But the déjà vu, as Brian Krebs points out this week, is because this isn’t the first time this has happened to Experian. It’s the second time this exact thing has happened to the firm.

“This actually wasn’t the first time that a hacking incident at Experian exposed sensitive T-Mobile customer data,” Krebs wrote, “and that previous breach may hold important clues about what went wrong more recently.”

So, why did last week see a little bit of history repeating, you might ask?

Adventures In Cybercrime Part I

As 2013 was drifting into 2014 and the breach that was on everyone’s mind was Target’s, T-Mobile released the news that a “relatively small” number of customers had their personal data exposed when servers owned by Experian were breached.

At the time, the culprit was identified as Decisioning Solutions, an identity-proofing and authentication company that Experian had recently acquired.

Decisioning Solutions, ironically enough, seems to have been more a decision to acquire a problem.

Decisioning Solutions became part of Experian’s Decision Analytics platform, which is designed to offer credit and non-credit data, customer data and fraud detection to Experian’s various clients — lenders, cable companies, mobile providers, debt collectors, utilities and various levels of government.

The culprit then was that the platform that issued support tickets did so in a wholly insecure fashion. And, if Krebs’ inside sources are correct, this latest one as well.

“KrebsOnSecurity was contacted by an anonymous source who sent this author a Web link that, when clicked, opened up a support ticket within that Decision Analytics platform in the United Kingdom — with absolutely no authentication needed,” Krebs wrote. “That support ticket I viewed appears to have been filed by someone in an office cube at Experian’s data center in Costa Rica who was requesting hardware support for a component of the company’s Global Technology Services division.”

It seems that most, if not all, of the various support requests — requests filled with important information about employee names, addresses, network shares, IP addresses, LanIDs — were more or less wholly exposed to the open Internet with no authentication required at all. Even better…

“The support site also apparently allowed anyone to file support tickets, potentially making it easy for clever attackers who’d studied the exposed support tickets to fabricate a request for access to Experian resources or accounts on the system.”

And, of course, various experts Krebs spoke to noted that the site also allowed anyone to upload any file attachment of any file type — capabilities that attract cybercriminals like sugar water attracts flies.

All in, Krebs noted that the entirety of the panel of experts he interviewed noted the set-up was so insecure as to be “asking for trouble.”

Experian has since (as in earlier this week — not two years ago when it first made trouble) deactivated that portal.

“We take any unauthorized access to our systems very seriously, and when we detected the unauthorized activities, we shut down the website and notified law enforcement,” the company said in a statement. “Our credit database and core infrastructure were not impacted — nor could they be accessed through this website. This site was a legacy version of a service to enable clients and internal users to create and log tickets for issues they may have, and we had already deployed its replacement solution.”

Adventures In Cybercrime Part II

And while Decisioning Solutions was bad insofar as it was the weak point behind not one but two breaches, they are not, in fact, the worst security-related acquisition Experian has made in recent years.

That crown goes to Court Ventures, Inc., a firm Experian grabbed in 2012 that specialized in the sale of aggregated and repackaged public record data from more than 1,400 state and county sources.

“This acquisition strengthens Experian’s consumer data assets in North America and is a further step in Experian’s strategy to extend its global lead in credit information and analytics,” the company says on its website.

Or it might have, except that when the company acquired Court Ventures, it also acquired its client list, including “human Trojan horse” Hieu Minh Ngo. Ngo told Court he was a private detective — they didn’t look into to it too closely — and thus never found out that he was actually a data broker who catered to identity thieves. He had about 1,000 customers when Experian bought Court Ventures.

Once Experian bought out Court Ventures, however, Ngo saw his business boom, since he suddenly had access to records on about 200 million Americans. That figure comes from documents filed by the U.S. prosecutors who successfully tried and jailed Ngo earlier this year.

Experian’s answer to the U.S. Congress — who called them in to question just how exactly the acquisition vetting process failed to turn up the face of a professional black market data broker — was that it didn’t have much in the way of direct answers.

“There’s been no allegation that any harm has come, thankfully, in this scam,” said Tony Hadley, Experian’s senior vice president of government affairs and public policy, testifying in front of the Senate Commerce Committee in Dec. 2013.

What Went Wrong

Experian is not exactly a new or naive player at the table in data security, nor was it a company that had a reputation for being particularly behind the times. In fact, according to employees and former employees Krebs spoke to, for a while, Experian was doing everything right. Then, one personnel change changed everything.

“All interviewed directly attributed that progress to the leadership of then-Chief Information Officer (CIO) John Finch, who helped hire and build up a staff of nearly 30 talented professionals to monitor Experian’s security ‘brain’ — the ‘security operations center,’ or SOC for short.”

The SOC had a big vision: It was meant to be a real-time defense against cyberattacks, as well as a tool to assess and fix vulnerabilities before they turned into breaches. But Finch left Experian to be the CIO at the Bank of England, and with his departure, the department floundered. The best talent left, morale suffered and the team fell from 30 to about 12.

“I don’t have any ill will toward the company, but what happened there was just a culmination of wrong decisions made outside of the security team’s control,” said one of Experian’s former security employees.

And that is a shame, those employees said, because the team feels they almost accomplished something great.

“We had a period of time there where security was viewed in a positive light, and things weren’t being swept under the rug for the sake of uptime,” an employee said. “He left, and it kind of went the opposite direction. Once the leadership changed, the focus changed to controlling costs and not taking systems down for maintenance, and investments started disappearing from a lot of areas. We were in the middle of putting into operation certain tools to do next-generation detection of [cyber]threats, but we weren’t able to get many of them out into production. And that’s how Experian wound up where they are now.”

Why It Could Go Wrong For Anyone

The other emerging consensus that appeared in Krebs’ conversations is that it wasn’t just the focus that left when Finch did, it was also just the understanding of what security is and how it works best.

“When I was there, the board was very big on security and wanting to invest in it and make sure we were doing what we needed to do in order to avoid situations just like this,” a source told Krebs. “In my opinion, there’s no way the board was told the whole story, because if they had been, then things wouldn’t be where they are are now. We wouldn’t be talking about this. Some things had to have been hidden or spun in a way to look positive somehow.”

Jasun Tate, on the other hand, offered a different view. He thinks the issue may not have been hiding information so much as an issue around a focus gone totally off the rails.

“What the board of directors at Experian wanted security-wise and the security capabilities on the ground were two completely different things,” Tate said. “Senior leadership there said they were pursuing a very aggressive growth-by-acquisition campaign. The acquisition team would have a very strict protocol on how they assess whether a business may be viable to buy, but the subsequent integration of the business into our core security architecture was just a black box of magic in terms of how it was to be implemented. And I’m not saying successful magic at all.”

Because security is not magic, and as it turns out, the “black box” is a much bigger help for the black hats than the white hats.

Experian will soon be before Congress again explaining how yet another breach of T-Mobile data happened. But even those who won’t be appearing in front of Congress next week should come out of this with some questions, especially if securing data is important to what you do.

Main among them: Do you believe in your security platform because you believe in your tech or because you believe in magic?