Now, we’re getting into the nitty-gritty. How did criminals get inside Equifax, and what were all the missteps inside the company that led to the hack? In episode three, we hear from the GAO investigators who actually got inside Equifax and interviewed the principals involved. We find out that it took Equifax 76 days to notice the attack. We also found out that the attack itself was “not sophisticated.” In fact, Equifax made things easy. Once inside, criminals found a text file with usernames and passwords for 51 other databases. There’s plenty of other mistakes, too.
Nevertheless, when CEO Richard Smith testified before Congress, he dished out a lot of the blame to a single worker and “human error.” We tracked down the human error and — in an interview you won’t hear anywhere else — we get his side of the story. Ever wonder what it’s like to get really, really bad news as a security professional? On your birthday? You’ll find out.
Here’s a sampling of the transcript.
—-PARTIAL TRANSCRIPT—
RICK SMITH: I’m here today to say to each and every person affected by this breach, I am truly and deeply sorry for what happened. We know now that this criminal attack was made possible by a combination of human error and technological error.
GRAEME: When he mentioned the cause of the breach was human error, I found it troubling, to be honest. I think it’s really an incredible simplification of the issues and the complexity of managing cyber security and a large organization.
ALIA: Later on in one of these hearings, Smith elaborates on that “human error” –
RICK SMITH: The individual who I just discussed that was responsible for the patching process is no longer with the company.
GRAEME: The day before I had been terminated from the company, so I was able to put two and two together and work out that the person that he was talking about was me.
ALIA: But that “one human error” of course, isn’t the full story –
BOB: Not at all! So many more mistakes.
ALIA: Today, we look at the full story. We sat down with the two guys who went to Equifax, investigated, and wrote the detailed report on *all* of the big and little things that went wrong –
BOB: And we have a conversation with the“human error” himself – Graeme Payne –
ALIA: To get to the bottom of what *really* went wrong at Equifax, to really break down the perfect storm that led to (what some people are calling) the worst breach ever.
{snip}
RICK SMITH: The human error was the individual who was responsible for communicating in the organization to apply the patch… did not.
CHAIRMAN: So does that mean that that individual knew that the software was there and it needed to be patched and did not communicate that to the team that does the patching? Is that the heart of the issue here?
RICK SMITH: That is my understanding, sir.
ALIA: Well we talked to Graeme, the quote human error himself, to see if he thinks that’s actually what went wrong. This is the person who got fired, and then turned on the tv the next day to discover he was being blamed for causing the breach — so does he agree that it was his fault?
GRAEME: I think the committee concluded that it didn’t align with the backdrop of the facts.
{snip}
NICK: The attackers, once they got in the door, were able to identify, usernames and passwords for other databases that were being stored in clear text. That is to say there was a file available that didn’t require any password itself to get to that information.
BOB: A file containing unencrypted credentials – usernames and passwords, completely easy to read.
ALIA: Why would someone store passwords without encrypting them?
NICK: Well, if you think about it, an administrator, assistant administrators’ roles and responsibilities, they’re tapping and they’re accessing a lot of different systems. And so a fast way of doing so is kind of like for those of us that stick Post It notes under our keyboards would be to keep that information stored somewhere so that you could easily access it. Sometimes to just copy and paste it in so that you don’t have to worry about it.
BOB: So our lives are full of all these little conveniences, all these little work arounds. When we do these things, we just never imagine these workarounds in the hands of a hacker.
NICK: Some of these passwords can be very long and complex, um, can be automatically randomly generated and so it. It’s something that I think demonstrates the fact that humans are going to be, they’re going to be fallible themselves.
BOB: This is the absolute Catch 22 of passwords. The longer and more complex your passwords are, the less likely you are to remember them, and the more likely you are to put them in a text file or on a Post It note so that you can get them when you need them. Simpler passwords, easy to remember, complex passwords, hard to remember. Which one is safer? I don’t know.
ALIA: So after waltzing into the dispute resolution portal, these hackers are armed with a bunch of other usernames and passwords –
NICK: And so they were able to then use those to navigate to about 51 other databases or 51 databases total.
ALIA: 51 databases to enter search queries for our useful PII.
NICK: There’s another thing, that I, that I’d mention to, you know, it, it took 9,000 queries. So those are like searches, to tap into these databases, right?
ALIA: Hackers have to search through these databases. These queries are their more sophisticated version of CTRL + F: Social Security Number. Nick’s point is: even if the data exiting the servers wasn’t detected (thanks to the expired certificate) – those queries themselves could have been detected.
NICK: For example, you know, in hindsight you could put a limit on how many queries get done within a certain period of time or let’s say you have queries coming from one source, you know, over and over and over again. there could be ways to restrict that. So that, okay, once they hit 100, okay, there’s something up. That was another area that Equifax informed us that in hindsight they, and I think following that they’re looking at putting some restrictions in place.
{snip}
ALIA: Well we talked to Graeme, the quote human error himself, to see if he thinks that’s actually what went wrong. This is the person who got fired, and then turned on the tv the next day to discover he was being blamed for causing the breach — so does he agree that it was his fault?
GRAEME: I think the committee concluded that it didn’t align with the backdrop of the facts.
BOB: To put it in my words: Bullshit.
ALIA: So what, what exactly like talk to me about this email that I keep reading about.
GRAEME: Right. So this one specific email went to about 430 people. And in my role as uh CIO, I was copied on that was just like all the other CIOs and these things would come out periodically, and I’d look at them and that was sort of the, uh, the end of the story as far as I was concerned.
BOB: But that wasn’t the end of the story – yeah – as far as your company was concerned
GRAEME: Right, right. So I was never under the impression or direction, and there was nothing ever stated in policy that required me to forward those emails to anyone. Um, I, my assumption was that the appropriate people were getting the notifications. I got hundreds of emails a day, so this was just one of many, many emails a day that would come through my inbox.
You know, the company did their investigation and they concluded that, uh, I should have forwarded that email and because I didn’t forward that email because it didn’t get to the people that were actually administering the system that I was the breakdown in the process.
ALIA: And how did that make you feel?
GRAEME: well when I was first terminated, I wasn’t quite sure what email we’re referring to honestly. It was only when I heard the testimony the next day that I put two and two together and worked out that that was the email they were referring to.
ALIA: And what do you make of that? LIke what do you make of the fact that, that, that an email essentially, at least in the way that I’m hearing it, the email was sort of what took everything down.
GRAEME: You know, to me that’s an oversimplification of the complexity of this issue.
ALIA: I like how Graeme put it in his *testimony* in the “House Committee on Oversight and
Government Reform”: “If that’s the process that the company has to rely on, then that’s a
problem.”
ALIA:I don’t know that Graeme wasn’t a human error that contributed to the breach. I do know he’s not the one human error that caused the breach.
I mean that Apache Struts email had 430 people on it. That’s 429 other possible human errors.
BOB: We asked Graeme, what did he think went wrong then? If it shouldn’t just be reduced down to him just not forwarding an email?
ALIA: He pointed us back to what Nick and Michael found in the GAO Report, (and what a House Committee report later categorized) as Equifax’s “specific points of failure” – it breaks down to three problems:
GRAEME: The first was around a lack of accountability and no clear lines of authority in the equifax IT management structure.
ALIA: Problem 1: Management Structure Lacking Accountability and Organization.
BOB: So really when somebody makes an order, how do you make sure that that order is followed through on?
GRAEME: And because of that they concluded that there was a gap in the execution between policy development and, and operation.
ALIA: Problem 2: Gaps Between IT Policy Development and Execution
BOB: So somebody might say here’s a great idea that we move everything to the Cloud but then there’s 100 things that have to happen before you accomplish that objective and those two groups of people weren’t communicating.
ALIA: And Problem 3: Running Critical Systems on Legacy IT (aka Old IT) with Documented Security Risks
GRAEME: The company’s aggressive growth strategy and accumulation of data had resulted in a really complex IT environment. And because of that it just made management of legacy systems and security, um, difficult.
BOB: This happens at all big companies that swallow up little companies. Merging systems is a nightmare and especially when some of those systems you acquire are old or they get old.
ALIA: The report points out that some of these systems were so old, only a few people at Equifax even knew how to operate them.
BOB: Only Mary or Jack in the back are the only ones who can actually fix or update something and if one of them are sick then nobody can.
ALIA: I mean like isn’t Equifax one of the first companies to have digitized credit?
BOB: Yeah you’re right that’s an irony isn’t it? They basically took the credit system and put it on computers and that was their claim to fame and in the end it betrayed them.
ALIA: Each of these points of failure combine to form: the perfect environment for an email to go unnoticed, for a patch to go unpatched, for 145 million people’s most important data to go missing.
ALIA: Do you think that all of this sort of email, this email fiasco demonstrates a lack of accountability that the report talks about?
GRAEME: Yes. I mean this is this execution gap, right? So at equifax we had the security team that reported up to the chief legal officer and then we had the IT organization that I was part of. And the security team had responsibility for managing this global threat and vulnerability process.
ALIA: In other words: the Security team (who manages these vulnerability emails) and the Tech team (who patches these servers) didn’t cross reporting paths. You could make the case that, organizationally, they really aren’t communicating with each other. According to another report I read, this is the result of an inherited problem – apparently a former Chief Security Officer and former Chief Information Officer just didn’t get along, so they built a system where Security and Tech stayed pretty independent of each other.
GRAEME: But because of this, you know, reporting to two different organizations, I think there were definitely some gaps in execution and as the report points out and, that was an example in this case.
BOB: If something goes wrong in the morning, Alia, and you can’t feed Ethel, how do you tell Brandon?
ALIA: Heads up – Brandon is my husband, Ethel is my sweet baby angel dog.
ALIA: I usually text him. But that doesn’t always work because sometimes he’s in a place where he can’t receive a text message. So sometimes I’ll leave like tape on the wall, a little note in tape. I know it sounds weird but it’s not. And sometimes he won’t see it?! I mean we just, we actually don’t have a system. And it’s a problem, we actually just had a conversation about this. How we need a system.
BOB: You actually don’t have a system, and that’s the truth at many many companies about updating patches. I can picture someone putting masking tape on a server saying “applied Apache Struts” or in red “Still needs patch” and the person who’s supposed to read the masking tape being out sick that day, and the next thing you know 100 million social security numbers go flying out the door.
GRAEME: And I’ve seen this a lot in my career that people have developed policies and put them out on their intranet and just expect everything to happen and it just doesn’t work like that.
BOB: There was a detail in a different report, also in front of the House Committee on Oversight and Government Reform — that Equifax’s patch policy basically operated on the “honor system”, and that’s crazy, like that’s insane, that’s hard to imagine.
You can listen to episode one by clicking play below, if that embedded link works for you. If not, click :
here for the Stitcher page
https://www.carbonite.com/podcasts/breach/s02e01-Equifax-data-breach
or
here for our iTunes page
https://itunes.apple.com/us/podcast/breach/id1359920809?mt=2
8 Trackbacks / Pingbacks