How the Infamous APT 1 Report Exposing China’s PLA Hackers Came to Be

This is the first in a series of pieces I’ll publish that take an in-depth look at significant events, people and cases in security and surveillance from the past. If there’s something you think would make a good topic for this series, reach out to me at countdowntozeroday@gmail.com.
In February 2013, Mandiant, then a company little known outside the security community, published a blockbuster report exposing the Comment Crew aka APT 1 — a prolific state-hacker group from China’s People’s Liberation Army who had been cutting a wide swath through various industries and research organizations for years. Instead of stealing state secrets — though they were doing that, too — APT 1 was nabbing intellectual property from the aerospace industry, pharmaceutical companies and others to bolster China’s own industries and research and development efforts.
Security researchers believed the Chinese government was at least aware of the activity if not directing the operations but were reluctant to publicly accuse the government without solid evidence.
Mandiant changed this with its groundbreaking report, which not only tied the activity directly to a specific PLA unit — Unit 61398 — and to specific IP addresses, networks and buildings in Shanghai where the activity originated, but also asserted that the Chinese government was likely sponsoring the operations. They also identified three specific individuals — Wang Dong (aka UglyGorilla), Mei Qiang (aka SuperHard) and an individual who used the handle “DOTA” — who were part of the operations. The latter underscored the fact that the thieves weren’t a faceless all-powerful army, but real individuals who made foolish errors in their operational security (OPSEC) that allowed them to be traced to real locations, phone numbers and personal social media accounts. A year after the report came out, the US Justice Department announced the indictment of Dong and four other PLA hackers for engaging in cyber operations against US corporations to steal trade secrets for economic gain. It was the first US indictment of nation-state hackers.
Mandiant took a risk in publishing the information. It meant that infrastructure the hackers were using — and researchers were actively monitoring — would get taken down, burning those investigations. And Unit 61398 would learn from its mistakes and become more stealth, making it harder to track them in the future. It also put a target on the company’s back. “We expect reprisals from China as well as an onslaught of criticism [from the security community],” the authors of the report wrote. For this reason, the staffers who wrote it, published the report anonymously. It’s only in recent years, that the lead author and architect of that report, Visi Stark, has begun to talk about his role in it.
Stark, now cofounder of The Vertex Project, an intelligence-analytics technology firm, worked for years as a contractor for the US intelligence community, doing research to uncover software vulnerabilities and develop exploits and other hacking tools for the NSA, CIA, and other parts of the intelligence community to use in their offensive cyber operations. While working as a contractor for Mantech, he and two colleagues created an innovative cyber intelligence and analysis platform called Nucleus that was designed to help security researchers collect and correlate lots of disparate pieces of data in order to map and unmask threat actors and their operations, giving them a holistic view of what was occurring over time. In March 2011, they planned to launch a commercial business around Nucleus to offer insights and services to the US intelligence community, when Travis Reese, a former intel colleague who had become president of Mandiant, suggested they bring their tool and talents to his company instead.
Stark and his colleagues were still steeped in the secrecy of the intelligence community and didn’t want other workers at Mandiant to know what they were doing, and aside from Reese and a few others, most Mandiant staff didn’t even know Stark worked for the company. They created a shell company with a generic name to lease work space away from Mandiant’s offices — they called it The Lair — and set up a computer network with no connection to Mandiant’s network. And instead of announcing their hire internally with their photos and bios, Reese just sent out a picture of three squirrels dressed as Ninjas climbing rope ladders and a message that said essentially: I've hired three squirrels to do secret stuff for the company. if you need to be in contact with them, you’ll know.
Mandiant was quickly becoming one of the top firms to investigate when businesses and governments got hacked, and as Stark and his colleagues plugged data from Mandiant investigations into their platform, then augmented it with open-source investigations of their own, a clear pattern began to emerge that pointed to Unit 61398. Alarmed by the amount of intellectual property the group was stealing — and the fact that the government didn’t seem to be doing anything about it — they decided they had to do something to make the US public aware of the operations and force the government to take action.
The following is an interview with Stark about how the APT 1 report came to be and the unexpected response it generated. The interview has been edited for organization and clarity.
Walk me through the backstory. How did the APT 1 report come to be?
We had already created the initial analysis platform that we were using [Nucleus] … and at the time we were also engaged in what we called the “hops program.” When you were owned by the Chinese at the time, you would never see a Chinese IP address connecting to your network. They were always going through mostly compromised systems. So they would compromise some rando small company that had a vulnerable server facing the internet and then use that as a proxy … for a long time. Like, years.
So we would occasionally go to those organizations and be like, Hey listen. Your server is compromised by the Chinese and [is being] used to compromise other organizations. We’ll build you out a new server if you’ll leave this one in place and let us kit it out.
Not … many private organizations … had these hops programs at the time. It was something that the OSI [Air Force Office of Special Investigations] and FBI were doing, and Mandiant was doing because they had [former] OSI and FBI people [on staff]. So there weren’t many other organizations that were … getting up on compromised boxes and leaving them in place and monitoring them. Because … that’s pretty risky…. Who’s liable when that system gets used to hack some other organization? … Which is why … that hops program was also a pretty closely guarded secret inside of Mandiant at the time.
[The hops program] ended up being super valuable, because in a lot of instances [the hackers] were [transmitting] the [stolen data] through these systems. And so we were watching terabytes of .rar files move through these servers…. We wanted to … put exploit docs into those .rar files [going back to the hackers] … but that got the kibosh [from Mandiant lawyers]. They were like, ehh, that’s not defensible.
Was the stolen data encrypted?
Some of it was, [but] in a lot of instances we had the commands that they had used to actually generate the .rar files.… So we had the passwords…. We were pulling [the stolen files] back and decrypting them and then putting them through document-parsing and analysis and determining … what they were. In some instances we even had the commands that they had used to select [the files], so we knew … what the keywords [were] that they were looking for in documents, or in document titles.
And seeing just the wide variety of things go by … many of which were clearly economic espionage targets and not traditional military spy-vs-spy statecraft kind of things … the sheer volume of it was just staggering to us. We were like, what the fuck could we ever possibly do about this to try to put a dent in it? That … was the beginning of the thinking process that eventually led to the APT 1 report.
Clearly, continuing to just do [incident response] and help people secure systems wasn’t working…. And none of that was producing any action on the side of the US government. That was … part of the reason that I [left government work].… The government is kind of sitting on this trove of knowledge that all this stuff is happening, and they’re just kind of not doing anything about it, [because they don’t want to reveal how they discovered it]. I had definitely been … on the intel side long enough to see a lot of things that should have been able to be used as leverage get buried due to [not wanting to reveal] sources and methods…. But at a certain point, sources and methods are worth burning … when you’re hemorrhaging your national economy out through .rar files….
PLA units … were kind of [engaged in a] Wild West at the time, doing whatever they wanted…. We had actually watched instances where [different] PLA units scanned the same boxes and tried to take them over to use them as hops, and they fought over them…. The de-confliction … that happens on the US side [between the NSA, CIA and other hacking teams] … that just didn’t exist over there. It was like literally everybody do the fuck what they want with no jurisdiction, no lanes.
PLA units … were kind of [engaged in a] Wild West at the time, doing whatever they wanted…. And so part … of our goal was to just make it so that Chinese cyber operations had the same amount of red tape as the US … [and slow own the tempo of their operations].
And so part … of our goal was to just make it so that Chinese cyber operations had the same amount of red tape as the US … to make them do some of the left-hand-right-hand red tape things that we and other western intelligence services have to do. In order to action systems or carry out operations [in the US], you have to do all of this red tape and de-confliction work … and they didn’t have to do that [in China]…. So mostly [we were] trying to get them to play by the same rules as everybody else [and slow own the tempo of their operations].
Your strategy was to embarrass the PLA so the Chinese Communist Party would step in and say because of your bad OPSEC we’re giving you more oversight?
Yeah, 100 percent. That was absolutely part of the goal.
The … sheer scale of it at the time was just stunning. I think at the time we estimated … that APT1 — PLA Unit 61398 — was 30,000 people…. Their only mission [was] economic espionage against English-speaking organizations…. If English was the lingua franca of that business, then they would potentially be a target… That … PLA unit, designated just to economic espionage, was like 30,000 people.
How do you calculate that?
There were … a number of … forums where people who had previously been inside those PLA units had discussed … the sizes of the organizations. PLA Unit 61398 … had their own dedicated daycare facility inside their building. Like that’s the scale [of it]. They had a place where PLA Unit 61398 people could drop off their kids…. A significant bulk of the people were actually involved in analysis and dissemination of data. You’ve got to keep in mind that on the other side of that [stolen data] is just a massive apparatus that is filtering these gigantic .rar files that come back full of everything from soda recipes … to aerospace studies … to pharmaceutical studies on the efficacy of different variants of drugs.
Were there any incidents you tracked that you considered especially egregious?
It was just sort of the net effect of the entire thing. It was watching them go in and just … company after company after company. It was overwhelming…. And so it’s not so much any one individual instance,… it was mostly just the net effect of the volume of it and the lack of specificity and targeting. They were just there to take everything.
Had you already identified some of the individuals behind the operations?
Oh yeah definitely. Like we knew where they dropped their kids off at school. We know which noodle restaurants are their favorite noodle restaurants.
It’s actually harder than everyone thinks to have good OPSEC. It really takes only one or two very small mistakes to make it so that you sort of cross the proverbial streams, right? And that can never be undone…. As an example, these guys were using these … hops… to do their targeting. But then, because that was the way that they could access the internet [without going] through the Great Firewall of China, they were also using it to do other things.… There were things where they had like weather apps installed that … were calling out with their geo-location data to get weather information.
So essentially we started brainstorming on like what could we actually do about this problem.… And one late night, you know, drinking scotch at The Lair… we started joking. Like hahaha, wouldn’t it be funny if we were to just dox these guys?… And then the joke kind of went on for an hour or two, and we sort of realized that we were serious and that … this could actually potentially have an impact on the problem.
One late night ... drinking scotch at The Lair… we started joking. Like hahaha, wouldn’t it be funny if we were to just dox these guys?… And then ... we sort of realized that we were serious
[So] we … proposed this — we called it Project Nightmare — to Kevin [Mandiant-founder Kevin Mandia] and Travis. And they both looked at me like I had two heads. They were like, this is the opposite of what intel teams normally do…. [Travis] sort of looked at my compatriots and was like, Has Visi gone off the deep end? And they were like, No, we think this is a good idea actually. [But Kevin and Travis] literally said no we will not do this. We can’t do this.
What eventually changed their mind was, there was [a rumor] that there was going to be a big … joint DHS-FBI disclosure, that was going to burn all of this APT 1 infrastructure [that the PLA hackers were using] to the ground…. The government was going to publish a big expose that was going to focus on APT 1 or APT-1 related actors.
And [this] changed the calculus for us. The fact that we were going to potentially have this [APT1] infrastructure that we had invested time and energy in … be disclosed [and burned down]…. When [Kevin and Travis] heard that was going to happen, they were like, well that sort of changes things, cause we might lose this hop infrastructure anyway. So like, sure Visi, go ahead and do this. What do you need?
Do you remember that scene from “Leon: The Professional” where the dude’s like, “Bring me everyone.... Eeeveryone!” At the time, there was like the very beginning … of an intel team … forming inside Mandiant…. And so we basically put everybody [from that team] on a plane and flew everybody into town and brought everybody into The Lair. We were sleeping in The Lair and at a hotel across the street for the better part of two weeks…. We [already] had all the data, we had all the story. So from the “yes, we’re going to do this” to having a first draft of the gigantic report was only two weeks.
Part of the reason that we ended up working so hard was … I wanted to make sure that [the report] got … released during Chinese New Year … as a sort of mild Fuck you [to China]. Because [the PLA] always [hack] a bunch of shit during Christmas [and ruin our holidays]…. I wanted to make sure it happened during Chinese New Year so that we could see who got called back [to the office to deal with the fallout]….
Did you get insight into who got called back?
With some of the infrastructure we were monitoring at the time, we definitely saw that … tear down [of] infrastructure, so there was definitely a little of that. As far as the SIGINT side of being able to capture the actual calling so-and-so to come home, or sending messages out to individual PLA officers to come back, we didn’t capture any of that. But I’m hoping there were useful tidbits that came out of that [for] the government.
What kind of guardrails did you have about what went in the report and what didn’t?
It has to pass the "Who cares?” test, [and] storytelling is [also] important. One of the things that traditional intel analysts don’t necessarily do a good job of conveying [is] the emphasis on the things that are important. Because they want to leave the decision about what’s important to the decider. But ultimately a good analyst is supposed to be making recommendations … supposed to be using their judgment to interpret the data. So … I wanted to have like a story arc in mind … which is: these people are absolutely pillaging the US economy, and it’s an unfair fight because of nation-state vs private industry…. [We also wanted to include] the details about those [specific PLA hackers]…. [Instead of] it being some nebulous shape that’s attacking you … it’s this guy right here. Here’s a picture of him, here’s his address, here’s his gamer handle. Those are the things that make it real.
Why did you focus on the specific hackers that you highlighted in the report? Were they just the ones for whom you had the clearest data?
They were definitely some of the ones that we had the easiest time getting kind of the highest resolution data on. But at the same time … Ugly Gorilla was sort of one of the first…. We watched him transition personally-owned infrastructure into PLA [infrastructure] early on … domains that had kind of gone from [being] his personal registration to being used for PLA…. But it was mostly a lot of those OG players that had been there mostly since the beginning and things like that.
Was Kevin involved during these two weeks that you were assembling the report?
No. He got basically his first chance to review it when we were at final draft…. We had told him what we were going to do … but it sort of didn’t crystallize…. [When he saw the report] he was shocked. He was like, “Visi, they’re going to be mad.” And I’m like, “Yeah, Kevin, that’s kind of the point.” And he’s like “No, Visi. They’re going to be fucking mad.” And I’m like “Well, we’ve got to decide either to do it or not. Batten down the hatches or not.” So they sent out this big message to like the [IT security team] within Mandiant: “Something is coming, so get ready. Because we’re about to poke the bear.” He even kind of made an argument to try to pull the plug at the last second. He was super concerned.
What was his argument — that you were going to put a target on the company’s back?
There was definitely that. But also … what are customers going to think about this? Are you going to damage our incident response business by doing this?… There were also concerns over what’s the government reaction going to be.
Was there concern about retribution from China?
There were definitely concerns. [But] at the time, it was my assessment that China doesn’t … really do … like the Russians do or like the [drug] cartels do, where they’ll actually send someone to assassinate people. So … we determined that the risk [of physical harm] was pretty low….
Did you contact the government beforehand to let them know what you were going to do?
Right before it actually released, we did give them a draft copy of the final document. Just to be able to say this is coming out in like two days. [W]e gave advance copies to NSA and FBI…. Nothing directly to the White House, but both of those organizations report to the White House….
It would seem to me that the National Security Council would have wanted to know about this.
We thought that it would be better to ask for forgiveness than permission.
Did the NSA or FBI ask you to change anything?
They asked us to omit like one indicator that they still had [some investigation] stuff going on…. It was an omission that, in my opinion … was benign.
How did your marketing team react to the plan when you told them about it?
The Mandiant marketing team had no idea what was coming…. I remember I was having a discussion with the director of marketing at the time and him being like … Visi, we need to release it on a Thursday or something like 10 am to catch the news cycle. And I’m like Man, you do not understand. You can whisper this shit at 2 am on Sunday on the internet, and the internet is still going to go ape-shit about this.
What was the response to the report? You acknowledged in the report that people were going to be pissed off if, as a result of the report, the PLA took down infrastructure that people in the security community were actively monitoring.
By far the reactions were overwhelmingly positive on both private and public sector…. The private sector was sort of just like Yay!
We didn’t realize quite the impact [the report] was going to have. I knew we were going to piss off the American public — hopefully — and get them to … feel something about this problem, and hopefully get the Chinese just to at least put a damper on the ops tempo. But what I didn’t realize is that we were going to set off the [cyber threat intelligence] arms race.
There had already been some CTI stuff around the Aurora hack that targeted Google.
A little. But it wasn’t something where everybody was clamoring to be involved in it and knew what to do and … what the shape of results should look like and that sort of thing. It sort of became a much more understood discipline afterwards.
And how did the government react to your report?
Government, I thought, would have a much more negative reaction, because they tend to be … a lot more conservative on things like sources and methods [and] burning infrastructure…. Travis got requested by … the National Geospatial-Intelligence Agency (NGA), who were like, hey, can you come talk to us about the APT 1 report? … And I’m like no, why would I want to go talk to them? And so he was like no thank you, we’re busy.
They approached him again … and he [asks me]: We still don’t have any reason to go talk to them? And I was like no, I see no reason to go talk to them. By the third time [they asked], he was like Ok, Visi, I’m going to go talk to them even if you don’t want to, just cause like it’s almost getting rude at this point to tell them no…. [We were] just expecting it to be a small meet with a couple of people, … but [the NGA] put out an interagency memo [announcing] that they were going to be doing this thing [with the authors of the APT 1 report].
And we get there and it was pretty surreal…. They had [video-conferenced] in like 30 remote locations, including internationally. So there were people both who had flown in from other government organizations outside of the US, and [there were others] being [video-conferenced] in from remote locations in this gigantic auditorium.
So we talked through … our sources, methods and analysis…. [We] hadn’t offered Q&A because I was still super concerned about what the government feedback was going to look like. [But when we finished the panel], they were like ok, now we’re going to … do a Q&A. I literally covered my mic and I’m like, We’re not here to do a Q&A. And they were like, Well, … we kind of told them that you would. And so … like half of the auditorium gets up and stands in a line that goes all the way up the auditorium … around the back, and starts looping…. And I’m like holy shit. What is going on here? What is it that all of these people want to yell at me about?
But … almost every … person that stood up to ask a question wasn’t there to ask a question. They basically said, I am here on behalf of Australian NSA or US NSA, or whatever, and I want to say that this is the most high-impact thing that anyone has ever done about this problem ever.
So it was like extremely complimentary and extremely kind of moving to me to have all of these organizations and all of these people that came from different inter-agency collaboration groups … all want to stand up and say bravo.
How much of China’s APT 1 infrastructure came down after the report was published?
A lot of it. Some of it they just straight abandoned in place. But a lot of it they did try to sort of clean up. But … these were boxes in a lot of instances that they had been consecutively using for years. So like how are they going to clean up all of the things on those boxes that they’ve been using for years? But in a few instances they did actually go in and remove tools … and take down [command and control servers] and such.
And how long did it take for them to re-establish new infrastructure?
That kind of depends on who you ask…. It’s my understanding that they issued sort of a general stand down of cyber operations PLA-wide for a little while…. [That’s] Not to say that they didn’t re-organize and come back…. [But] whether or not APT 1 came back as a force of its own is a bit debatable, because they sort of got mostly re-orged out of existence, and all those people took tools and infrastructure in some instances with them to their new jobs…. We’re fairly certain that people and tools just got moved from one PLA unit to a different PLA unit that was maybe less of a lightning rod [than APT 1].
But [the hacking has] never quite reached that same crescendo of just unadulterated brazen intellectual property theft that it was at the time…. They’re still brazen, but ultimately I think that they’re a little bit more cautious [now]; they’re a little bit more targeted. They make sure that there’s some high-value interest in the thing that they’re doing, rather than just culling everything from everywhere…. There’s always debate over whether or not it’s had a lasting impact, but I think that it [did] give those [offensive organizations in China] a little bit of additional oversight, a little bit of additional red tape that they have to cut through and things like that.
See also:
Former NSA Hacker Describes Being Recruited for UAE Spy Program
What It Means that the U.S. Is Conducting Offensive Cyber Operations Against Russia