Documents Stamped “Top Secret” Are Not Always Accurate

None of us have a magic crystal ball for seeing the future in Ukraine.

There is something magical about reading a document marked “Top Secret.”  I remember the exact day when I first did so – January 21, 2009. That was my first day of work at the National Security Council as a Special Assistant to the President and Senior Director for Russian and Eurasian Affairs. It felt like I was entering a brand-new world of superior facts and “better” knowledge than what I had previously been exposed to in my several decades in the unclassified world. I remember the first time one of my NSC colleagues asked me to look for something on the “high side.” “High side” was the computer system where you could find TOPSECRET/SCI emails, documents, and attachments. And the “low side” was where lesser mortals found their information. I also remember the day when I read a super-secret “compartmentalized” document. I had to go to a special room to do that. My office at the National Security Council was a “SCIF,” a Sensitive Compartmented Information Facility. But even my SCIF was not secure enough for this incredibly sensitive intelligence. And before being “read in” to this compartment, I had to sign a government version of a non-disclosure agreement. I did not read this super-secret document on a computer but in hard copy. And it was really interesting! Of course, I can’t tell you about it.

But in parallel to gaining all sorts of knowledge from reading “Top Secret” documents, there is a distorting effect that comes with it. In government, a document marked “Top Secret” is assumed to be special, possessing better and more nuanced information than what is available in the public domain. And often this is true. But sometimes it is not true, especially when assessing predictions about complex events in the future. During my time in government, I found secret intelligence to be most interesting and useful when it came to spelling out concrete facts that could not be found in the unclassified world. I found assessments about big, complicated events less useful. Classified predictions about complicated phenomena such as revolutions, coups, or wars were often no better than what specialists in the unclassified world wrote. In fact, professors I knew often provided just as useful of an assessment, even when working with inferior, unclassified data. Predicting the future is equally hard both for the CIA and for the Stanford political science department.        

It is with this context that we should read the latest U.S. classified predictions, tragically and illegally released to the public via a Discord channel, concerning the next phases of the war in Ukraine. Out of approximately 100 documents that leaked, a good chunk of them had to do with Ukraine. Some of these documents appeared to have been altered, adding even more confusion to how to read them.  But even with this caveat in mind, they do reveal a lot about U.S. assessments of the war.  

Those documents providing facts are most revealing (and rather depressing). For instance, we learned, as  The Washington Post reported, that  Ukraine’s “ability to provide medium-range air defense to protect the [front lines] will be completely reduced by May 23. UKR assessed to withstand 2-3 more wave strikes.” This, as the intelligence reveals, is because (1) Ukraine uses its ammunition at a much higher rate than its benefactors produce it, and (2) despite Western support, Russian capabilities at times remain unmatched.

The leaked documents also claimed that the death toll numbers “are around 10 times bigger than any public casualty figures published by either Moscow or Kyiv,” totaling as many as 354,000 killed or wounded from both Ukrainian and Russian sides. The leaks estimate Russia’s losses to be at 35,500-43,000 killed in battle and 154,180 wounded and Ukraine’s losses at 15,500-17.500 killed and 109,000-113,500 wounded. For comparison, data published by the  Russian Ministry of Defense from September 21, 2022, suggests that the Russian army suffered a total of 5,937 fatal casualties, whereas the Ukrainian army counted 61,207 fatal and 49,368 non-lethal losses. In turn, Ukraine’s Ministry of Defense states that Russian losses amount to 183,750 people as of April 19, 2023, while Ukrainian losses are not disclosed on the website. These numbers, however, are intelligence estimates; we do not know the sources for these data. They should be treated less definitively than the facts about ammunition supplies.

But the most speculative “data” revealed from this leak concerns intelligence assessments about Ukraine’s 2023 spring counteroffensive, which is estimated to result in “modest territorial gains,” falling short of Kyiv’s initial aims, according to The Washington Post reporting. National Intelligence Council’s classified assessment similarly suggests that “Ukraine is unlikely to recapture as much territory as it did last fall.” A stalemate is the most likely outcome.  

I have no doubt that those who wrote these assessments are doing the best work they can with the data at their disposal. While working in the U.S. government for five years, I developed a deep appreciation for the work of our analysts in the intelligence community. They are real experts, and I learned a lot from them. But when predicting outcomes of wars, they are still just guessing. They have no crystal ball to see into the future. Coups, wars, revolutions, and things of such nature, in general, are extremely hard to predict. On top of that, some variables – like “will to fight” or “strategy” – are very difficult to measure leading into a battle. So maybe the spring offensive will indeed produce only marginal territorial gains for Ukraine. But maybe not. We are all guessing.

During my time in government, I also noticed a bias for equilibrium or continuity predictions. For instance, if you were an analyst of Egyptian politics and every day for decades you predicted that autocracy would prevail, you would have been right 99% of the time. But during the Arab Spring in 2011, huge anti-governmental protests mobilized quickly and spontaneously in Egypt, overthrowing Egypt’s dictator Muhammad Hosni Mubarak. Bias for continuity would have had a distorting effect on your ability to predict this revolution. The same is true in war. If there was a stalemate yesterday, it is logical to predict a continued stalemate tomorrow. By definition, anticipating unexpected events is very hard, which leads many to avoid even thinking through such possibilities.

Moreover, as we know from history, intelligence assessments can be wrong; it’s hard, inexact work. In the war in Ukraine - or to be more precise, Russia’s barbaric full-scale invasion of Ukraine - we already have good reason to believe that our assessments about Russian military superiority and Ukrainian military weakness at the onset of the invasion were inaccurate. Many predicted that Kyiv would fall in a matter of days. As evidence of such U.S. intelligence assessment back then, the United States pulled out all of the embassy personnel (including, I’m guessing, our intelligence officers working at the embassy at the time), and destroyed lots of embassy equipment, among other things, because, it seems to me – and I obviously don’t know for sure – we didn’t think we were coming back any time soon. We thought Russians were going to occupy Kyiv, and we didn’t want them to access our embassy stuff. We got that wrong.

Of course, I don’t have access to the classified information. But from reading the press, it also appears that we may have underestimated the Ukrainian Armed Forces’ abilities to push the Russians out of the territories near Kharkiv last fall. It was an unexpected, lighting, and highly-successful operation, which allowed Ukraine to recapture more than a third of the occupied Kharkiv region. This is over 2,400 square km (926 square miles) and 502 settlements. As for the Ukrainian military campaign to push Russian armed forces out of  Kherson, it seems like our assessments of that counteroffensive as a long, hard, and costly fight were right.

I am hoping that this upcoming Ukrainian spring and summer counteroffensive will be more surprising like the successful Kharkiv counteroffensive and less like the Kherson one. Judging from leaked documents, it appears that the U.S. government has a different assessment. But, at the end of the day, we – both on the inside and outside of the intelligence community – are just guessing. Just because something is stamped “Top Secret” doesn’t always make it right.