The blog of Xeno, a slightly mad scientist
… The story of automated conflict prediction starts at the Defense Advance Research Projects Agency, known as the Pentagon’s R&D wing. In the 1990s, DARPA wanted to try out software-based approaches to anticipating which governments might collapse in the near future. The CIA was already on the case, with section chiefs from every region filing regular forecasts, but DARPA wanted to see if a computerized approach could do better. They looked at a simple question: will this country’s government face an acute existential threat in the next six months? When CIA analysts were put to the test, they averaged roughly 60 percent accuracy, so DARPA’s new system set the bar at 80 percent, looking at 29 different countries in Asia with populations over half a million. It was dubbed ICEWS, the Integrated Conflict Early Warning System, and it succeeded almost immediately, clearing 80 percent with algorithms built on simple regression analysis.
STATISTICS DON’T HAVE TO WORRY ABOUT HURT FEELINGS
Why was it so easy to beat the CIA’s best analysts? To some extent, the answer has more to do with humans than machines. Imagine the agency’s Indonesia expert, for example. He wants to make accurate predictions, but he’s also subject to a range of biases that never show up in the data. He wants his work to be exciting and relevant, earning the attention of his superiors; he wants Indonesia to be important in the world. Predictions are also used to direct resources within the CIA, and he may want to attract more of the resources than the Indonesia bureau would otherwise receive. By the time all the biases are accounted for, he’s doing only slightly better than a coin flip. The statistics, on the other hand, don’t have to worry about internal politics or hurt feelings.
IT’S A MIRROR OF THE SAME OPEN-VS-CLOSED DEBATE IN SOFTWARE
It’s a lesson that conflict prediction has taken to heart, growing more transparent even as the tools become more powerful. ICEWS itself was reclassified and taken back into the secretive corners of the Pentagon, but the most exciting projects right now are fully open endeavors that publish their predictions for anyone to see. On the data side, researchers at Georgetown University are cataloging every significant political event of the past century into a single database called GDELT, and leaving the whole thing open for public research. Already, projects have used it to map the Syrian civil war and diplomatic gestures between Japan and South Korea, looking at dynamics that had never been mapped before. And then, of course, there’s Ward Lab, releasing a new sheet of predictions every six months and tweaking its algorithms with every development. It’s a mirror of the same open-vs.-closed debate in software — only now, instead of fighting over source code and security audits, it’s a fight over who can see the future the best.
Of course, the secretive predictors are still alive and well. Conflict prediction is a lucrative business for national security consultants, even if it’s harder to check their work. And it’s impossible to know what the Pentagon’s working on. They could be beating Ward’s predictions each month — but even if they were, it wouldn’t give them much of an edge over the publicly available information. And just like the CIA’s analysts, they’re handicapped by closed sources and institutional biases. As long as public predictions are getting more outside insight, more revisions, and more scrutiny, it’s easy to like their odds. “Look, it’s a messy and complicated world,” Ward says. “I don’t think we’ll ever get to a place where things are predicted perfectly.” In the meantime, the trick is to keep getting better. …
I like the idea of human brains building better and better artificial brains to better predict the future.