When Trust is Not Verified at All
In two earlier posts I showed that science is verified trust, but that the verification is not always well done. What happens when the verification is not done at all? We can see the horrid results in many different areas of life.
War Reporting
By the 1990s, before the scary Internet started to steal their lunch money, it was clear that Anglo journalists were not competent to cover wars. The journalists who could not distinguish between tanks and other armoured fighting vehicles in the 1991 gulf war were just the camel’s nose in the tent. Their competence has not improved since. After the recent armed bank robbery in my city, some reporters went to press calling an IED an integrated explosive device. Twenty years of the wars in Mesopotamia, Syria, and Afghanistan did not teach them this basic term. Many Canadian news reports on the wars in Syria and Iraq in the 2010s were summaries of press releases by one of the foreign militaries fighting there. You could choose between the journalists who paraphrased NATO propaganda, Russian propaganda, and Turkish propaganda (with the occasional column or article which explained why people in northern Iraq were willing to side with the head-choppers over Baghdad). These reports did not give a clear explanation of the course of the war, but did indicate that whoever that foreign military was bombing was BAD and whoever that foreign military was arming was GOOD. I have not even bothered following Canadian or British coverage of the Russo-Ukrainian war when there are Ukrainian sources and independent experts like Oryx or Dr. Phillips O’Brien whose claims have been verified and who correct their own mistakes when they make them.
Punditry
Those independent experts have their own problems. Timothy Snyder has a PhD in history, speaks and reads Russian, and is a specialist in the history of Nazi and Soviet atrocities. But Aarhus University professor Dr. Jeremy Morris, who has spent decades travelling across Russia and building relationships with locals, has serious concerns with what Snyder writes for newspapers:
Tim Snyder’s New York Times piece is a mishmash of historical analogism that focuses on Putin and sidesteps scholarship on Russian society. Snyder claims Russia (what, all Russians?) is fascist because it has a ‘leader cult’; celebrates a ‘cult of the dead’ via Victory Day; and is hostage to a myth of an imperial golden age. Very little of the essay, in fact almost nothing apart from a passing mention of Z people and rallies actually pertains to Russia beyond the Kremlin and some ideologists of questionable relevance (‘not Dugin again!’, said one of my undergraduate students).
All of these things are ‘true’, but they don’t really mean what Snyder says they mean. … At home, Putin has always been an ambivalent figure and never enjoyed unalloyed ‘enthusiasm’, even among his voting constituency. … The guy never had an iota of charisma. He could never build a following like Trump. … Russians marching with placards of their ancestors who fought in the war is mainly not even a patriotic statement. It’s a rare permission to express personal loss, to experience connectedness, and to give voice to frustrated feelings of a need for communal activity. (The current regime in Russia restricts almost all public events, including pro-regime demonstrations which do anything the state did not tell them to do – S.M.)
https://postsocialism.org/2022/05/24/is-russia-fascist/
You could argue back and forth whether Morris’ counter-arguments are strong or empty, but lets look at one of Snyder’s theses.
The idea that Victory Day in Russia is a Neolithic Cult of the Dead comes from Kamil Galeev, a Tartar dissident settled comfortably into US think tanks and onto hellbirdsite (where he has received some very generous gifts). Galeev is entertaining, but its hard to know what to do with his essays because he is not a disinterested observer but an advocate for a specific outcome. He wants Russia to break up so that ethnic minorities such as the Tartars will no longer have their cultures and language suppressed, their wages and resources stolen, and their young men raped and killed in the army. The ethnic Russian population is shrinking, and young Russian men in St. Petersburg and Moscow mostly avoid the draft, but some ethnic minority populations are growing, and poor rural people can’t evade the draft so easily. Galeev writes in an emotive, loosely-cited style which he believes will help achieve that goal. So nothing he writes should be trusted. And because Snyder does not cite the sources for his ideas its difficult for a reader to know that some of his claims come from this excitable, politically motivated source. People who hang out on birdsite tell me that a disturbing number of news articles are paraphrases of Twitter threads which don’t acknowledge the original source (just like a disturbing number are paraphrases of press releases and public statements). Military situation reports also rely heavily on unnamed sources and analysts, and some of those may be randos on social media too.
Textbooks and Trade Books
If we are skeptical of punditry and social media, we might turn to textbooks and trade books by experts. But those are not well verified either. Textbooks usually undergo a brief external review, but are not fully referenced. So its not uncommon to find a university textbook teaching a theory which was refuted 20 years before it was published, just because the textbook author never read that article (or the author is aligned with someone on the losing side). Trade books (books from big publishers marketed to bookstores, as opposed to books from university presses marketed to libraries) usually undergo even less scrutiny before they are printed. One infamous example is how The Better Angels of our Nature cites a woodblock print clearly labelled MARS and SATURN as evidence that life in ‘the middle ages’ was crude and brutal. Any editor with a degree in History or English could have pointed out what Mars means in astrology and that printing is not a typical medieval European technology, but they did not.
University Rankings
We might think that people who sell information would go to the trouble of verifying it. If they make a mistake, their reputation and their income are at risk! And yet statistician Andrew Gelman reports that one university ranking in the United States appears to have accepted a university’s self-reported numbers which are blatantly incorrect (the university ranking agrees that the numbers cannot be confirmed). This is why social scientists (and reformers of military promotion systems and software development pay scales) speak of Goodhart’s Law: if you use a proxy to evaluate people’s performance, that proxy becomes useless as soon as they learn what it is. An old joke in the Soviet magazine Krokodil about a nail factory illustrated how this works.
Peer Review
Even peer review has one flaw. Peer reviewers in many disciplines check the structure of an argument, but don’t check whether the data which supports it is correct. The authors are expected to create the data, the reviewers take it as a premise and just see if it leads to the conclusion. The paper by economists Reinhart and Rogoff which built on a misformatted Excel formula is one infamous example. But a University of Guelph botanist appears to have systematically invented data to support his published articles (he denies this, and an internal university investigation supports him). The institutions of science are not well equipped to handle liars, whether that is convicts talking to the nice psychologist or a professor with wonderful data. And once one sub-field develops a blind spot, it can turn out article after article with the same flaws, peer-reviewed by people who make the same mistake in their own publications (“your comments about the emperor’s supposed lack of sartorial integrity reflect outmoded positivistic assumptions, and in out field we …”).
Science is verified trust, but it treads water over an ocean deeper and darker and filled with more squirming things than any of us can imagine. Without constant effort, it will sink below the surface. That is why it is so important to keep making sure we are doing the work, and not to posture and try to bully anyone who asks awkward questions. We are never going to win by charisma or by saying what people want to hear, but we can win by being calm, factual, and mercilessly fair.
Edit 2022-07-10: for another take on these ideas see Darshana Narayanan, “The Dangerous Populist Science of Yuval Noah Harari,” Current Affairs, 6 July 2022 https://www.currentaffairs.org/2022/07/the-dangerous-populist-science-of-yuval-noah-harari/ {my first impression of Sapiens was not nearly so negative as Narayanan’s, I classified it as a ‘big ideas book which overstates its claims to get the conversation started’ like Gwynne Dyer’s War. The clickbait title also erases the difference between popularizers and populists: Harari does not attack experts or expertise, just repackage other people’s research and ideas in a digestible format}
Edit 2022-08-05: after this post was scheduled, O’Brien started objecting to what seem to be true if imprecise statements about violations of the laws of war (to paraphrase: Ukrainian videos show Ukrainian special forces launching attacks from unmarked civilian vehicles and Russians transporting munitions in marked ambulances, which would give the other side the right to fire on civilian vehicles and ambulances – see this page on International Humanitarian Law if you don’t believe me) https://nitter.it/PhillipsPOBrien/status/1552931706298155010#m So reader beware!
Edit 2023-07-11: for another example of how someone can be reliable in their peer-reviewed books for academic presses, but unreliable as soon as they become famous and publish with a trade press that just cares about sales, see: Kevin Matthews, “Review: The Deluge: The Great War, America, and the Remaking of the Global Order, 1916-1931 by Adam Tooze” https://reviews.history.ac.uk/review/1805 {I have not read the book discussed}
Help pile this blog ever higher with a monthly donation on Patreon or paypal.me or even liberapay
(scheduled 28 June 2022)
[…] do not fit have been hidden or removed to make the idea sound compelling, and partially because its impossible to verify all the things that the argument relies upon. Trying to do this would require a team of reviewers […]