> And if you can't cite two papers that serve as paragons or exemplars of the vast literature, it means that the knowledge contained in that vast literature must be very diffuse and sparse. Which means it has a high likelihood of being a mud moat.
For the counterfactual idiocy in me, I can't help but think of the recent ATIS article on messing around with bad ideas or sparse literature (not as a good foundation but as greenfield research) https://atis.substack.com/p/embrace-interesting-ideas
If we were to try and "do our own (fun) research" are there similar rules that can be applied (other than avoiding comspiracies)?
"Voila - a peer-reviewed literature chock full of misinformation. In practice, I doubt anyone ever does this intentionally. It takes too much coordination and long-term planning."
And yet that is what the Kochs have funded over 50 years or so of libertarian economics propaganda. Pretty much all of Austrian economics, Public Choice economics and large amounts of Chicago economics have been funded that way. And there's a wealth of critiques of their basic reasoning that has been emerging recently, such as Donut Economics and Ha-Joon Chang's books. Most philosophy, libertarian and otherwise, works similarly.
I'm not sure that would work with every field. If suddenly string theory became a political battleground, how much would citing "String theory dynamics in various dimensions" and "The Large N limit of superconformal field theories and supergravity" actually help you to understand if string theory is a worthwhile research subject ?
String theory is obviously a pretty extreme example, but econ can get very math heavy, very fast and I personally would not be comfortable judging whole economic theories by reading two recommended papers.
It's a good point and a useful tactic. One risk is that the person who is giving you the two articles may be part of a subgroup within the supporters of the ideas in question which is particularly bad.
A while back I agreed to debate an Austrian and asked him in advance for an account of the position he would be defending. He pointed me at a book by Rothbard. It was clear from that that the views in question were wrong [http://www.daviddfriedman.com/Ideas%20I/Economics/Critique%20of%20a%20Version%20of%20Austrian%20Economics.pdf], but I also concluded that Rothbardians were only one faction among Austrians and I am still not sure if the movement as a whole has serious problems.
One other approach to your problem that I have been thinking about is looking for work that you can audit. I recently did a back of the envelope calculation of the increase in the amount of useful land from climate change shifting temperature contours towards the poles. I then looked at a couple of published articles on the subject. They were much more sophisticated than mine, looking at things like soil quality and humidity as well as temperature, which was the only variable I considered. One got a result similar to mine, a huge increase, one concluded that the amount of arable land might go up or down a little.
Their procedure is much better than mine but mine had one large advantage — it was simple enough so a reader willing to spend a few hours could check it, see what I was doing and that I did it honestly and competently. There is no way anyone outside the field could do the equivalent for either of those papers, and it would take more than a few hours even for someone in the field. Given both the fact that the issue is politically loaded and that the two professional papers disagreed so much, the fact that I can't check them for myself is a strong argument in favor of looking at simpler models like mine instead of sophisticated ones.
I think the principle generalizes. Try to find work with an important result that you can check. If it turns out to be bogus — I'm thinking of one example in the climate field — and other people in the field haven't pointed it out, that tells you something useful about the quality of work in the field.
It works for people too. You wrote things about Adam Smith, trying to represent him as a modern progressive rather than the 18th c radical that he was, that could not have been written by an honest man who had read the texts he was quoting. [http://daviddfriedman.blogspot.com/2021/07/noah-smith-on-adam-smith.html] That sharply reduced my interest in reading other things you wrote on other subjects, although in the case of this post I thought you were making a good point.
"Suppose you and your friends wanted to push a weak argument for political purposes. You could all write a bunch of papers about it, with abstracts and numbered sections and bibliographies and everything. You could cite each other's papers. If you wanted to, you could even create a journal, and have a peer review system where you give positive reviews to each other's B.S. papers."
Climate change?
To be fair, it appears to me as an outsider to academe, that this is always a risk. Academics have always to try to build a reputation, which requires publishing. Academics will try to establish their own ideas (and hence their own reputations) as obviously dominant in the field, leading publications and reviewers to summarily reject competing ideas. If you add political implications and support for political agendas, you greatly magnify the effect.
A couple of non-political examples I can think of: Einstein and relativity were initially rejected, even though competing physicists were working on very similar theories and furiously working to flesh out the details in order to be the first to publish. Plate tectonics (continental drift) was initially rejected because it seemed so different from then-contemporary understandings of the nature of the earth. I suppose these are both examples of science working as it should: revolutionary theory should require huge amounts of support before it can displace current understanding.
But, in today's frenetic academy, there are more researchers than ever before, dividing knowledge into ever smaller sub-specialties, and lots of new dominant narratives to establish.
Fun post. I've recently been wading through the vast "brain games" literature (can playing games improve your brain), and this rule works. On the cognitive side (do they make you smarter), I could not find two articles worth recommending. On the emotional side (can they make you feel better" I could name two - including one where games outperformed anti-depressant drugs. Probably says more about drugs than brain games.
Can you site two successful used of this strategy. I like the concept and I want it to be true... Or a bit probable. What a good compendium that would be. "The 2 Paper Library of Usefulness". My approach ... Is more like "The 2 book rule". I would give a new science or school of thought 2 books to explain the ideas and usefulness. Short books I hope. Like Toynbee, if the ideas are solid, they can be condensed. This does remind me of the Google Page index approach. Ah, IF only citations did not suffer from manipulation and sycophants.
Ties back to your research subpoint on your recent post about China. There's good research being conducted there, no doubt, but also a fair amount of the low-quality, insular garbage you describe. (There's, of course, garbage publications from all countries as well, but the sense is that it's more systematic in China.)
This is a recent medical article about research publication in China that's made a bit of a splash due to a very alarming claim. I'm still not sure how much to attribute to poorly written or edited publications and how much to attribute to malicious practices or shoddy medical care (including incorrectly performed apnea tests), but none of these option is a good defense of the other. Either way, it does show the difficulty of assessing the quality of literature when there are so many unknown variables.
Your two paper rule makes a lot of sense. Back in the days of the Partisan Review there used to be a lot of respect for public intellectuals. If that respect is gone, it should return. If people cannot present a good case in layman's terms for a theory, there is good reason to suspect that they are hiding behind jargon and credentials.
As a defensive rule, I don't know a better one, but there's one weakness here. It's entirely possible that a credible and relevant field has two "foundational" papers that contain a lot of weak thought. They may have introduced new ideas, but done it poorly, but if a group of academics are unwilling to replace those papers despite their flaws. You'd then end up with a situation in which certain key ideas are introduced but poorly defended in the "foundational" papers, and the good defense is diffused throughout.
Of course, an inability to take one of the possible fixes to this situation indicates a weakness in the field. The original authors could replace their flawed papers with better ones. The field could recognize the need to fix this problem trumps the academic credit system. But despite that, I know this can persist a long time.
A real-life example is Philosophy. Socrates had a great mind and did say many interesting things, but there are many flaws as well. I've personally engaged with philosophical groups that refused to move past that and only wanted to engage with the "vast literature" of ancient Greek philosophers, to the exclusion of all else.
You've written a lot about American suburbs so this piece I discovered (love substack for that very reason), well let me quote part of it.
The mind loves a mystery. And there’s a mystery to be solved in California. Why does a state with purportedly progressive values have such regressive outcomes?
The national hypothesis is to view things through a partisan lens: for left-leaners, Dem = good, GOP = bad.
But in California, that tribal shortcut doesn’t get us any closer to the answer.
At Modern Power we’re early on the journey to solve this mystery. So far we’ve discovered three noteworthy barriers to progressive outcomes: Narrow Interests, Progressive Proceduralism, and Anti-Statism.
> And if you can't cite two papers that serve as paragons or exemplars of the vast literature, it means that the knowledge contained in that vast literature must be very diffuse and sparse. Which means it has a high likelihood of being a mud moat.
For the counterfactual idiocy in me, I can't help but think of the recent ATIS article on messing around with bad ideas or sparse literature (not as a good foundation but as greenfield research) https://atis.substack.com/p/embrace-interesting-ideas
If we were to try and "do our own (fun) research" are there similar rules that can be applied (other than avoiding comspiracies)?
"Voila - a peer-reviewed literature chock full of misinformation. In practice, I doubt anyone ever does this intentionally. It takes too much coordination and long-term planning."
And yet that is what the Kochs have funded over 50 years or so of libertarian economics propaganda. Pretty much all of Austrian economics, Public Choice economics and large amounts of Chicago economics have been funded that way. And there's a wealth of critiques of their basic reasoning that has been emerging recently, such as Donut Economics and Ha-Joon Chang's books. Most philosophy, libertarian and otherwise, works similarly.
I'm not sure that would work with every field. If suddenly string theory became a political battleground, how much would citing "String theory dynamics in various dimensions" and "The Large N limit of superconformal field theories and supergravity" actually help you to understand if string theory is a worthwhile research subject ?
String theory is obviously a pretty extreme example, but econ can get very math heavy, very fast and I personally would not be comfortable judging whole economic theories by reading two recommended papers.
It's a good point and a useful tactic. One risk is that the person who is giving you the two articles may be part of a subgroup within the supporters of the ideas in question which is particularly bad.
A while back I agreed to debate an Austrian and asked him in advance for an account of the position he would be defending. He pointed me at a book by Rothbard. It was clear from that that the views in question were wrong [http://www.daviddfriedman.com/Ideas%20I/Economics/Critique%20of%20a%20Version%20of%20Austrian%20Economics.pdf], but I also concluded that Rothbardians were only one faction among Austrians and I am still not sure if the movement as a whole has serious problems.
One other approach to your problem that I have been thinking about is looking for work that you can audit. I recently did a back of the envelope calculation of the increase in the amount of useful land from climate change shifting temperature contours towards the poles. I then looked at a couple of published articles on the subject. They were much more sophisticated than mine, looking at things like soil quality and humidity as well as temperature, which was the only variable I considered. One got a result similar to mine, a huge increase, one concluded that the amount of arable land might go up or down a little.
Their procedure is much better than mine but mine had one large advantage — it was simple enough so a reader willing to spend a few hours could check it, see what I was doing and that I did it honestly and competently. There is no way anyone outside the field could do the equivalent for either of those papers, and it would take more than a few hours even for someone in the field. Given both the fact that the issue is politically loaded and that the two professional papers disagreed so much, the fact that I can't check them for myself is a strong argument in favor of looking at simpler models like mine instead of sophisticated ones.
I think the principle generalizes. Try to find work with an important result that you can check. If it turns out to be bogus — I'm thinking of one example in the climate field — and other people in the field haven't pointed it out, that tells you something useful about the quality of work in the field.
It works for people too. You wrote things about Adam Smith, trying to represent him as a modern progressive rather than the 18th c radical that he was, that could not have been written by an honest man who had read the texts he was quoting. [http://daviddfriedman.blogspot.com/2021/07/noah-smith-on-adam-smith.html] That sharply reduced my interest in reading other things you wrote on other subjects, although in the case of this post I thought you were making a good point.
"Suppose you and your friends wanted to push a weak argument for political purposes. You could all write a bunch of papers about it, with abstracts and numbered sections and bibliographies and everything. You could cite each other's papers. If you wanted to, you could even create a journal, and have a peer review system where you give positive reviews to each other's B.S. papers."
Climate change?
To be fair, it appears to me as an outsider to academe, that this is always a risk. Academics have always to try to build a reputation, which requires publishing. Academics will try to establish their own ideas (and hence their own reputations) as obviously dominant in the field, leading publications and reviewers to summarily reject competing ideas. If you add political implications and support for political agendas, you greatly magnify the effect.
A couple of non-political examples I can think of: Einstein and relativity were initially rejected, even though competing physicists were working on very similar theories and furiously working to flesh out the details in order to be the first to publish. Plate tectonics (continental drift) was initially rejected because it seemed so different from then-contemporary understandings of the nature of the earth. I suppose these are both examples of science working as it should: revolutionary theory should require huge amounts of support before it can displace current understanding.
But, in today's frenetic academy, there are more researchers than ever before, dividing knowledge into ever smaller sub-specialties, and lots of new dominant narratives to establish.
What about the qualities of the 'you' in your article who is recommending the 2 papers, doesn't that also enter into the equation?
I remember reading this article when your first published it - it's a good heuristic!
Fun post. I've recently been wading through the vast "brain games" literature (can playing games improve your brain), and this rule works. On the cognitive side (do they make you smarter), I could not find two articles worth recommending. On the emotional side (can they make you feel better" I could name two - including one where games outperformed anti-depressant drugs. Probably says more about drugs than brain games.
Good shtuff! Thanks
Can you site two successful used of this strategy. I like the concept and I want it to be true... Or a bit probable. What a good compendium that would be. "The 2 Paper Library of Usefulness". My approach ... Is more like "The 2 book rule". I would give a new science or school of thought 2 books to explain the ideas and usefulness. Short books I hope. Like Toynbee, if the ideas are solid, they can be condensed. This does remind me of the Google Page index approach. Ah, IF only citations did not suffer from manipulation and sycophants.
Ties back to your research subpoint on your recent post about China. There's good research being conducted there, no doubt, but also a fair amount of the low-quality, insular garbage you describe. (There's, of course, garbage publications from all countries as well, but the sense is that it's more systematic in China.)
This is a recent medical article about research publication in China that's made a bit of a splash due to a very alarming claim. I'm still not sure how much to attribute to poorly written or edited publications and how much to attribute to malicious practices or shoddy medical care (including incorrectly performed apnea tests), but none of these option is a good defense of the other. Either way, it does show the difficulty of assessing the quality of literature when there are so many unknown variables.
https://onlinelibrary.wiley.com/doi/10.1111/ajt.16969
Reminds me of religion discourse.
"I think you should read the whole Bible before critiquing it"
Unfortunately in Islam, the moat isn't as wide because the book is much shorter. The good news is it only counts if you read it in Arabic.
Sorry for bringing the discussion back to the 00s.
this helps me get over my concern about being bored reading papers that repeat what others have said.
Your two paper rule makes a lot of sense. Back in the days of the Partisan Review there used to be a lot of respect for public intellectuals. If that respect is gone, it should return. If people cannot present a good case in layman's terms for a theory, there is good reason to suspect that they are hiding behind jargon and credentials.
As a defensive rule, I don't know a better one, but there's one weakness here. It's entirely possible that a credible and relevant field has two "foundational" papers that contain a lot of weak thought. They may have introduced new ideas, but done it poorly, but if a group of academics are unwilling to replace those papers despite their flaws. You'd then end up with a situation in which certain key ideas are introduced but poorly defended in the "foundational" papers, and the good defense is diffused throughout.
Of course, an inability to take one of the possible fixes to this situation indicates a weakness in the field. The original authors could replace their flawed papers with better ones. The field could recognize the need to fix this problem trumps the academic credit system. But despite that, I know this can persist a long time.
A real-life example is Philosophy. Socrates had a great mind and did say many interesting things, but there are many flaws as well. I've personally engaged with philosophical groups that refused to move past that and only wanted to engage with the "vast literature" of ancient Greek philosophers, to the exclusion of all else.
You've written a lot about American suburbs so this piece I discovered (love substack for that very reason), well let me quote part of it.
The mind loves a mystery. And there’s a mystery to be solved in California. Why does a state with purportedly progressive values have such regressive outcomes?
The national hypothesis is to view things through a partisan lens: for left-leaners, Dem = good, GOP = bad.
But in California, that tribal shortcut doesn’t get us any closer to the answer.
At Modern Power we’re early on the journey to solve this mystery. So far we’ve discovered three noteworthy barriers to progressive outcomes: Narrow Interests, Progressive Proceduralism, and Anti-Statism.
https://modernpower.substack.com/p/why-california-politics-is-interesting?r=8ahwm&s=r&utm_campaign=post&utm_medium=web