doc_type: hypothesis-highlights
authors: Michael Nielsen, Kanjun Qiu
url: 'https://scienceplusplus.org/metascience/index.html'
In this essay we sketch a vision of how the social processes of science may be rapidly improved ↗
We introduce the notion of a ==metascience entrepreneur==, a person seeking to achieve a scalable improvement in the social processes of science ↗
metascience is an imaginative design practice ↗
that exploration aims to ==find new social processes which unlock latent potential for discovery== ↗
decentralized change must be possible ↗
more structurally diverse set of environments for doing science ↗
metascience must develop and intertwine three elements: an imaginative design practice, an entrepreneurial discipline, and a research field ↗
it also seems likely such aliens will have radically different social processes to support science ↗
new social processes for science; that is, new ways to select and support human beings to make discoveries ↗
this essay explores the question: how well does the discovery ecosystem learn, and can we improve the way it learns ↗
explores what it would mean for humanity to do metascience seriously. And it's about placing that endeavor at the core of science ↗
Encourage people to swing for the fences by offering a large payout if they fail to receive tenure ↗
In their heyday, senior scientists at places such as Bell Labs and Cambridge's Laboratory for Molecular Biology often carried out research work themselves, or in direct, hands-on collaboration with 1-3 others20. Yet modern universities often strongly encourage scientists to take on a managerial role, applying for grants but being hands off in research work ↗
PIs don’t do the work anymore
Q. In what cases might it make sense to fund senior scientists to spend their time doing science instead of managing? Where the opposite?
A. Makes sense for senior scientists to do science when there are increasing returns to focused expertise; and to manage when there are diminishing returns. ↗
Q. Explicate "printing press for funders" concept
A. An endowment of $10B could annually spin out e.g. a new $500M funder, or many smaller funders. ↗
In a healthy discovery ecosystem the improved idea could come from anywhere. ↗
Rather, the point is that a flourishing ecosystem would rapidly generate and seriously trial an enormous profusion of ideas, including many ideas far more imaginative than anything listed above ↗
Q. Vivid example given in response to "just fund good people doing good work"
A. Think of Renaissance shipping, which was revolutionized by maritime insurance. Their answer wasn't "just get good ships, crewed by good sailors" ↗
The fundamental underlying questions are: How do social processes in science change? Is there a general theory of such change? Is it possible to speed up and improve that change? ↗
Q. What sorts of questions do detector-oriented funding ideas ask?
A. e.g. What signals are present but ignored? What is hidden? What do scientists know but funders don't? How can this be surfaced? ↗
Q. What sorts of questions are important to the predictor part of the detector/predictor funding heuristic?
A. e.g. Where's the risk, and how might it be transformed or hedged? What information should we collect to make predictions? Which opportunities have opportunity for unique marginal impact? ↗
Q. What might science funders learn from Bessemer?
A. BV maintains a list of companies (like Apple, Google, Facebook, Tesla) where they had an opportunity to invest but didn't. Science funders should do the same! ↗
Q. Two-part model MN and KQ propose as a heuristic for exploring the meta science design space?
A. Funders as detectors and predictors ↗
Q. Explicate the high-variance funding model from detector and predictor perspectives.
A. Detector: it might encourage people with riskier ideas to apply; predictor: use variance to infer future outcomes ↗
Q. Give a few examples of detector-oriented funding models.
A. e.g. endowed professorships by 25; FROs; para-academic fellowships; discipline-switching fellowships ↗
Q. Funders-as-detectors aim to locate… (evocative phrase)
A. "intellectual dark matter" ↗
Q. Explicate the Century Grant Program in terms of the detector-and-predictor heuristic.
A. It's primarily detector-oriented: it's about finding projects which might last a century or more (which might otherwise have been "dark matter"). ↗
it would also surface useful aggregate information: if people are unexpectedly going into unglamorous fields or leaving high status fields, that's a striking signal that something is afoot ↗
even "failed" programs would be successful: they will contribute crucial knowledge to our understanding of metascience ↗
Q. MN/KQ's "unifying motivating question" underlying the notion of intellectual dark matter?
A. "what does the body of scientists know that is important, and yet is currently either invisible (or not visible enough) to funders?" ↗
Q. On what basis do MN/KQ suggest that PARC is much more successful than Bell Labs?
A. BL had >10x larger budget; per-dollar, PARC seems more successful. ↗
even "failed" programs would be successful: they will contribute crucial knowledge to our understanding of metascience ↗
it would also surface useful aggregate information: if people are unexpectedly going into unglamorous fields or leaving high status fields, that's a striking signal that something is afoot ↗
Q. How do MN/KQ frame FROs relative to prior projects like LIGO, LHC, and the human genome project?
A. Those were all bespoke; FROs generalize the notion of eliciting and creating such entities. ↗
Q. Give a few examples of predictor-oriented funding programs.
A. e.g. eliciting the secret thesis; quota for young program managers; a "Nobel prize" for funders ↗
Q. Key reason MN/KQ give as to why science is bottlenecked on field formation?
A. Current social processes support work in existing fields but strongly inhibit work necessary to create new fields. ↗
Q. How do institutional investors and science funders differ in their methods for approaching risk?
A. Institutional investors develop sophisticated hedging, insurance, and rebalancing strategies to maintain a target risk profile; science funders mostly seem to use informal heuristics and wishful thinking. ↗
creating far stronger alignment between individual incentives and collective social good48 ↗
diversify exploration and unleash creativity by decreasing the amount of grant overhead, ↗
would it lead to an explosion of discovery if we temporarily but drastically decreased (and then increased) funding to entities such as the NSF ↗
We believe the most important and powerful social primitives in this design space are yet to be discovered. ↗
Q. What exception do MN/KQ give to the claim that "garage band research orgs" don't grow to worldwide pre-eminence?
A. DeepMind, which has received increasing sums in response to increasing success. (But nb: that investment is more about commercial viability than scientific merit—those things are only partially correlated) ↗
Q. According to MN/KQ, why aren't funding lotteries an encouraging rebuttal to charges of stasis in science funding?
A. It's one fairly modest idea which has taken ~25 years to get serious trials. There should have been a flood of ambitious ideas tried in this time; science funding should look vastly different today because of what we've learned. ↗
Q. Four main reasons MN/KQ give why scalable change to social processes is hard in science?
A. control centralized in a few institutions; no single org/person who can make a change; network effects homogenize collectively held social processes; no feedback loop rewards successful new institutions with growth ↗
Q. How do the Shanghai Rankings of research universities demonstrate a contrast in dynamism with tech?
A. The top 10 have remained consistent over 19 years (with one exception); by contrast, a parallel list in tech is utterly transformed (e.g. Meta didn't exist 19 years ago). ↗
Q. In what sense is Kariko's funding nightmare a story of ecosystem stasis, not just of mistakes made by the NIH and her university?
A. The discovery ecosystem hasn't seemed to learn anything from these mistakes. There's no serious post-mortem. There doesn't seem to be a viable path to incumbents incorporating such learnings and supplanting existing institutions. ↗
Q. One might argue: institutions failed Kariko… but mRNA did get invented after all. Maybe the system's working fine. MN/KQ's response?
A. How many Kariko's have been missed? We only see people who barely managed to find their way through. ↗
Q. Which of "Thinking, Fast and Slow"s studies failed to replicate in Doyen's 2012 study?
A. Priming ↗
Q. In what year did Nosek et al ("The Open Science Collaboration") publish "Estimating the reproducibility of psychological science"?
A. 2015 ↗
Q. What was the title of Nosek et al's first mass reproducibility paper?
A. Estimating the reproducibility of psychological science ↗
Q. In broad strokes, what did Nosek et al do in "Estimating the reproducibility of psychological science"?
A. Attempted to reproduce 100 experimental social psychology papers ↗
the problems underlying the replication crisis had been widely known for decades before 2011-2015 ↗
Q. In contrast to Nosek et al's 2012 reproducibility paper, what did Protzko et al's 2020 reproducibility study find?
A. It demonstrated successful reproductions of social psychology studies which were done with "best practices" (pre-registered study designs, large samples, and open materials) ↗
Q. What contrarian response to MN/KQ have to the negative portrayal of the social psychology reproducibility crisis?
A. This is a Renaissance! In this period many people began to believe in these problems, started taking action to address them. ↗
Q. How did Nosek fund his metascience work in years before founding the Center for Open Science?
A. Speaking fees from talks about his prior work (his NSF/NIH grants were all rejected). Later, he got private funding, in particular from the Arnold Foundation. ↗
{metascience entrepreneur}: MN/KQ's term for {a person who aims to make a scalable improvement in the social processes of science} ↗
Q. Why do MN/KQ argue it's important to have outsiders as metascience entrepreneurs?
A. Decentralization—you don't need to be rich or a director of a big incumbent institution. ↗
Similarly, some of those people at the funders have expressed to us that they are doing metascience entrepreneurship. And yet they're not: their obstacles are quite different, and require separate analysis ↗
Q. What do MN/KQ mean by a "decisive result" (e.g. in metascience)?
A. A result strong enough to routinely convince someone predisposed against the conclusion (e.g. as in the social psychology replication crisis) ↗
Q. Give some examples of ways in which new social processes in science can enable the "same" person to do radically different types of work.
A. e.g.: exposed to different problems, different values and incentives shaping their work, different tools, different networks of expertise helping them, etc ↗
Q. MN/KQ argue metascience must develop and intertwine three elements:
A. an imaginative design practice, an entrepreneurial discipline, and a research field ↗
Q. Striking motivating analogy: "Could aliens have developed scientific institutions…"
A. "…as superior to ours as modern universities are to the learned medieval monasteries?" ↗
Q. What does the "tenure insurance" idea propose?
A. Encourage risk-taking by offering a large payout to people if they fail to receive tenure. ↗
Q. What does the "failure audit" idea propose?
A. Grant programs should track the fraction of their programs which fail and take drastic action if it's too low. ↗
Q. What might science funders learn from Bessemer Ventures?
A. BV maintains a list of companies (like Apple, Google, Facebook, Tesla) where they had an opportunity to invest but didn't. Science funders should do the same! ↗
Q. What does the "at-the-bench fellowship" idea suggest?
A. Fund senior scientists to spend their time actually doing science, rather than managing grad students and applying for grants. ↗