Saturday, January 15, 2011

What are you writing that will be read in 10 years?

This was a question asked to an acquaintance during a job interview for a professorship in the humanities. It's one hell of a question, and one that I find unfortunately unasked in the sciences.

In my other life, I submitted a paper this week. It's not a bad paper - it shows something new,  but like too many papers being published today, it's incremental and generally forgettable. It's not something that will be read much in 10 years.

I love reading old papers. They are from a time when authors were under less pressure to produce by volume. They are consequently more theoretical, thoughtful and broad than most papers published today because the authors had the luxurious time to sit and think about the results, and place them in context.

As I've pointed out earlier, the competitive academic environment tends to foster bias in publications: when trying to distinguish oneself amongst the fray of other researchers, one looks for sexy and surprising results. So do the journals, who want to publish things that will get cited the most. And so do media outlets, vying for your attention.

Jonah Lehrer's new piece on the "decline effect" in the New Yorker almost gets it right. The decline effect, according to Lehrer, is the phenomenon of a scientific finding's effect size decreasing over time. Lehrer dances around the statistical explanations of the effect (regression to the mean, publication bias, selective reporting and significance fishing), and seems all-too-willing to dismiss these over a more "magical" and "handwave-y" explanation:

"This is largely because scientific research will always be shadowed by a force that can’t be curbed, only contained: sheer randomness" 

But randomness (along with the sheer number of experiments being done) is the underlying basis of the other effects he wrote about and dismissed. The large number of scientists we have doing an even larger number of experiments is not unlike the proverbial monkeys randomly plunking keys on a typewriter. Eventually, some of these monkeys will produce some "interesting" results: "to be or not to be" or "Alas, poor Yorick!" However, it is unlikely that the same monkeys will produce similar astounding results in the future.

Like all analogies, this one is imperfect as I am not trying to imply that scientists are only shuffling through statistical randomness. What I am saying is that given publication standards of large, new, interesting and surprising results, it is very likely that any experiment meeting these standards is an outlier and that its effect size will regress to the mean. This cuts two ways: although some large effects will get smaller, some experiments that were shelved for having small effects will probably have larger effect sizes if repeated in the future.

 This gets us back to my penchant for old papers. With more time, a researcher could do several replications of the study, and find the parameters under which the effects could be elicited. And often, these papers are from the pre-null-hypothesis significance testing days, so the effects tend to be larger as they need to be visually obvious from a graph. (A colleague once called this the JFO statistical test for "just f-ing obvious". It's a good standard) This standard guards against many of the statistical sins outlined by John Ioannidis.

This is also why advances in bibiometrics are going to be key for shaping science in the future. If we can formalize what makes a paper good, and what makes a scientist's work "good", then (hopefully) we can go about doing good, rather than voluminous, science.

7 comments:

  1. Nice post. These days, it's hard to imagine being able to do REAL science. You get grant money and you gotta produce something, no matter what, so you're forced to fish through you data for "significant" findings. The scientific method is broken indeed.

    ReplyDelete
  2. Your bibliometrics comment got me thinking about how its done... so I went and looked up some of the common approaches currently used. I tried ReaderMeter and I skimmed some of the articles you recommend in the side-bar. Looks like fun, but I don't see how these metrics would be immune to abuses by scientists who use their peer-review positions to force others to cite their work.

    Just sayin'

    M

    ReplyDelete
  3. Is it the scientific method that's broken, or the funding system? Science is a low-yield beast, and we need our representatives knowing and accepting this.

    @ M, no nothing is perfect or immune from abuse, but I'm heartened that we're asking the question of how to get better metrics.

    ReplyDelete
  4. I can't recall who said this, but I thought it was a good analogy, one researcher I met once likened research science funding to venture capitalism, essentially funding the randomness.

    ReplyDelete
  5. Great post. "TERRIFIC QUESTION", I added to my cut and paste notes with URL, on disk.

    But one problem (global; nothing in particular to do with THIS post): your blog is nearly physically unreadable. Thin white sans-serif type on a black background is horrible. Even increasing the size (control-+) helps only a little. And highlighting all (edit: select: all) helps only slightly, turning the body text to a medium blue that is only slightly more readable. If I were going to hang out here and be a regular reader (which I might, though have to read a few more posts) I would be reduced to having to edit:select-all:copy and paste to a notepad window, just to READ the sucker!

    White type on black background is GREAT as an opening splash screen, for visual effect. It is very striking. I like using it myself. But it IS NOT FOR **READING**.

    This white-on-black thing is one (among many) reflection of what I call the marginalization of content: the message getting buried in a heap of formatting complexity, weirdness, garishness, inappropriateness, etc.

    ReplyDelete
  6. Thanks for the constructive suggestions. :)

    ReplyDelete