Research Works Act - seriously?

I am not a fan of the academic publishing industry, and have written before on the need for more openness in the publishing process. My position is very simple: it is not ethical for taxpayers to be forced to buy access to scientific articles whose research was funded by the taxpayer.

I am very dismayed at the introduction of the Research Works Act, a piece of legislation designed to end the NIH Open Access policy and other future openness initiatives.

Sigh... even in academic publishing, we're socializing the risks and privatizing the gains. Here, I agree completely with Michael Eisen's statement in the New York Times:

 "But the latest effort to overturn the N.I.H.’s public access policy should dispel any remaining illusions that commercial publishers are serving the interests of the scientific community and public."

As this bill was written by representatives taking money from the publishing industry, perhaps we should include lawmakers in that group as well.

Happy Open-Access publishing week!

Is the academic publishing industry evil?

Like most people, I didn't think much about the profit model for academic journals until I was publishing in them. Even after going through the process a few times, I am still struck by a feeling that academic journals are the toll trolls on the road of knowledge dissemination.

While a non-academic journal such as The Atlantic or the New Yorker pays its authors for content, academic journals get massive amounts of content volunteered to them. While non-academic journals pay an editor to hone and perfect the content, academic journals have volunteer peer reviewers and volunteer action editors doing this work for the cost of a line on the academic CV. Both types of journals offset some publication costs with advertising, but while non-academic journals sell for ~$5 per issue and under $50 for a year's subscription, an academic journal will charge $30-40 per article and thousands for a subscription. This means that the tax payer who funds this research is not able to afford to read the research.

Let's say you're an author, and you're submitting your article to a scientific journal. It gets reviewed and edited, and is accepted for publication by the action editor. Great! Your excitement gets diminished somewhat from two documents that get sent to you: one that signs over your copyright to the journal, and a publishing bill based on the number of pages and color figures in your work (often a few hundred dollars). Now, if you want to use a figure from this article again (say, for your doctoral dissertation), you must write the journal to get permission to use your own figure. Seriously. Other points against academic journals can be found in this entertainingly inflammatory piece.

But what about open access journals? Good question. These journals exist online, and anyone can read them, which is great for small libraries struggling to afford journal costs and citizens wishing to check claims at the source. They're not so great for the academic, who gets slapped with a $1000-2000 fee for publishing in them. As inexpensive as online infrastructure is these days, I would love for someone to explain to me how it costs the journal so much just to host a paper.

I was excited to read this interview with academic publishers Wiley and Elsevier on these issues. However, I find most of the responses to be non-answer run-arounds. A telling exception to this is in the first question "what is your position on Open Access databases?". Wiley responded:

"The decision to submit a manuscript for publication in a peer-review journal reflects the researcher’s desire to obtain credentialing for the work described. The publishing process, from peer review through distribution and enabling discovery, adds value, which is manifest in the final version of the article and formally validates the research and the researcher."

(Emphasis mine).

In other words, we do this because there is a demand for our journal as a brand. You, researcher are creating the demand. However, I do hold out hope that as more publishing moves online, more researchers and librarians realize that there are both diamonds and rough in all journals, and this will wear away at brand prestige, allowing the illusion of "publisher added value" to wear away.

Managing scholarly reading

Reading, after a certain age, diverts the mind too much from its creative pursuits. Any man who reads too much and uses his own brain too little falls into lazy habits of thinking. —ALBERT EINSTEIN

How much literature should one read as an academic? Of course, the answer will vary by field, but even within my own field, I find little consensus as to the "right" amount of reading to do.

It is true that no one can read everything that is published, even in a single field such as cognitive science, while maintaining one's own productivity. In my Google reader, I subscribe to the RSS of 26 journals, and from these, I get an average of 37 articles per day. However, in an average day, I feel like I should pay attention to 5 of these. If I were to closely read all of these, I would run out of time to create new experiments, analyze data and write my own papers.

It turns out that in an average day, I'll read one of these papers and "tag" the other 4 as things I should read. But this strategy gets out of control quickly. In May, I went to a conference, didn't check my  reader for a couple of days and came back to over 500 journal articles, or around 35 that I felt deserved to be read. I have over 1300 items tagged "to read" in my Zotero library. At my current rate of reading, it would take me over 3.5 years to get through the backlog even if I didn't add a single article to the queue.

So, how to stay informed in an age of information overload? It seems that there are a few strategies:

1. Read for, rather than read to. In other words, read when knowledge on a particular topic is to be used in a paper or grant review, but don't read anything without a specific purpose for that information. According to proponents of this method, information obtained when reading-for-reading's-take will be lost anyway, leading to re-reading when one needs the information.

This method vastly decreases the overwhelming nature of the information, and makes info acquisition efficient. However, it is not always practical for science: if you're only reading for your own productivity, you're going to miss critical papers, and at worst, are going to be doing experiments that were already done.

2. Social "reading", augmented by abstract skimming. In this method, one does not spend time reading, but spends time going to as many talks and conferences as possible, learning about literature by using the knowledge of one's colleagues. This method seems to work best in crowded fields. The more unique your research program, the more you'll have to do your own reading. And all of this traveling is time and money consuming.

3.  Don't worry about checking through many journals, but set alerts for the specific topics. My favorite is PubCrawler, suggested by Neuroskeptic. Works well when my key words and the authors' key words coincide, but I seem to have set too many topics and I get both too many "misses" and "false alarms".

How do you keep up with literature?

What are you writing that will be read in 10 years?

This was a question asked to an acquaintance during a job interview for a professorship in the humanities. It's one hell of a question, and one that I find unfortunately unasked in the sciences.

In my other life, I submitted a paper this week. It's not a bad paper - it shows something new,  but like too many papers being published today, it's incremental and generally forgettable. It's not something that will be read much in 10 years.

I love reading old papers. They are from a time when authors were under less pressure to produce by volume. They are consequently more theoretical, thoughtful and broad than most papers published today because the authors had the luxurious time to sit and think about the results, and place them in context.

As I've pointed out earlier, the competitive academic environment tends to foster bias in publications: when trying to distinguish oneself amongst the fray of other researchers, one looks for sexy and surprising results. So do the journals, who want to publish things that will get cited the most. And so do media outlets, vying for your attention.

Jonah Lehrer's new piece on the "decline effect" in the New Yorker almost gets it right. The decline effect, according to Lehrer, is the phenomenon of a scientific finding's effect size decreasing over time. Lehrer dances around the statistical explanations of the effect (regression to the mean, publication bias, selective reporting and significance fishing), and seems all-too-willing to dismiss these over a more "magical" and "handwave-y" explanation:

"This is largely because scientific research will always be shadowed by a force that can’t be curbed, only contained: sheer randomness" 

But randomness (along with the sheer number of experiments being done) is the underlying basis of the other effects he wrote about and dismissed. The large number of scientists we have doing an even larger number of experiments is not unlike the proverbial monkeys randomly plunking keys on a typewriter. Eventually, some of these monkeys will produce some "interesting" results: "to be or not to be" or "Alas, poor Yorick!" However, it is unlikely that the same monkeys will produce similar astounding results in the future.

Like all analogies, this one is imperfect as I am not trying to imply that scientists are only shuffling through statistical randomness. What I am saying is that given publication standards of large, new, interesting and surprising results, it is very likely that any experiment meeting these standards is an outlier and that its effect size will regress to the mean. This cuts two ways: although some large effects will get smaller, some experiments that were shelved for having small effects will probably have larger effect sizes if repeated in the future.

 This gets us back to my penchant for old papers. With more time, a researcher could do several replications of the study, and find the parameters under which the effects could be elicited. And often, these papers are from the pre-null-hypothesis significance testing days, so the effects tend to be larger as they need to be visually obvious from a graph. (A colleague once called this the JFO statistical test for "just f-ing obvious". It's a good standard) This standard guards against many of the statistical sins outlined by John Ioannidis.

This is also why advances in bibiometrics are going to be key for shaping science in the future. If we can formalize what makes a paper good, and what makes a scientist's work "good", then (hopefully) we can go about doing good, rather than voluminous, science.

What is the real value of effective writing?

This shocking article, written by a man who makes a living writing college papers for other people, has had a lot of mileage around the web lately.

I had two immediate reactions to the essay: “I would love to invite this guy to a dinner party, he sounds really interesting” and “this is just another example of the profoundly broken economics of the American higher educational system”

“In the midst of this great recession, business is booming. At busy times, during midterms and finals, my company's staff of roughly 50 writers is not large enough to satisfy the demands of students who will pay for our work and claim it as their own.”  Stated the pseudonymous Mr. Dante.

Alex Reid wrote about some economic observations from the article. “… it's a little sad that people who are clearly accomplished writers (to be able to produce quickly good academic material across the disciplines) are willing to work for such little pay.”  This is in stark opposition to my own reaction, which was along the lines of “wow, I could increase my post-doc salary by a substantial margin by doing this!”  And recall that despite my whining, my salary is quite reasonable when compared to my adjunct peers in the humanities.  I whole-heartedly agree with Mr. Reid’s assertion that we should really start questioning our paradigms about college education.

To review: our students can’t afford not to get a college degree, and end up paying smart individuals who might otherwise be teaching them if it wasn’t more worth their while economically to pass them through the system. These students are in college because it’s just “what you do” to get a job that doesn’t involve flipping burgers.  What kind of education do people need to have for a typical job? How do we best scale this to the largest number of people?

Of course, this ghostwriting problem is far from limited to the college population. This week’s Nature had this article about “editorial services” that help with everything from experimental logic to type setting that help researchers get work published.  Such a service operates in a massive ethical gray area of authorship ethics – if a service organizes your ideas and suggests a critical control experiment, is that not a unique intellectual contribution?

Although both cases are very different, it is evident that the inability to clearly communicate one’s ideas is a primary barrier to academic and life success, and although we do not compensate teachers for this skill, its value is shown on the black market.

How many published studies are actually true?

I’d like to point readers to this excellent new article in The Atlantic on meta-researcher John Ioannidis. Ioannidis is building quite the career on exposing the multiple biases in medical research. He has taken a field to task publishing papers with shy titles such as “Why most research findings are false”. He is rapidly becoming a personal hero of mine.

Ioannidis has examined and formally quantified research biases at all levels of “production”: in which questions are being asked, in the design of experiments, in the analysis of these experiments, and in the presentation and interpretation of the results. “At every step in the process, there is room to distort results, a way to make a stronger claim or to select what is going to be concluded,” says Ioannidis in the article. “There is an intellectual conflict of interest that pressures researchers to find whatever it is that is most likely to get them funded.”

While I have examined some of these biases for both general research and fMRI experiments, it’s worth noting that in the context of medical research, the stakes are even higher as they affect patient care. It is also unfortunate that medical studies are, according to Ioannidis, more likely to contain bias as there are stronger financial interests vested in the results, compared to cognitive neuroscience. 

An unfortunate result of the competitive research environment is a lack of replication of scientific results. Although replication is the gold standard of a result’s truth, there is little acknowledgment, and thus little motivation for researchers to do this, except for the most bold of claims. Without replication, bias in research increases. However, even when a failure to replicate a major study is published, it often gets very little attention. A case in point is the failure to replicate the “Mozart effect”: the finding that listening to 10 minutes of a Mozart sonata significantly increased participants’ performance on a spatial reasoning test. A quick Googling of “Mozart effect” will show you several companies selling you Mozart recordings to increase your child’s IQ, despite the failure to replicate.

It is very easy to get discouraged by this, after all, science should be a science, right? Ioannidis seems less discouraged, and reminds us of the following: “Science is a noble endeavor, but it’s also a low-yield endeavor… I’m not sure that more than a very small percentage of medical research is ever likely to lead to major improvements in clinical outcomes and quality of life. We should be very comfortable with that fact.”

Should we crowd-source peer review?

Peer review has been the gold standard for judging the quality of scientific work since World War II. However, it is a time consuming and error-prone process. Now, both lay and academic work is questioning whether the peer review system should be ditched in favor of a crowd-sourced model. 

Currently, a typical situation from an author’s perspective is to send out a paper, and receive 3 reviews about three months later. Typically, the reviewers will not completely agree with one another, and it is up to the editor to decide what to do with, for example, two mostly-positive and one scathingly negative review. How can the objective merit of a piece of work accurately be judged on such limited, noisy data? Were all of the reviewers close experts in the field? Were they rushed into doing a sloppy job? Did they feel the need for revenge against an author that unfairly judged one of their papers? Did they feel like they were in competition with the authors of the paper? Did they feel irrationally positive or negative towards the author’s institution or gender?

And from the reviewer’s point of view, reviewing is a thankless and time-consuming job. It is often a full day’s work to read, think about, and write a full and fair review of a paper. It requires accurate judgment on all matters from grammar and statistics to a determination of future importance to the field. And the larger the problems the paper has, the more time is spent in the description of and prescription for these problems. So, at the end of the day, you send your review and feel 30 seconds of gratitude that it’s over and you can go on to the rest of your to-do list. In a couple of months, you’ll be copied on the editor’s decision, but you almost never get any feedback about the quality of the review from an editor, and very little professional recognition of your efforts.

The peer review process is indeed noisy. A study of reviewer agreement of conference presentations found that the rate of reviewer agreement was not different from chance. In a study described here, women’s publications in law reviews were shown to have more citations than mens’. A possible interpretation of this result is that women are treated harsher in the peer review process, and as a consequence publish (when they can publish) better quality articles than men who do not have the same level of scrutiny. 

In peer review, one must also worry about competition and jealousy. In fact, a perfectly "rational" (Machiavellian) reviewer might reject all work that is better than his own for the purpose of advancing his career. In a simple computational model of the peer review process, it was found that the ratio of either "rational" or random reviewers needed to be kept below 30% for the system to beat chance. It also concludes that the refereeing system works the best when only the best papers are published. One can easily see how the “publish or perish” system hurts science.

It is a statistical fact that averaging over many noisy measurements provides a more accurate answer than any one answer. Francis Galton discovered this when asking individuals in a crowd to estimate the weight of an ox. Pooling over noisy estimates works when you ask for one measurement from many people, or when you ask the same person to estimate multiple times. A salient modern example of the power of crowd-sourcing is, of course,Wikipedia.

In a completely crowd-sourced model of publication, everything that is submitted gets published, and everyone who wants to can read and comment. Academic publishing would be quite similar to the blogosphere, in other words. The merits of a paper could then be determined by the citations, track backs, page views, etc.

On one hand, there are highly selective journals such as Nature who reject more than half of submitted papers before they even get to peer review and finally publish 7% of submissions. In this system, too many good papers are getting rejected. On the other hand, a completely crowd-sourced model means that there are too many papers for any scientist in the field to keep up with, and too many good papers won’t be read because it’s not worth one’s time to find diamonds in the rough. Furthermore, although the academy far from settled on the matter of how to rate professors for hiring and tenure decisions, it is more unclear what a “good” paper would be in this system as more controversial topics would get more attention.

The one real issue I see is that without editors seeking out reviewers to do the job, I worry that the only people reviewing a given paper will be the friends, colleagues and enemies of the authors, and this could make publication a popularity contest. Some data bear out this worry. In 2006, Nature conducted an experiment on the addition of open comments to the normal peer review process. Of the 71 papers that took part in the experiment, just under half received no comments at all, and half of the total comments were on only eight papers!

So, at the end of the day, I do believe that with good editorial control over comments, that a more open peer-reviewing system would be of tremendous benefit to authors, reviewers and science.