Bayesian truth serum, grading and student evaluations

In one of my last posts, I examined some proposals for making university grading more equitable and less prone to grade inflation. Currently, professors are motivated to inflate grades because high grades correlate with high student evaluations, and these are often the only metrics of teaching effectiveness available. Is there a way to assess professors' teaching abilities independent of the subjective views of students? Similarly, is there a way to get students to provide more objective evaluation responses?

It turns out that one technique may be able to do both. Drazen Prelec, a behavioral economist at MIT, has a very interesting proposal for motivating a person to give truthful opinions in face of knowledge that his opinion is a minority view. In this technique, awesomely named "Bayesian truth serum"*, people give two pieces of information: the first is their honest opinion on the issue at hand, and the second is an estimate of how the respondent thinks other people will answer the first question.

How can this method tell if you are giving a truthful response? The algorithm assigns more points to responses to answers that are "surprisingly common", that is, answers that are more common that collectively predicted. For example, let's say you are being asked about which political candidate you support. A candidate who is chosen (in the first question) by 10% of the respondents, but only predicted as being chosen (the second question) by 5% of the respondents is a surprisingly common answer. This technique gets more true opinions because it is believed that people systematically believe that their own views are unique, and hence will underestimate the degree to which other people will predict their own true views.

But, you might reasonably say, people also believe that they represent reasonable and popular views. They are narcissists and believe that people will tend to believe what they themselves believe. It turns out that this is a corollary to the Bayesian truth serum. Let's say that you are evaluating beer (as I like to do), and let's also say that you're a big fan of Coors (I don't know why you would be, but for the sake of argument....) As a lover of Coors, you believe that most people like Coors, but feel you also recognize that you like Coors more than most people. Therefore, you adjust your actual estimate of Coors' popularity according to this belief, therefore underestimating the popularity of Coors in the population.

It also turns out that this same method can be used to identify experts. It turns out that people who have more meta-knowledge are also the people who provide the most reliable, unbiased ratings. Let's again go back to the beer tasting example. Let's say that there are certain characteristics of beer that might taste very good, but show poor beer brewing technique, say a lot of sweetness. Conversely, there can be some properties of a beer that are normal for a particular process, but seem strange to a novice, such as yeast sediment. An expert will know that too much sweetness is bad and the sediment is fine, and will also know that a novice won't know this. Hence, while the novice will believe that most people agree with his opinion, the expert will accurately predict the novice opinion.

So, what does this all have to do with grades and grade inflation? Glad you asked. Here, I propose two independent uses of BTS to help the grading problem:

1. Student work is evaluated by multiple graders, and the grade the student gets is the "surprisingly common" answer. This motivates graders to be more objective about the piece of work. We can also find the graders who are most expert by sorting them according to meta-knowledge. Of course, this is throwing more resources after grading in an already strained system.

2. When students evaluate the professor, they are also given BTS in an attempt to elicit an objective evaluation.

* When I become a rock star, this will be my band name.

Is college worth it for everyone?

In yesterday's New York Times, David Leonhardt opined that we ought to send as many young adults to college as possible. His economic arguments ran as follows:

- The income delta between college grads and non-college grads has increased from 40% to over 80% in the last three decades.
- If one calculates a return on investment for a college education, it is 15%, higher than stocks, and certainly higher than current real-estate.

Unfortunately, he completely glosses over the problem of cost. He writes:

"First, many colleges are not very expensive, once financial aid is taken into account. Average net tuition and fees at public four-year colleges this past year were only about $2,000 (though Congress may soon cut federal financial aid)."

As if the eminent cutting of federal financial aid can be reduced to a parenthetical! The reality is that college prices have increased over 130% since 1988 while median family incomes have remained stagnant. This situation makes college possible only through the amassing of large amounts of student debt. Indeed, for the first time in this country, student loan debt has surpassed credit card debt. Taking on this kind of debt in this lackluster economy is problematic. Furthermore unlike mortgages, student loan debt does not go away with bankruptcy, loading some thinkers to forecast education as the next bubble.

Leonhardt also unhelpfully compares the arguments against universal college education to the arguments against universal high school education from over half a century ago. This would be fine if we were in the position to make four years of university education part of public education. However, calling for all families to take on this debt seems irresponsible and elitist.

Growing PhDs "like mushrooms"

If you have been following this blog, it comes as no surprise that I frequently worry about the state of the university system. I believe there are structural problems in the system that are a disservice to students (both at the undergraduate and graduate levels) as well as staff (particularly adjuncts and non-tenure track faculty, but also to junior tenure-track professors as well).

Recently, Nature published a series of opinion articles on the over-production of PhDs in the sciences. We are producing too many people who are apprenticed in a career path that can accommodate only a fraction of them.

As a result, we are spending longer in graduate school and in our postdocs, but the number of people passing through the needle eye to professorship is shrinking as tenure-track jobs get replaced with temporary and adjunct positions. In 1973, 55% of US biology PhDs secured tenure-track positions within six years of completing their degrees, and only 2% were in a postdoc or other untenured academic position. By 2006, only 15% were in tenured positions six years after graduating, with 18% un-tenured. This largely fits with my perception: it has been seven years since I began graduate school, and considering my incoming class,  we are evenly spread across remaining in school, having a post-doc and getting a job in industry. Not one of us currently has a tenure-track faculty position. Something must be very broken in the system for prospects to be this bleak for graduates of a top-five department.

So why doesn't the market change such that supply meets demand? Essentially, it's that the system runs on cheap graduate and postdoctoral labor. "Yet many academics are reluctant to rock the boat as long as they are rewarded with grants (which pay for cheap PhD students) and publications (produced by their cheap PhD students). So are universities, which often receive government subsidies to fill their PhD spots." In fact, faculty members who are reluctant to perpetuate this cycle are punished in grant review, writing in costs for a research scientist at $80,000 per year when others have the same work done by a postdoc at $40,000 per year.


So, how did we get here? Part of the issue has to be that more people are going to college than ever before and the university system does not properly scale to the demand. In the US in 1970, only 11% of people over the age of 25 had a bachelor's degree, but this number had climbed to 28% by 2009. So more graduate students, postdocs and adjuncts are being used to teach the courses to accommodate all of these new students. While some claim that it is just too expensive to have tenure-track faculty teaching all of these courses, one must also consider the recent trend towards massive salaries for university professors.

Actually, if anyone could explain university economics to me, I'd be grateful.

And where do we go from here? Personally, I love the suggestions made by William Deresiewicz in this fantastic article. Particularly, "The answer is to hire more professors: real ones, not academic lettuce-pickers."

What is the real value of effective writing?

This shocking article, written by a man who makes a living writing college papers for other people, has had a lot of mileage around the web lately.

I had two immediate reactions to the essay: “I would love to invite this guy to a dinner party, he sounds really interesting” and “this is just another example of the profoundly broken economics of the American higher educational system”

“In the midst of this great recession, business is booming. At busy times, during midterms and finals, my company's staff of roughly 50 writers is not large enough to satisfy the demands of students who will pay for our work and claim it as their own.”  Stated the pseudonymous Mr. Dante.

Alex Reid wrote about some economic observations from the article. “… it's a little sad that people who are clearly accomplished writers (to be able to produce quickly good academic material across the disciplines) are willing to work for such little pay.”  This is in stark opposition to my own reaction, which was along the lines of “wow, I could increase my post-doc salary by a substantial margin by doing this!”  And recall that despite my whining, my salary is quite reasonable when compared to my adjunct peers in the humanities.  I whole-heartedly agree with Mr. Reid’s assertion that we should really start questioning our paradigms about college education.

To review: our students can’t afford not to get a college degree, and end up paying smart individuals who might otherwise be teaching them if it wasn’t more worth their while economically to pass them through the system. These students are in college because it’s just “what you do” to get a job that doesn’t involve flipping burgers.  What kind of education do people need to have for a typical job? How do we best scale this to the largest number of people?

Of course, this ghostwriting problem is far from limited to the college population. This week’s Nature had this article about “editorial services” that help with everything from experimental logic to type setting that help researchers get work published.  Such a service operates in a massive ethical gray area of authorship ethics – if a service organizes your ideas and suggests a critical control experiment, is that not a unique intellectual contribution?

Although both cases are very different, it is evident that the inability to clearly communicate one’s ideas is a primary barrier to academic and life success, and although we do not compensate teachers for this skill, its value is shown on the black market.