Why brain-based lie detection is not ready for "prime time"

We are in a new and interesting legal world. Although to date, no US court cases have used brain-based lie detection techniques as evidence, several cases have sought such evidence and settled out of court. fMRI is the most frequent type of brain-based lie detection technology, with two companies, Cephos and No Lie MRI providing this service in the legal domain. There have also been attempts made to use EEG for deception detection. Notably, such a technique was used in part to prosecute a young woman for murder in India in 2008.

I am far from the first to point out that this technology is highly exploratory and not accurate enough to be used in the court of law. My goal here is to outline a good number of the reasons this is the case.

9. We do not know how accurate these techniques are. Although the two aforementioned companies boast lie detection accuracy rates of 90%+, these cannot be independently verified by an independent lab as the methods used by these companies are trade secrets. For example, there are few peer-reviewed studies of the putative EEG-based marker of deception, the P300, and most come from the lab that is commercially involved with a company trying to sell the technique as a product. Interestingly, an independent lab studying the effect of countermeasures on the technique found an 82% hit rate in controls (not the 99% accuracy claimed by the company), and this was reduced to 18% when countermeasures were used!

8. In the academic literature, where we do have access to methodology, we are limited to testing typical research participants: undergraduate psychology majors (although see this). For a lie detection method to be valid, it would need to be shown as accurate in a wide variety of populations, varying in age, education, drug use, etc. This population is not likely to be skilled in deception as a career criminal might, and it has been shown that the more often one lies, the easier it is to lie. Most fMRI-based lie detection techniques are based on the assumption that lying is hard to do, and thus requires the brain to use more energy. If frequent lying makes lying easy, then it could be the case that practiced liars don't have this pattern of brain activity.
     Although a fair amount has been made lately about WEIRD subjects, participants in these studies are actually beyond WEIRD: they are almost exclusively right handed, and predominantly male.

7. Along this same line, the "lies" that are told in these studies rarely have an impact on the lives of the student participants. Occasionally, an extra reward is given if the participant is able to "trick" the system, but in the real world, with reputations and civil liberties at stake, one might imagine that one might do a better job at tricking the scanner. However, being instructed to lie about a low-stakes laboratory situation is not the same as the high-stress situations where this technology would be used in real-life. Occasionally, a study will try to ameliorate this situation by using a mock crime (such as a theft) as the deceptive stimuli. However, these are also of limited use as participants know that the situation is contrived.

6. Like traditional polygraph tests, it is possible to fool brain-based lie detection systems with countermeasures. Indeed, in an article in press at NeuroImage, Ganis and colleagues found that deliberate countermeasures on the part of their participants dropped deception detection from 100% to 30%. Most studies of fMRI lie detection have found more brain activation for lies than truth, suggesting that it is more difficult for participants to lie. However, is this still the case with well-rehearsed lies? What about subjects performing mental arithmetic during truth to fool the scanner?
    
5. A general lack of consistency in the findings in the academic literature. To date, there are ~25 published, peer-reviewed studies of deception and fMRI. Of these studies there are at least as many brain areas implicated in deception, including the anterior prefrontal area, ventromedial prefrontal area, dorsolateral prefrontal area, parahippocampal areas, anterior cingulate, left posterior cingulate, temporal and subcortical caudate, right precuneous, left cerebellum, insula, putamen, caudate, thalamus, and various regions of temporal cortex! Of course, we know better than to believe that there is some dedicated "lying region" of the brain, and given the diversity of deception tasks (everything from "lie about this playing card" to "lie about things you typically do during the day"), the diversity of regions is not surprising. However, the lack of replication is a cause for concern, particularly when we are applying science to issues of civil liberties.

4. An additional issue surrounds the fact that many of these studies are not properly balanced. In other words, participants are instructed to lie more or less often than they are instructed to tell the truth.

3. There is a large difference between group averages and finding deception within an individual. Knowing that on average, brain region X is significantly more active in a group of subjects during deception than during truth does not tell you than for subject 2 on trial 9 than deception was likely to occur due to the differences in activation. Of course, some studies are trying to study this level of analysis, but right now they are the majority.

2. Some things that we think that are not true are not necessarily lies. Most of us believe we are above-average drivers, and smarter and more attractive than most even when these beliefs are not true. Memories, even so-called "flash-bulb" memories are not fool proof.

1. Are all lies equivalent to the brain?  Are lies about knowledge of a crime the same in the brain as white lies such as "no, honey those pants don't make you look fat" or lies of omission or self-deceiving lies?

Getting Savage on evolutionary psychology

Now, I love sex advice columnist Dan Savage. I have been a faithful purveyor of his columns, podcasts and blogs for some time now. And sure, I don’t agree with him on every bit of advice, but he bats a solid .900 and articulately calls out many forms of B.S. But right now, I have a beef with Mr. Savage over his love affair with the new evolutionary psychology-inspired book Sex at Dawn by Christopher Ryan and Cacilda Jethá. This is an example of the all-too-common use of science-y thinking as justification for a particular belief, in this case, the use of evolutionary psychology to endorse the “naturalness” of polyamory.

My contentions are the following: 1. while I am all for the promotion of reading and scientific literacy, we need to be especially vigilant against accepting poor science that confirms what we already believe; 2. that we need to critically examine whether science can inform social policy discussions; and 3. we need to divorce the notion that the “naturalness” of an act means that the act is desirable.

Problems with evolutionary psychology

I need to point out in the spirit of full-disclosure that I have not read Sex at Dawn. However, from Mr. Savage’s multiple interviews with Dr. Ryan, it is evident that the apple of this book does not fall far from the tree of Buss and Baker.

Evolutionary psychology offers only post-hoc fits of theory to data

            In evolutionary psychology, one asks how human evolutionary history can explain aspects of current human behavior. Functionally, it amounts to doing thought experiments on questions such as “how did a cave man’s life influence the shape of the human penis”? The problem with this kind of problem statement is that you are looking at some data (in this case the shape of human penises) and looking for a model that fits this data. You can come up with many such models, because you are fitting the data after the fact, but you have no guarantee that your model is correct.

Let’s take a case in point of an issue brought up in the latest interview with Dr. Ryan on the Savage Lovecast. The question: why are human penises larger than gorilla penises when gorillas are larger than men? The given answer: because they were designed as plungers to remove the semen of rival males from the reproductive tract of a female. The larger theory behind this answer lies in the idea of sperm competition, the notion that females practice selective non-monogamy as a means of maximizing genetic quality in the offspring. The male, worried that he might be cuckolded into investing resources into offspring not genetically related to him needs adaptations to keep his partner from being impregnated by rivals. Therefore, it is to his advantage to have a “plunger penis” that will reduce the probability of pregnancy from a rival.

           

It’s kind of like an intellectual Rube-Goldberg machine, isn’t it? Or perhaps more fittingly, like one of Kipling’s “just-so stories”.

The “scientific data” for this claim come from this paper, which might be the most hilarious scientific study I’ve ever read (and this includes the smoking pot in the fMRI scanner study). From the abstract:

Inanimate models were used to assess the possibility that certain features of the human penis evolved to displace semen left by other males in the female reproductive tract. Displacement of artificial semen in simulated vaginas varied as a function of glans/coronal ridge morphology, semen viscosity, and depth of thrusting. Results obtained by modifying an artificial penis suggest that the coronal ridge is an important morphological feature mediating semen displacement.

 

Yes, kids… this is research with dildos and masturbation sleeves. Other great sound bites from the article include the “recipe” for artificial semen: 

Simulated semen was created by mixing 7 ml of water at room temperature with 7.16 g of cornstarch and stirring for 5 min. After trying different mixtures of cornstarch and water, this recipe was judged by three sexually experienced males to best approximate the viscosity and texture of human seminal fluid.

And in addressing limitations of the current paradigm:

A limitation of our attempts to model semen displacement was the greater rigidity of the prosthetic as compared to real genitals. The artificial vaginas did not expand as readily as real vaginal tissue nor did the phalluses compress, and, as a result, semen displacement was assessed on the basis of a single insertion. The effects, however, were robust and generalized across different artificial phalluses, different artificial vaginas, different types of simulated semen, and different semen viscosities.

…Sigh…. My own research seems so vanilla in comparison! But in all seriousness, extraordinary claims require extraordinary evidence, and this is not that evidence.

 

Evolutionary psychology does not make uniquely falsifiable claims

The hallmark of actual science is that it makes predictions that can be falsified and separated from other possible explanations. Evolutionary psychology does not do this. For example, the fact that men who have spent more time away from their partners find their partners more attractive and desirable, and ejaculate semen with higher sperm counts during copulation are taken as evidence for the sperm competition hypothesis. The argument is that as the man has not observed his partner, he is threatened by sperm competition, so it is to his advantage to copulate often and with… uh, greater virility. Although these studies control for time since last copulation, it doesn’t take much creativity to come up with alternative explanations.

Another example: the sperm competition hypothesis would predict that men would be more concerned with sexual infidelities of a partner (as this could result in cuckoldry) and women would be more concerned with emotional infidelities (as this could result in him leaving her without resources, or diverting resources into another partner). To test this prediction, David Buss conducted many surveys with many different groups asking them whether they would theoretically be more upset by a sexual or emotional infidelity. As nicely shown in David Buller’s critique of evolutionary psychology, although more men than women say that sexual infidelity is more upsetting, half of the men are still choosing emotional infidelity as more upsetting, so this model is far from complete.

Evolutionary psychology assumes that we know what psychological pressures existed for our ancestors in the Pleistocene.

We don’t. 

A closely related problem is that evolutionary psychology assumes that the mind evolved to the problems of the Pleistocene and then remained static for over 12,000 years. This seems implausible as large species-wide shifts have been observed in as little as 18 generations (less than 500 years for human generations).

However, many people who hate evolutionary psychology do so for irrational reasons

Evolutionary psychology is fine for intellectual masturbation, but we should strongly question its place as an actual science. However, many of its loudest critiques are based on emotional and political responses, rather than on the quality of the academic content.

Consider Megan McArdle’s critique of Sex at Dawn for The Atlantic. She writes:

“For example, like a lot of evolutionary biology critiques, this one leans heavily on bonobos (at least so far).  Here's the thing:  humans aren't like bonobos. And do you know how I know that we are not like bonobos?  Because we're not like bonobos.”

(Emphasis in original). 

Although I am sure Ms. McArdle is more articulate in other matters, it is true that when our beliefs are challenged, we are quick to say that scientific inquiry into the matter in question is useless.

Evolutionary psychology stirs up a political hornet’s nest. If we believe that our minds evolved to solve problems of the Pleistocene and have remained largely unchanged, this suggests that our minds have little capacity to change. Therefore, we can do little about real social problems such as war, racism and rape.

As Steven Pinker points out, ignoble tendencies do not have to lead to ignoble behavior. In other words, what “is” is not the same as what “ought”. The confusion between these two concepts comes from a fallacy confounding what is natural with what is good. Which leads me to my last problem with Dan Savage’s promotion of this book…..

Things that are natural are not necessarily desirable

Let’s step back and assume for a moment that the science of evolutionary psychology was solid, and that Ryan’s hypothesis about the polyamorous nature of humans was true. There would still be a major problem with Dan Savage’s use of this book to endorse polyamorous relationships: just because some behavior is fundamental to the nature of human beings does not mean that it’s a desirable state for current human beings.

Let me be clear on this point - I am not saying that humans shouldn’t be polyamorous. I believe consenting adults should do whatever they like. However, I am saying that the “naturalness” of polyamory does not inform its desirability.

Savage and Ryan are implicitly stating that since polyamory occurs throughout animal species and in human evolutionary history, it is natural. OK, but so are war, conquest, exploitation and rape and we do not condone these.

Dan, you are a smart guy…. Don’t get sucked in to poor science just because it tells a compelling story that you want to believe!

Mind reading?

I first volunteered to be a participant in an fMRI study as a wide-eyed college freshman ten years ago. I was so excited to get to see a picture of my brain, but once I was tightly packed into the scanner, a few worries entered my mind: would it turn out that I had a tumor, or was one of those people with half a brain? Would the experiment show that I’m not very smart, or vulnerable to mental illness? Would the graduate student administering the experiment know what I was thinking?

While my concerns were rather common, they were also rather unfounded. fMRI is rather good at predicting what you are thinking about in laboratory situations where you are given a very short list of things to think about. For example, an fMRI scan can predict whether you are thinking about a face or a place as the mental imagery for places and faces recruits different brain areas. And it is of note that this happens when your subjects are willing and able to think only about faces or houses for a 20 second run inside the scanner.

Earlier, I wrote a little about the analysis of fMRI data. The kind of inferences researchers make in these kinds of studies is in the form of “what area of the brain is more active for task 1 compared to task 2?” In prediction, the question becomes “given a pattern of brain activity, what was the participant seeing/hearing/doing?” Early prediction techniques relied on correlation: a voxel was predictive if its activity to a particular stimulus in one run was more highly correlated with activity to the same stimulus from another run than to activity from a different stimulus. More modern prediction studies make use of machine learning and statistical classifiers such as linear discriminant analysis, and support vector machines in particular. This approach to data analysis has been both popular and fruitful, particularly for vision research. I recommend this review for more on the state-of-the-art.

As impressed as we tend to be with both math-y and brain-y things, it is important to remember that we are still not able to predict arbitrary patterns of brain activity. When you are lying there in the scanner, it is possible to determine if you are daydreaming, but not the contents of your daydream. I have a particular concern over the over-selling of these studies in the titles of both lay and academic papers. Although the technology for fMRI decoding is advancing rapidly, I also do not see the scanner as a place where future civil liberties go to die. Getting readable data from the scanner requires your participant to be very co-operative, so its use as an interrogation device is limited. But for you paranoid types, perhaps you should consider some metal dental work… or even a hair tie!

New York Times' retraction of Alzheimer's test

Last month, an article in the New York Times proudly proclaimed a new 100% accurate test for the prediction of future Alzheimer’s disease. (This is the part where you should scroll down and read the retraction.).

Confused about the issue? Don’t feel bad – many doctors also fail at this type of reasoning.

Let’s say for the sake of argument that I tell you that I have a brand-new medical test. It’s called the “sleep gives you cancer” test, and it is one question long: Do you sleep? If you answered “yes”, then you will get cancer! As everyone who has ever had cancer has slept, this test is (according to the logic of the New York Times) 100% accurate. But, as a smart reader you will now tell me that my test isn’t so great because there are plenty of people in the world who sleep all the time and have never had cancer.

A test must do 2 things in order to be accurate: it must predict which people will get the disease (this is called sensitivity in the medical literature, and hit rate in psychology), and it also must predict which people won’t (the specificity in medical-speak, correct rejections in psych-speak). The test in question had a 100% sensitivity (everyone in their sample who later got Alzheimer’s tested positive), but 36% of people in the sample who didn’t get Alzheimer’s also tested positive.

So, how good is this test really?  Fortunately, some useful math exists to help us figure this out. Let’s say we have 1000 55-year olds. We know that 10% of them will develop Alzheimer’s by age 60. We give all 1000 people this test, and wait 5 years.  Looking at our sample, we’ll find that all 100 patients with Alzheimer’s tested positive for the test, as well as 324 (36%) of the non-Alzheimer group. Therefore, if one participant tested positive for the test there is only a 100/424 chance that s/he will have AD.

We also need to examine how useful an Alzheimer’s prediction test would be because, as of this writing, there isn’t a whole lot that can be done for AD. As pointed out here, the test described in the New York Times is based on a painful and invasive spinal tap, which makes the cost-benefit ratio quite large. However, there exist several predictive tests for AD based in neuroimaging that are less invasive.  However, given the high degree of uncertainty in the tests, coupled with the lack of meaningful therapeutic options spells years of needless anxiety for patients and families, in the opinion of this writer.

Why scientists aren’t going to find my neurotic gene any time soon

There has been some talk lately about a recent study that found that no robust statistical relations could be found between the whole genome and a standard personality test.

Seem surprising? Not so much.

So, a gene codes a protein. What does the protein do? Such a great number of things that it’s difficult to even list: a protein can become a structural element of a cell (such as actin or myosin that make up muscle tissue), or they can become neurotransmitters or other messengers in a complicated cascade of signaling events. For example, the PubMed description of the protein neuregulin 1 (statistically associated with schizophrenia, and high creativity) starts with “The protein encoded by this gene was originally identified as a 44-kD glycoprotein that interacts with the NEU/ERBB2 receptor tyrosine kinase to increase its phosphorylation on tyrosine residues.” Aside from the technical language, the first description of this protein is the relationship that it has to another protein, to provide a specific biochemical context (phosphorylation).

Alright. So a gene codes a protein, and this protein is a widget that works in concert with other such widgets in a particular biochemical and environmental context.  Does it even make sense to say that there is a gene for a complex behavioral phenomenon such as schizophrenia, depression, or a neurotic personality?  Not very much, really. Not at least in the sense of “I have this computer for writing this blog post”. Kenneth Kendler points out the lack of causal link further by making the following analogy:

“A jumbo jet contains about as many parts as there are genes in the human genome. If someone went into the fuselage and removed a 2-foot length of hydraulic cable connecting the cockpit to the wing flaps, the plane could not take off. Is this piece of equipment then a cable for flying?”

While most people would answer that no, the tube does not directly cause the jet to fly, this is the exact same logic that is used when we try to find a gene for X.

The issue is that we expect genes to have very lawful 1:1 correspondences with specific traits because in school we learned about Mendel’s pea pods, or cystic fibrosis, or Huntington’s disease that show such a relationship. This type of inheritance seems to be the exception, rather than the rule. There exists a wide distribution of association strengths between a single gene and a particular outcome. Scientists express this strength using a statistic called the odds ratio. Briefly, this is the odds that someone with gene A will have disease X, versus the odds that someone without gene A will have disease X. For a completely Mendelian disease (one like cystic fibrosis that cannot be contracted through the environment), the odds ratio is infinite because if you have the gene, you will always have the disease, and if you don’t have the gene, you never will. Statistical associations that we perceive as strong (such as the link between heavy smoking and lung cancer) have an odds ratio of about 20.  Psychiatric associations, on the other hand have an odds ratio of 1-2.  In other words, don’t go rush out to get genetically tested for depression. It won’t do you much good.

 

Partially, this lack of association is due to complex interactions between genes and the environments. For example, people with a particular variant of a serotonin transporter are more likely to experience depression, but only in the context of having experienced a stressful life event. 

A possible exception to the single-genes-don’t-change-behavior-in-isolation rule might be the COMT gene. This gene makes the enzyme that breaks down several neurotransmitters in the brain, including dopamine.  Like many genes, individuals may have different variations (or alleles) of the gene.  However, unlike alleles that change an individual’s hair color, different alleles in the COMT gene have been associated with striking differences in cognitive function.  Incredibly, these differences arise due to a single amino acid difference in the enzyme!  Substituting valine for methionine at position 158 in the gene is associated with a host of poorer psychological outcomes.  As each person inherits one copy of the gene from each of his parents, individuals can either have two valines, two methionines, or one of each.  Interestingly enough, the number of valines correlates with the degree of negative outcome.  For example, a 2008 study was conducted in which people recorded the events that were taking place in their lives, and rated how positive these events were.  The authors found that valine-valine individuals found a very pleasant event only as positive and methionine-methionine people found a sort-of positive event.  Given these results, it is easy to see how these individuals have difficulties with major depression and addiction.

 

So, genes code proteins which work together in an incredibly complex biochemical context created by other genes, the environment, and interactions of the genes, the biochemical milieu and the environment. Instead of asking ourselves why we haven’t found the gene for X, we should really be asking ourselves why we keep asking that question.