Do Survey Respondents Pay Attention?

Imagine taking a survey that had the following question. How would you answer?

If you answered anything but "None of the Above", I caught you in a trap.  You were being inattentive.  If you read the question carefully, the text explicitly asks the respondent to check "None of the Above."  

Does it matter whether survey-takers are inattentive?  First, note surveys are used all the time to inform us on a wide variety of issues from who is most likely to be the next US president to whether people want mandatory GMO labels.  How reliable are these estimates if people aren't paying attention to the questions we're asking?  If people aren't paying attention, perhaps its no wonder they tell us things like that they want mandatory labels on food with DNA.

The survey-takers aren't necessarily to blame.  They're acting rationally.  They have an opportunity cost of time, and time spent taking a survey is time not making money or doing something else enjoyable (like reading this post!).  Particularly in online surveys, where people are paid when they complete the survey, the incentive is to finish - not necessarily to pay 100% attention to every question.

In a new working paper with Trey Malone, we sought to figure whether missing a "long" trap question like the one above or missing "short" trap questions influence the willingness-to-pay estimates we get from surveys.  Our longer traps "catch" a whopping 25%-37% of the respondents; shorter traps catch 5%-20% depending on whether they're in a list or in isolation.  In addition, Trey had the idea of going beyond the simple trap question and prompting people if they got it wrong.  If you've been caught in our trap, we'll let you out, and hopefully we'll find better survey responses.  

Here's the paper abstract.

“This article uses “trap questions” to identify inattentive survey participants. In the context of a choice experiment, inattentiveness is shown to significantly influence willingness-to-pay estimates and error variance. In Study 1, we compare results from choice experiments for meat products including three different trap questions, and we find participants who miss trap questions have higher willingness-to-pay estimates and higher variance; we also find one trap question is much more likely to “catch” respondents than another. Whereas other research concludes with a discussion of the consequences of participant inattention, in Study 2, we introduce a new method to help solve the inattentive problem. We provide feedback to respondents who miss trap questions before a choice experiment on beer choice. That is, we notify incorrect participants of their inattentive, incorrect answer and give them the opportunity to revise their response. We find that this notification significantly alters responses compared to a control group, and conclude that this simple approach can increase participant attention. Overall, this study highlights the problem of inattentiveness in surveys, and we show that a simple corrective has the potential to improve data quality. ”

Experimental Auction Summer School 2016

Applications are now being accepted for a summer school on Experimental Auctions that is organized by organized by Maurizio Carnavari and co-taught with Rudy Nayga and Andreas Drichoutis.  This will be our 5th installment.  In the past we've had the summer school near Bologna Italy, but last year we venturing out to Crete, Greece with great success.  This year's course is scheduled from July 5 to July 12, 2016 in Catania, Sicily.

Experimental auctions are a technique used to measure consumer willingness-to-pay for new food products, which in turn is used to project demand, market share, and benefits/costs of public policies. Two weeks ago, I got to meet with a company in Amsterdam, Veylinx (see my previous mention of them here), who is using the method in an online format at a commercial level for marketing research. The content of the course is mainly targeted toward graduate students or early career professionals (or marketing researchers interested in learning about a new technique).  You can find out more and register here.

Here's last year's class in Crete.

And, of course, one shouldn't forget what is perhaps the most valuable part - the after-class networking and brainstorming sessions!

Impacts of Agricultural Research and Extension

About a month ago, I posted on some new research suggesting decline rates of productivity growth in agriculture.  Last week at a conference in Amsterdam, I ran into Wally Huffman from Iowa State University, and knowing he's done work in this area, I asked him if he had any thoughts on the issue.  As it turns out, along with Yu Jin he has a new paper forthcoming in the journal Agricultural Economics on agricultural productivity and the impacts of state and federal spending on agricultural research and extension.  

Jin and Huffman also find evidence of a slowdown in productivity growth, writing: 

“We find a strong impact of trended factors on state agricultural productivity of 1.1 percent per year. The most likely reason is continued strong growth in private agricultural R&D investments. The size and strength of this trend makes it unlikely for average annual TFP growth for the U.S. as a whole to become negative in the near future. However, for two-thirds of the states, the forecast of the mean ln(TFP) over 2004-2010 is less than trend. The primary reason is under-investment in public agricultural research and extension in the past. For public agricultural research where the lags are long, it will be impossible for these states to exceed the trend rate of growth for TFP in the near future.”

They also find large returns to spending on agricultural research, and even larger returns to spending on extension.  They find the following:

“For public agricultural research with a productivity focus the estimated real [internal rate of return] is 67%, and for narrowly defined agricultural and natural resource extension is over 100%. Stated another way, these public investment project could pay a very high interest rate (66% for agricultural research and 100% for extension) and still have a positive net present value. Hence, these [internal rate of return] estimates are quite large relative to alternative public investments in programs of education and health. In addition, there is no evidence of a low returns to public agricultural extension in the U.S., or that public funds should be shifted from public agricultural extension to agricultural research. In fact, if any shifting were to be recommended, it would be to shift some funds from public agricultural research to extension. ”

The paper includes a couple really interesting graphs on research spending and extension employment over time.  First, they show that for four major agricultural states, real spending on agricultural research peaked in the mid 1990s. 

And, while extension staff has declined in some states, it hasn't in others.  

The Behavioral and Neuroeconomics of Food and Brand Decisions

That's the title of a special issue I helped edit with John Crespi and Amanda Bruce in the latest issue of the Journal of Food and Agricultural Industrial Organization.  

Here's an excerpt from our summary:

“To economists interested in food decisions, progress seen in other fields ought to be exciting. In the articles for this special issue, we gathered information from a wide range of research related to food decisions from behavioral economics, psychology, and neuroscience. The articles, we hope, will provide a useful reference to researchers examining these techniques for the first time…The variety of papers in this special issue of JAFIO should provide readers with a broad introduction to newer methodological approaches to understanding food choices and human decision-making”

A complete listing of the authors and papers are below (all of which can be accessed here)

•       The Behavioral and Neuroeconomics of Food and Brand Decisions: Executive Summary
o   Bruce, Amanda / Crespi, John / Lusk, Jayson

•       Cognitive Neuroscience Perspectives on Food Decision-Making: A Brief Introduction
o   Lepping, Rebecca J. / Papa, Vlad B. / Martin, Laura E.

•       Marketing Placebo Effects – From Behavioral Effects to Behavior Change?
o   Enax, Laura / Weber, Bernd

•       The Role of Knowledge in Choice, Valuation, and Outcomes for Multi-Attribute Goods
o   Gustafson, Christopher R.

•       Brands and Food-Related Decision Making in the Laboratory: How Does Food Branding Affect Acute Consumer Choice, Preference, and Intake Behaviours? A Systematic Review of Recent Experimental Findings
o   Boyland, Emma J. / Christiansen, Paul

•       Modeling Eye Movements and Response Times in Consumer Choice
o   Krajbich, Ian / Smith, Stephanie M.

•       Visual Attention and Choice: A Behavioral Economics Perspective on Food Decisions
o   Grebitus, Carola / Roosen, Jutta / Seitz, Carolin Claudia

•       Towards Alternative Ways to Measure Attitudes Related to Consumption: Introducing Startle Reflex Modulation
o   Koller, Monika / Walla, Peter

•       I Can’t Wait: Methods for Measuring and Moderating Individual Differences in Impulsive Choice
o   Peterson, Jennifer R. / Hill, Catherine C. / Marshall, Andrew T. / Stuebing, Sarah L. / Kirkpatrick, Kimberly

•       A Cup Today or a Pot Later: On the Discounting of Delayed Caffeinated Beverages
o   Jarmolowicz, David P. / Lemley, Shea M. / Cruse, Dylan / Sofis, Michael J.

•       Are Consumers as Constrained as Hens are Confined? Brain Activations and Behavioral Choices after Informational Influence
o   Francisco, Alex J. / Bruce, Amanda S. / Crespi, John M. / Lusk, Jayson L. / McFadden, Brandon / Bruce, Jared M. / Aupperle, Robin L. / Lim, Seung-Lark

Big Fat Surprise

I just finished reading Nina Teicholz’s best selling book The Big Fat Surprise, which takes issue with our long-held belief that low-fat diets in general, and diets free of animal fat in particular, best promote good health.  

It’s been an enjoyable read, and the history of the development of our dietary beliefs and guidelines is both fascinating and eye opening.  There is a bit of a tendency in the book for the author to nit pick any study which doesn’t support her hypothesis without applying the same skepticism of those studies which do support it, but overall, she makes a compelling case.  I probably found chapter 10 on "Why Saturated Fat is Good For You" most interesting in that regard.  Teicholz lays bare the sad state of affairs associated with the science behind much of the nutritional advice we’re given.  One takeaway is that we really don’t know as much as is often presumed about what sorts of diets increase/decrease chances or heart attack or cancer.  

There is one nit I want to pick with a phrase in Teicholz’s book.  It is a technical one, but because it is the sort of thing I expect my students to fully understand, I'll delve into it.  On page 167 of the paperback version she writes (about an epidemiological study finding no relationship between breast cancer and consumption of dietary fat), “These conclusions were all associations.  But although epidemiology cannot demonstrate causation, it can be used to reliably show the absence of a connection.” (the emphasis is hers)

That claim is patently false (I'm presuming by "connection" she means "causation").  The trouble with the sort of correlation analysis used in many epidemiology studies is that of missing variables.  We can't observe everything about people's behaviors or about the effects of dietary changes, and that results in "omitted variable bias."  That bias can inflate or reduce the size of a measured effect.   In fact, contrary to Teicholz's claims, omitted variable bias can make a "real" effect look like nothing.

Wikipedia describes the problem, but similar treatments can be found in almost any introductory econometrics textbook.     

Suppose we have the following true relationship:

y=b0 + b1*x + b2*z + e

where y is the chance of breast cancer among women, x is amount of fat consumed, and z is a personality trait reflecting the person's overall health conscientiousness.  The "true" relationship we want to know is given by b1.  

But, suppose we only observe y and x and we don't observe z.  Also suppose that z is related to x in the following way: z = a0 + a1*x + u.  Substituting this equation into the first means that when the epidemiologist runs their analysis they calculate:

y = b0 + b1*x +b2*(a0 + a1*x + u)+ e

or, re-writing:

y = b0 + b2*a0+ (b1+b2*a1)*x + b2*u+e.

So, the researcher looks at the relationship between x and y, and thinks they're estimating the "true" effect b1, but in reality, they're estimating the effect (b1+b2*a1), which could be larger or smaller than b1.  

Suppose b2 takes the positive value of +1.5 (more conscientious women are less likely to develop breast cancer) and a1 is also positive and takes the value of +2 (more conscientious women pay more attention to all that health advice and eat less fat).  This means the effect b2*a1 is positive at the value of +3.  But b1 could be negative (more fat = more breast cancer).  Say, b1=-3.  If the positive effect of b2*a1=+3 outweights the negative effect of b1=-3, so the estimated effect is 3-3=0.  It will look like there is no effect even though there really is one.  Even if the effects don't precisely outweigh each other, the estimated effect could be small enough that it the research concludes it isn't statistically different from zero.

Now, I'm not saying that there is a relationship between fat consumption and breast cancer - rather, I'm just making a conceptual point that omitted variables can result in upward or downward bias.  What I can more confidently say is that only the last part of Teicholz's claim is right:  "epidemiology cannot demonstrate causation."  

Now, there are regression methods that can get us much closer to the truth, but I don't often see these used in epidemiology studies.  In economics, the so-called "credibility revolution" has led to more specification testing and attention to causal-identification using instrumental variables, discontinuity designs, differences-in-differences, and others.  A good introduction to the topics and methods is given in Mostly Harmless Econometrics.