Blog

Dealing with Lazy Survey Takers

A tweet by @thefarmbabe earlier this week has renewed interest in my survey result from back in January 2015, where we found more than 80% of survey respondents said they wanted mandatory labels on foods containing DNA. For interested readers, see this discussion on the result, a follow-up survey where the question was asked in a different way with essentially the same result, or this peer-reviewed journal article with Brandon McFadden where we found basically the same result in yet another survey sample. No matter how we asked this question, it seems 80% of survey respondents say they want to label foods because they have DNA.

All this is probably good motivation for this recent study that Trey Malone and I just published in the journal Economic Inquiry. While there are many possible reasons for the DNA-label results (as I discussed here), one possibility is that survey takers aren’t paying very close attention to the questions being asked.

One method that’s been around a while to control for this problem is to use a “trap question” in a survey. The idea is to “trap” inattentive respondents by making it appear one question is being asked, when in fact - if you read closely - a different question is asked. Here are two of the trap questions we studied.

trapquestions.JPG

About 22% missed the first trap question (they did not click “high” to the last question in figure 2A) and about 25% missed the second question (the respondent clicked an emotion rather than “none of the above” in question 2B). So far, this isn’t all that new.

Trey’s idea was to prompt people who missed the trap question. Participants who incorrectly responded were given the following prompt, “You appear to have misunderstood the previous question. Please be sure to read all directions clearly before you respond.” The respondent then had the chance to revise their answers to the trap question they missed before proceeding to the rest of the survey. Among the “trapped” respondents, about 44% went back and correctly answered the first question, whereas about 67% went back and correctly answered the second question. Thus, this “nudge” led to an increase in attentiveness among a non-trivial number of respondents.

After the trap questions and potential prompts, respondents subsequently answered several discrete choice questions about which beer brands they’d prefer at different prices. Here are the key findings:

We find that individuals who miss trap questions and do not correctly revise their responses have significantly different choice patterns as compared to individuals who correctly answer the trap question. Adjusting for these inattentive responses has a substantive impact on policy impacts. Results, based on attentive participant responses, indicate that a minimum beer price would have to be substantial to substantially reduce beer demand.

In our policy simulations, we find a counter-intuitive result - a minimum beer price (as implemented in some parts of the UK) might actually increase alcohol consumption as it leads to a substitution from lower to higher alcohol content beers.

In another paper in the European Review of Agricultural Economics that was published back in July, Trey and I proposed a different, yet easy-to-interpret measure of (and way to fix) inattention bias in discrete choice statistical models.

Taken together, these papers show that inattention is a significant problem in surveys, and that adjusting results for inattention can substantively alter one’s results.

We haven’t yet done a study of whether people who say they want DNA labels are more or less likely to miss trap question or exhibit other forms of inattention bias, but that seems a natural question to ask. Still, inattention can’t be the full explanation for absurd label preferences. We’ve never found inattention bias as high as the level of support for mandatory labels on foods indicating the presence/absence of DNA.

GMO labels - not as bad as I thought

Science Advances (the open-access version of Science Magazine) just published a paper I co-authored with Jane Kolodinsky from the University of Vermont.  I suspect the paper's findings may raise a few eyebrows, as we find that opposition to GMOs in Vermont fell relative to that in the rest of the U.S. after mandatory labeling was adopted in that state.

Some background context might be useful here.  Several years go, I was decidedly in the camp that thought imposition of mandatory labels would cause people to be more concerned about GMOs because it would signal that something was unsafe about the technology.  Prominent scholars such as Cass Sunstein have argued the same.  A few years ago, Marco Costanigro and I put this hypothesis to the test in a paper published by Food Policy, and we found little evidence (in a series of survey-based experiments) that the label per se neither increased or decreased aversion to GMOs.  So, I was less convinced that this particular argument against mandatory GMO labeling was valid, but I was still unsure.  

Then, last summer at the annual meetings of the Agricultural and Applied Economics Association (AAEA), I saw Jane present a paper based on survey data she collected in Vermont before and after mandatory labels went into place there.  Her data suggested opposition to GMOs fell at faster rate after mandatory labels were in place.  Despite my findings in Food Policy, I remained dubious and Jane and I went back and forth a bit on the robustness of her findings. 

I'd been in enough conversations with Jane to know that we had different philosophical leanings about the desirability of GMOs, but this was an empirical question, so we put our differences aside and decided to join our data and put the hypothesis to the test.  Through the Food Demand Survey (FooDS), I had been collecting nationwide data on consumer's concerns about GMOs, and I suggested we combine our two sets of data and do a true "difference-in-difference" test: Did the difference in concern among consumers in VT and the result of the US increase or decrease after mandatory labeling was adopted in VT?

Our article in Science Advances has the result:

This research aims to help resolve this issue using a data set containing more than 7800 observations that measures levels of opposition in a national control group compared to levels in Vermont, the only U.S. state to have implemented mandatory labeling of GE foods. Difference-in-difference estimates of opposition to GE food before and after mandatory labeling show that the labeling policy led to a 19% reduction in opposition to GE food. The findings help provide insights into the psychology of consumers’ risk perceptions that can be used in communicating the benefits and risks of genetic engineering technology to the public.

One important caveat should be mentioned here.  Our result does NOT suggest people will suddenly support GMOs once mandatory labels are in place.  Rather, our findings suggest that people will be somewhat less opposed than they were prior to labels.  I mention this because in the wake of my paper with Marco in Food Policy some of the media's interpretation of our results (such as that of the New York Times editorial board), could have been construed as suggesting that imposition of mandatory labels would not cause economic harm.  That may or may not be true.  But, this new study suggest that labels per se may in fact reduce opposition.

It was great to work with Jane on this project, and for me it was a good lesson to test your beliefs, particularly when there are theoretical reasons that could support the opposing point of view.

I'll end with a key graph from the paper.

gmo_labels.JPG

Defining Meat

Meat and livestock producers are taking notice of the the rising interest in lab-based, cultured, and plant-based "meat."  Some of the larger meat packers and producers have chosen to invest in these new start-ups.  Other producers, facing the competitive threat, are turning to the legal system.  The U.S. Cattlemen's Association (not to be confused with the National Cattlemen's Beef Association) has officially petitioned the USDA to:

limit the definition of beef to product from cattle born, raised, and harvested in the traditional manner. Specifically, [the USDA Food Safety Inspection Service] should require that any product labeled as “beef” come from cattle that have been born, raised, and harvested in the traditional manner, rather than coming from alternative sources such as a synthetic product from plant, insects, or other non-animal components and any product grown in labs from animal cells.

The state of Missouri already passed a similar law (although it has yet to be signed by the governor).

The labeling requests are in keeping with a long list of "standards of identity" whereby the government defines how certain words can be used on food labels and in marketing.  The stated purpose of the laws are to protect consumers and to prevent consumers from being misled.  In some cases cases, I suspect they standards have done just that.  However, in other cases, the rules can be used by incumbent firms to ward off competition from potentially innovative entrants.  In one particularly egregious example, a small creamery marketing "natural milk" didn't want to add vitamin A to its milk.  However, because the standard of identity say that skim milk contains vitamin A, a judge ruled they must label their milk "imitation skim milk" even though they added literally nothing to the milk (the ruling was later overturned).  Another recent example is when Hampton Creek was told they couldn't label their product mayonnaise because it didn't contain eggs.  I wrote about that case here, and concluded by saying:

Ultimately, I think there are good arguments on both sides of this case, and it isn’t obvious what would be the consequences of the unraveling of these sorts of “food purity” laws. Sometimes it’s hard to know when consumer protection becomes protectionism.

In the case of beef, I am a bit skeptical that consumers will be mislead by the start-up meat alternatives.  Why?  These aren't generic products being sold by companies trying to water down or adulterate a product with cheaper inputs.  These are branded products created by firms whose whole marketing strategy is to tell people their product is NOT beef.  Here's a picture I took with my cellphone at a restaurant selling the Impossible Burger, where plain as day its says "Meat from Plants." 

impossibleburger2.JPG

Here is an image of package of Beyond Meat.  Again, plain as day, it says "Plant-Based Burger."

beyondmeat.JPG

In neither of the cases above, do the companies claim to be "beef" in the ads or packaging.  So, in a lot of ways, I suspect the calls for standards of identity may be much to do about nothing. 

Even without the identity standards, it is not as if consumers are totally unprotected.  If they are, in fact, misled, the legal system offers possible remedy. As witnessed by the numerous lawsuits over the use of the word "natural," I suspect there are plenty of lawyers out there willing to help a consumer who can show they've experienced damages.   

Impacts of health information on perceived taste and affordability

The journal Food Quality and Preference just released a new paper I co-authored with Jisung Jo, a former student who now works at the Korea Maritime Institute.

Here is the motivation for the work:

One of the key mechanisms policy makers have utilized to encourage healthier eating is the provision of information via nutritional labels. However, research has shown that the provision of health information does not necessarily increase consumption of healthy foods ... A possible reason for the largely ineffectual impact of nutritional labeling might be because health information not only updates consumers’ health perceptions but also affects other perceptions, such as taste and affordability, which are the primary drivers of consumer purchase behavior

In other words, if you see a new labeling indicating a food is healthier than you previously thought, do you now think it will be less tasty?  Or more expensive?  

To explore this issue, we surveyed consumers in three different countries (US, China, and Korea).  We showed consumers a picture of a food item and asked them to rate the item, on simple scales, in terms of perceived taste, health, affordability, and purchase intention.  We did this for 60 diverse food items. Then, the ratings of all 60 foods was repeated after the subjects had received information about each food item’s healthiness, which was conveyed via a "traffic light" labeling system (green=healthy, yellow=medium healthiness, red=unhealthy).   Here's an example of one of the questions asked before and after the information:

jisungFQP1.JPG

Unsurprisingly, the provision of "green" labels tended to increased perceived healthiness and the provision of "red" labels tended to reduce perceived healthiness.  Of more interest is how these labels affected perceptions of taste, affordability, and ultimately purchase intentions.  

Unexpectedly, we found that providing information that a food was healthier than people previously thought tended to increase perceived taste.  People also tended to think items that are less healthy than previously thought will ultimately be less expensive.

We created the following graph to look at how projected changes in purchase intentions (after provision of health information) would change if one ignores the fact that health information also affects perceived taste and affordability.

jisungFQP2.JPG
Across all scenarios and in all three countries, we find that negative health information has the biggest effects on purchase intention changes. Intriguingly, the average purchase intention in scenario B is larger than that in scenario A. The values for scenario D are the same as the actual average of purchase intention (since they are just the model evaluated at the mean effect changes of all variables included in the model). Comparing the purchase intention changes as one moves from scenario A to D shows the effect of ignoring integrated health-taste-affordability perceptions.

Overall, this research underscores the need to understand how labels which convey health information might also alter other perceptions related to taste and affordability.

Are Organic and Non-GMO Labels Substitutes or Complements?

For the first time today, I saw the following label on a packaged food.

organicisnongmo.JPG

In a way, the label seems a little odd.  An organic seal on a product should already convey to consumers that the ingredients came from a process that excluded GMOs.  However, the very presence of the label suggests many consumers may not be aware of this fact.  

I have a paper with Brandon McFadden forthcoming in journal Applied Economic Perspectives and Policy (sorry, I don't yet have a link to the paper on the AEPP's website; I'll pass it along when I get the link and discuss the whole paper in more detail).  In the paper we delve into this issue and others.  Here's part of the motivation.  

It appears that organic organizations are concerned that consumers perceive non-GM and organic labels to be substitutes. Although many organic food companies supported the general idea of mandatory labeling, now that the policy has passed, organic producers have expressed concern that non-GM verification may be perceived as a substitute for the more expensive and encompassing organic certification. For examples, California Certified Organic Farmers (CCOF) initiated a campaign “Organic is Non-GMO and More” to highlight the differences in the two claims, and the Organic Trade Association (OTA) emphasizes, “Organic = Non-GMO…and so much more!!” Despite these concerns, little is known about the extent to which the two most common non-GM labels, USDA Organic and Non-GMO Project, are demand substitutes or complements. Whether the labels are demand substitutes or complements can be determined, in our context, by investigating whether WTP [willingness-to-pay] is supra- or sub-additive when the labels are combined. If the premium for displaying both labels is less than the sum of individual premiums for each label, then the two labels must be providing some of the same underlying characteristics of value to the consumer and implies the two labels are substitutes. By contrast, if the premium for displaying both labels is greater than the sum of individual premiums, then the two labels are complements and provide more value when provided together.

We ultimately find that products with the organic seal and products with the non-GMO verified seal are indeed demand substitutes.  Here's one paragraph related to those results:

For apples, the results revealed large and statistically significant substitution effects for Non-GMO and USDA Organic labels. In fact, results indicated that the two are almost perfect substitutes as WTP [willingness-to-pay] premiums for apples with both Non-GMO and USDA Organic labels roughly the same as WTP premiums for apples that display only one label. This result is made obvious by the third column of results. The WTP premium for apples with the Non-GMO label only (vs text label) is $0.446, the WTP premium for apples with the organic label only (vs text label) was $0.474, and the WTP premium for apples with both Non-GMO and USDA Organic labels was $0.446+$0.447-$0.461=$0.432, which is actually lower than when either label is present in isolation.

Because it is more costly to be organic than non-GMO (since the latter is a subset of the former), it is easy to see why many food companies would want to add the additional label that "Organic is non-GMO and more".