Do Scientists Mislead?

"Yes" seems to be the unfortunate conclusion that stems from this paper by Mark Cope and David Allison published in the International Journal of Obesity.  ​Scientists may not (or may) distort their own research findings, but Cope and Allison show, pretty convulsively, that there is a general pattern of distorting the findings of others.   

They attribute this to a "white hat bias":​

which we define to be bias leading to distortion of research-based information in the service of what may be perceived as righteous ends.

Cope and Allison found two studies related to soda consumption that:

had both statistically and non-statistically significant results on body-weight, body mass index (BMI) or overweight/obesity status which allowed future writers to potentially choose which results to cite, and were also widely cited, permitting quantitative analysis of citations.

​Then, they looked at how other scientists subsequently cited the findings in their published papers.  Did they focus on the negative findings (that soda doesn't affect weight, etc.) or the positive findings (that soda does affect weight, etc.): ​

The majority, 84.3% for [2] and 66.7% for [3], described results in a misleadingly positive way to varying degrees (i.e., exaggerating the strength of the evidence that [nutritively-sweetened beverage] reduction showed beneficial effects on obesity outcomes). Some were blatantly factually incorrect in their misleading statements, describing the result as showing an effect for a continuous obesity outcome whereas no statistically significant effect for continuous obesity outcomes was observed. In contrast, only four papers (3.5%) were negatively misleading (i.e., underplayed the strength of evidence) for [2] and none were negatively misleading for [3]. Only 12.7% and 33% of the papers accurately described complete overall findings related to obesity outcomes from [2] and [3], respectively.

​

They went on to document a similar pattern for studies on the effects of breastfeeding. They ​conclude:

Evidence presented herein suggests that at least one thing has been demonized ([nutritively-sweetened beverage] consumption) and another sanctified (Breastfeeding), leading to bias in the presentation of research literature to other scientists and to the public at large, a bias sufficient to misguide readers. Interestingly, while many papers point out what appear to be biases resulting from industry funding, we have identified here, perhaps for the first time, clear evidence that white-hat biases can also exist in opposition to industry interests.

Do Small Reductions in Caloric Intake add up to Big Changes in Weight?

The answer is: probably not.​

​This is important question because there are many studies finding that various interventions (from fat taxes to menu labels) have very (though sometimes statistically significant) small effects on caloric intake.  Proponents of the policies are often undeterred - and say things like "well, a 20 kcal reduction every day can really add up to big weight loss over time."

As I've already discussed, some of this sort of analysis ​is based on the faulty logic that 3500kcal = 1lb.  But, as was mentioned in that post, our body does not react linearly to caloric changes in the fashion implied by this formula.  

Now, there's more on this topic by Trevor Butterworth in a well-written and catchy-titled post ​Sex And Lies! The Iffy Science Of Measuring Calories.  Here is a key excerpt:  

Hall was responsible for filling in the crucial measurements that elucidated one of the most widespread myths highlighted by Allison et al.: the idea that small, consistent changes in energy intake or expenditure will, over time, lead to large changes in weight. The assumption appears to have been based on the 1958 calculation by Max Wishnofsky that one pound of body fat gained or lost is equal to 3,500 kilocalories. This seemed to give people a convenient way to estimate weight loss through diet or exercise, while promising extremely convenient results. If you simply knocked off a 100 kilocalories from your energy intake each day—a ten-minute jog, or a mile walk—you'd end up losing over 50 pounds in five years. Little wonder that early proposals for soda and fat taxes promised to save Americans from themselves: pay a little more, consume a little less, watch a lot of weight disappear in a few years.

Hall first heard the claim listening to a dietician make a calculation for an obese patient. His intuition told him that this calculation was incorrect and would lead to exaggerated weight loss predictions. When he asked for a reference, he was pointed to a nutrition and dietetics textbook. "I subsequently found the mistake everywhere I looked." People weren't stopping to think "about the dynamic interaction between energy intake and expenditure, which is complicated," he says. What they failed to take into account was that "the rate of weight loss changes over time and is primarily determined by the imbalance between energy intake and expenditure—a value that also changes over time." To radically simplify his model, this means that cutting calories in your diet leads to a decreasing calorie expenditure, which in turn slows weight loss until weight eventually plateaus after a few years. "Of course," says Hall, "cheating on your diet will cause your weight to plateau much sooner." In the case of soda taxes, Hall and researchers at the US Department of Agriculture showed how static modeling overstated weight loss by 346 percent after five years.

Does Sugar Consumption Drive Diabetes?

A recent article in the journal PLoS ONE by the anti-sugar crusader Robert Lustig and three other co-authors has created quite a stir by purporting to show that increased sugar consumption causes diabetes.  In the paper, the authors hold up just shy of saying "cause" but that is the inference drawn by many in the media (see for example this story in Bloomberg among other places) who say things like:

Excessive sugar consumption may be the main driver of a global rise in diabetes,

Moreover, on Mark Bittman's NYT Blog, the author, Lustig, is cited as saying:​

This study is proof enough that sugar is toxic. Now it’s time to do something about it.

There is no way a study like this (comparing differences across countries) can firmly establish causation.  So, at a minimum the study indicates an interesting (and perhaps suggestive) correlation that might warrant a randomized control trial.  Nonetheless, I was intrigued and wanted to check out the evidence for myself.  

The evidence by Lustig and colleagues comes by linking data on diabetes prevalence rates across countries (which I was able to easily find online here) and data from the UN FAO on the availability of calories from different food stuffs in different countries (after a bit of digging, I was also able to find it online here - go the the "food balance sheets").  After a bit of effort, I downloaded both data sets for the most recent years available, merged them, and checked out the claims made in the paper.  

At first blush, I find very similar results to the ones reported in the paper.  Holding constant total calories available, a simple linear regression shows that for every 100 kcal increase in sugar availability, the prevalence of diabetes goes up by 1.3 percentage points (say from 8.5% (the sample mean) to 9.8%).  The estimated equation is:  ​

(% with diabetes)=​1.067+0.013*(per-capita available sugar kcal)+0.001*(per capita total available kcal)

My estimate is a little higher than the one reported in the paper probably because I'm not controlling for other factors (like GDP, kcal intake from meat, etc.) as the authors did.  Moreover, I'm using data on diabetes from 2012 whereas the authors used 2011 and older data (note: I use data from 174 countries in my estimates).  The only coefficient significant at the p=0.05 level in the above equation is the 0.013 estimate associated with sugar.   

So far so good - the correlation is confirmed.  

But let's get to the nitty gritty of the interpretation.  The data is at the country level.  So, what this implies is that a country that increases per-capita sugar availability by 100kcal will tend to have a 1.3 percentage point increase in the percent of the population with diabetes. 

But, we don't really care about countries per se.  We care about people.  There are a lot more people in some countries than others.  ​In the data set, the range is from a low of 0.00066 million adults to 980 million adults.  Shouldn't this factor into the analysis?  If we care about how many people in the world have diabetes, we'd better pay a lot more attention to China than to Luxembourg.  

We know from the mini-scandal associated with the claim that small schools outperform larger ones (see one account here)​ that outcomes from small schools (or small countries) tends to be a lot more variable (with more outliers) than data from large schools (or large countries).  That's just basic statistics.  

Intuitively, we should want a larger country to count more than a smaller one.  After all, there are many more people in larger countries - so if we want to think about the prevalence of diabetes in the world (rather than the average prevalence rate across countries)​, we'd want to calculate a weighted average, where larger countries get more weight (because they have more people).  The more people, the higher the weight.

Likewise, when we want to run analyses like the one above, we want to give more weight to countries with more people.  We can do this by running a weighted regression, where each country gets a weight proportional to it's population size.  This converts the equation to one about how countries differ to one about how individuals differ.  ​Stated differently, the weighted regression places the estimates at the level of the individual (picked at random from any country) rather than the level of the country (picked at random from a group of countries).

Here is the equation I get when I weight by a country's adult population:​

(% with diabetes)=0.692+0.002*(total available sugar kcal)+0.002*(total available kcal)

Now, the effect of sugar falls dramatically (and most importantly, it is no longer statistically significant at standard levels; the p-value is 0.074).  A 100 kcal increase in per-capita sugar availability only increases the % with diabetes by 0.2 (rather than 1.3 as previously estimated).  Moreover, total energy from all sources is now significant and roughly the same magnitude as sugar.  Thus, what matters in this framework is total kcal from any food source.  Moreover this regression suggests that a sugar calorie is roughly the same as any other calorie insofar as affecting diabetes.    ​

The paper at PLoS ONE says "regressions are population weighted."  But, I'm wondering that is indeed the case.  It could be true.  I don't have access to all their data and I'm not including all their controls.  

I'm happy to share the data and SAS code with anybody who cares to see it.​

​********

Addendum

​The nice thing about the web is that you get feedback.  Here's an update.  The source that reports diabetes prevalence actually reported three measures.  In the regressions above, I used national prevalence (total number with diabetes divided by total population).  However, as indicated at the data source here, they also report some sort of age adjusted measure that is likely more useful in comparing across countries that might have different mean ages.  

​When I use this "IGT comparative prevalence" measure, as they call it, then I get exactly the opposite of the results mentioned above.  When the data are NOT weighted, the sugar coefficient is only 0.0019 (p-value 0.27).  But, when the data ARE weighted by adult population, the sugar coefficient is 0.01277 (p-value < 0.001).  

So, there is an interesting mix of things going on here between the population, weighting, and age adjustment.  Just out of curiosity, and for some robustness checks, I did two things.  First, I re-ran the "preferred" model with population weighting using "IGT comparative prevalence" diabetes but included population as an explanatory variable. When I do this, sugar is no longer statistically significant (the estimate is 0.00242 with a p-value of 0.107), but population is (the estimate suggests larger populations have lower diabetes prevalence).  I can't quite figure out what is going on here but there has to be something weird going on in the sense that the model is  weighting by population and the dependent variable (and independent variables) are per-capita (i.e., are divided by population), that might be producing some unexpected results.    

Second, I ran a quantile regression to see how the results hold up at the median (rather than the mean, which is more sensitive to outliers), I find that (using IGT comparative prevalence and adult population as a weight with only sugar and total calories as explanatory vars) the sugar effect, at the median, is 0.0148 but the 95% confidence interval is (-0.0191, 0.0217) when using the SAS default rank method of calculating standard errors.  The 95% confidence interval changes to (0.0041, 0.0254) when using an alternative resampling method.  So, whether the median effect is statistically significant depends on which method of calculating standard errors is used.

Here is the plot of the "sugar effect" at each quantile.  The first shows the 95% confidence intervals determined by the resampling method and the second uses the SAS default (I have to admit that I'm not sure which method is preferred in this case). 

Our Research on Menu Labeling

For the past couple years, I've been working with one of my former graduate students, Brenna Ellison at the University of Illinois, on some papers related to effects of calorie labels on menus (for those who may not be aware, "Obamacare" mandated that chain restaurants include such menu labels but the FDA has yet to release the final rules and implementation date).  

The first part of that research was finally published last week in the International Journal of Behavioral Nutrition and Physical Activity, where we report on a smaller sample from the larger study which includes the group of people Brenna interviewed after they ordered.  The results have been picked up by a couple news outlets, including this one from Reuters: 

Showing diners how many calories are in restaurant food items may influence how much they eat - especially among the least health-conscious people, a new study suggests.

That's true - but only partially true.  We find that the numeric labels mandated by "Obamacare" do not have a statistically significant effect on the number of calories people order.  The labels that we find to be (somewhat) effective are stop-light labels, in which we put a red dot next to the high calorie foods, a yellow dot next to the medium calorie foods, and a green dot next to the low-calorie foods.  As the story suggests, the labels are less influential among people who we rate (based on their survey answers) as health conscious.  The result isn't terribly surprising - people who are health conscious are probably already familiar with the caloric content of the foods they eat, and as such, adding labels are unlikely to provide new information.  Still, we'd want to know something about the cost of the label to know whether the policy was a net-plus (this is an issue we take up in our other papers still in the works).    

The result I found most interesting from the whole study was only discussed in the conclusions (and was missed all together in the news story) is the following:

Interestingly, despite the calorie+traffic light label’s effectiveness at reducing calories ordered, it was not the labeling format of choice. When asked which of the three labeling formats was preferred, only 27.5% of respondents said they wanted to see the calorie+traffic light label on their menus. Surprisingly, 42% preferred the calorie-only label which had virtually no influence on ordering behavior. These responses imply diners may want more information on their menus (the number of calories), yet diners do not want to be told what they should or should not consume (i.e., green = good, red = bad).

Summer School on Experimental Auctions

Pardon the public service announcement, but I wanted to let readers know that applications are now being accepted for a summer school I've co-taught on Experimental Auctions for the past 2 years in Italy.  Experimental auctions are a technique used to measure consumer willingness-to-pay for new food products, which in turn is used to project demand, market share, and benefits/costs of public policies.  We've had a fantastic time the past two years and I'm looking forward to the third, which was just approved for credit hours by the University of Bologna.  The content is mainly targeted toward graduate students or early career professionals (or marketing researchers interested in learning about a new technique).  You can find out more here and register here.

For a little enticement, here are some pictures of the previous years' classes.