Shady Science? DCM Research and Expression of Concern

    This piece should be prefaced by saying that PLOS One's commitment to transparency and diligence in maintaining scientific integrity is commendable. Their timeline in following-up on this particular issue, however, is disappointing and has left many pet owners confused. In May 2020, new research was published evaluating correlation between diet types and heart size parameters in Golden Retriever. This research had an 'Expression of Concern' added to it four months later, in September: 

"The article is being reassessed in light of issues raised post-publication, and the journal is investigating the validity of the concerns raised as well as their implications for the reliability of the article’s conclusions. Meanwhile, the PLOS ONE Editors issue this Expression of Concern."

    Now three months later, no additional updates have been issued. While the PLOS ONE editors explicitly stated that the very validity of the raised concerns was under investigation, this addition has not stopped various parties from using this Expression to delegitimize the research and dismiss the concerns surrounding grain-free and boutique diets. In the interest in providing a balanced narrative, each concern should be individually examined. It is imperative that the conversation not be dominated by misplaced dissent. The current study provides one more piece to the puzzle, and will hopefully not find itself unduly compromised by an unreasonable expectation of perfection.

    The possible origin of the alleged concerns is worthy of note. They closely mirror those presented in a Pet Food Industry article authored by Dr. Ryan Yamka, PhD, owner of a pet food consulting group and co-founder of Guardian Pet Foods.  At the end of his article, Dr. Yamka indicates he had written to PLOS, and he includes a reference number for his case: 06641324. He welcomes readers to cite it if they decide to follow suit and write a letter themselves. The concerns published by PLOS One are as follows.

"Questions were raised around the criteria used to categorize diets, including about the rationale of including global company sales details in these criteria given the objectives of the study."

    The rationale for the criteria used was explained extensively by the authors in the ‘Materials and Methods,’ section as well as the ‘Discussion.’ Guided by clinical experience as well as the previous findings from the FDA’s preliminary report , the authors defined the categories on the basis of pulse legume content (represented in 93% of cases reported to the FDA), absence of grains (represented in 91% of cases reported to the FDA), and manufacturer size (inversely correlated with prevalence of case reports). The decision to use global company sales as a criteria, which was of particular interest in the expressed concern, has a precedent in earlier commercial pet food research, namely a 2014 study in JAVMA that assessed thiamine concentrations in canned cat foods. That study utilized the same criteria and did find a significant difference in thiamine concentration in foods produced by smaller companies.
    To further characterize the rationale for including sales as a criteria, it is helpful to closely assess the data reported from the FDA in the context of the annual revenue of those companies. As the authors of the current study noted, there appears to be an inverse relationship between the number of cases reported to the FDA and market share held by the manufacturing companies. In the June 2019 update, the FDA’s update included a graph representing brands with ten or more reports associated, totaling 431 reports. Of those 16 brands, 8 brands associated with 236 reports are produced by companies with less than $1 billion annual revenue, 4 brands associated with 145 reports are produced by companies with between $1 and $2 billion, and the remaining 4 brands associated with 50 reports are produced by companies with greater than $2 billion. Further evaluation of those 4 brands from the largest companies also supports the trend. While Nutro and California Natural are owned by Mars Pet Care, their individual sales are each less than $1 billion of the company’s $18 billion revenue. Given the study’s objectives, which included evaluation of differences in taurine and echocardiographic parameters in association with diet in light of the FDA’s investigation, these criteria are entirely justified.

"Concerns were raised about potential confounds in the study, for example due to the inclusion of adult, puppy, and prescription-only foods, and due to inclusion of raw and kibble diets in the non-traditional group whereas the traditional group only included kibble"

    While an argument could be made that exclusion of raw diets would have been more appropriate, it is expected that if they were included, they would not be included with traditional diets, because they are not traditional. Commercially available raw diets are a relatively modern development. Raw diets have been reported to the FDA in associated with development of DCM.  That said, many dogs are fed combination diets, and it’s likely that many of the dogs consuming raw foods were also consuming kibble diets in conjunction. While this presents a limitation to the study (and indeed, was discussed as a limitation by the authors), it does not invalidate the findings nor render them less useful. Further studies could better elucidate the differences between non-traditional kibble alone, raw alone, and combination diets of both.
    The inclusion of puppy food only pertained to two diets included in the sample, and one of those was labeled for all life stages. The prescription diets also meet the criteria for ‘traditional diets’ and are labeled as complete and balanced adult maintenance diets per AAFCO feeding trials. As the diets in question still met the criteria outlined for the purposes of the study, it isn’t problematic that they were included. Furthermore, their exclusion likely would not change the results and conclusions, as they represent such a small portion of the sample size. This study was concerned with overarching data trends, not individual points of data. The sample groups represent real dogs presented to the clinic for wellness checks. While these criticisms would be warranted for a randomized double-blind laboratory controlled study, applying them toward a study of this nature, a case-controlled study on publicly owned animals, is misplaced.

"Soybeans are in the legume family, but soybeans and soy products were not considered as legumes for the purpose of categorizing diets in this study. The authors have requested a correction to clarify that this was an intentional exclusion, and they have noted that this aspect of the study design aligned with how soy products are being handled in a related FDA investigation."

    The authors adequately addressed this concern. 

"Questions were raised about the statistical analyses reported in the article."

    The current study utilized typical, widely employed statistical analyses. This included not only the data means, standard deviations, and p-testing (null hypothesis significance testing) but also appropriate, more specific analyses: Fisher’s exact test and Spearman’s Rank-order correlation evaluation. Fisher’s exact test, which is intended to examine significant association between two different categories of data, was used appropriately to assess statistical associations with clinical findings between the diet groups. Spearman’s Rank-order correlation evaluation, which assesses two ranked variables for the strength and direction of an association between them, was used appropriately for evaluating correlation between diet groups and taurine levels.
    One of the publicly offered criticisms of the current study was the failure to employ ‘proximate component analysis.' The exact meaning of this statement is difficult to parse out. ‘Proximate component analysis’ typically refers to the percent make-up of various nutrients in a given formulation. If this was the intended meaning, this type of analysis would have been inappropriate for the current study, though it could prove useful for future investigations. Alternatively, the author of the criticism may have meant ‘principal component analysis,’ or even ‘factor analysis,’ which is commonly confused with the former. Both of those methods seek to reduce the number of variables in a given data set, albeit through differing means. In any case, while the goal of the present study was to determine if there are differences in the assessed clinical parameters (taurine and echocardiographic findings) between two broad categories of food, a future study seeking to further hone in on specific aspects associated with the differences within one of the broad diet groups could benefit from additional statistical analyses.

"Reporting errors were noted, including errors in Tables 1 and 2, for which the authors requested a correction, and other issues raised to the journal that may impact categorization of some included foods as traditional versus non-traditional."

    It may be fair to remove some included foods and request that the authors issue a correction to their data. Based on the public issue of these concerns, not all of the issues raised are as significant as others. For example, while a diet with traits that fit both the TD and NTD groups may be fair to omit, such an omission would be unlikely to influence the overall data trend, since the dog eating the diet in question did not have abnormal parameters. In fact, it’s possible that exclusion of this diet would strengthen the significance of the author’s findings, since it was included in the NTD group. However, concerns raised about a mistake in naming a diet (listed as a partially pork formulation, when that company does not produce a pork diet) are almost certainly secondary to a mistake in the owner's reporting of diet history, and while such inconsistencies should be investigated and clarified, it is very unlikely such an error would compromise the study, the data, or the conclusions. The protein source was not evaluated as part of this study.
    Concerns about the classification of Merrick based on their acquisition by Nestle Purina are also misplaced. While Merrick is owned by Nestle Purina PetCare, a public statement has been issued that they continue to operate independently, with no changes at the hands of Purina secondary to the transaction.

"Questions were raised as to whether three months on a given diet is a sufficient duration to impact cardiac/DCM outcomes, and whether diet history prior to a subject’s current diet may impact the study outcomes. Concerns were also raised about differences between groups in the average time on the current diet."

    This was discussed extensively as a limitation by the study authors. Namely, while it represents a limitation for the evaluation of the echocardiographic parameters and should be considered for future studies, it is not expected to impact the taurine analysis, based on prior knowledge that changes in taurine can occur in a matter of weeks, which is well within the three month (12 week) minimum period. The length of time on a certain diet, and the impact of prior diet history on findings, is a limitation not just to this current study, but to analysis of publicly owned animals as a whole. While it is significant that the average length of time on a given diet was nearly twice that for dogs in the NTD group when compared to the TD group, this may simply reflect consumer behavior differences between the owners that are most likely to feed each. Assessment in this area would best be explored in a randomized clinical trial with a population of laboratory animals. Given the investments necessary to conduct a study of that nature, in terms of both time and finances, it would be unreasonable to discount more feasible research on the basis of that limitation alone. These differences are something to consider when evaluating the data, but not something which invalidates the data or subsequent findings. 

"Questions were raised about the validity of the conclusion statement, “Grain free diets, produced by small companies, including legumes within the top 5 ingredients represent a risk for the development of taurine deficiency and echocardiographic abnormalities consistent with DCM in the golden retriever.”"

    Given that the questions surrounding the validity of the conclusions likely arise from the concerns stated above, it is fair to say that the conclusions remain valid in light of the concerns being misplaced. In medicine, a risk factor is something associated with an increased chance of developing a disease. The findings of the present study are consistent with a correlation between the consumption of NTD (grain-free, small company, containing legumes in the top 5 ingredients) and clinical traits associated with disease. That correlation represents a risk factor. 

________________________________________

While PLOS One is doing the responsible thing in transparently responding to received concerns, investigating the validity of said concerns, and reassessing the paper in light of them, that these illegitimate concerns were received at all illustrates one more example of what has become a concerning trend of industry pushback to researching this topic. There should be no question that the industry has been negatively impacted by this investigation, with market trends showing that consumers are drifting away from grain-free varieties of food. What has become problematic is that rather than proactively seeking to identify the underlying cause and change formulas appropriately, the response from the industry has largely been to downplay and minimize the importance, or even existence, of this issue. Read more here.  

Most of the content in this article was originally included in a letter to the editors of PLOS One.

Follow me on Social Media: @AllTradesDVM

Facebook     Twitter