Daubert's Troublesome Offspring: Litigation-Generated Science
Co-Authors: William L. Anderson and Barry M. Parsons.
In the few short years since the U.S. Supreme Court decided the Daubert case, setting new guidelines for the admissibility of expert evidence, the court system has experienced a significant shift in judges' approach to unreliable science. Despite early predictions that Daubert would weaken the court's role, federal and state courts have moved considerably to the side of excluding experts, particularly plaintiffs' experts.
Daubert's influence, however, may produce a phenomenon that will create difficulties for both courts and the scientific community - the increasing use of litigation-driven scientific studies, funded by and performed for the purpose of surviving a Daubert review. Litigation-generated science is problematic because it raises risks of validity, credibility, and even fraudulent research in much larger measure than science performed by independent researchers or for regulatory (e.g., EPA or FDA approval) purposes, where GLP standards apply and scrutiny of the research is intense.
Daubert recognizes the value of scientific "peer review" as a means of assuring the validity and quality of scientific testimony and the methodologies by which litigation experts derive their opinions. But does "peer review" and publication in a journal vouch for the veracity of litigation-based research? This will become an increasingly important question as both sides in health-claim tort cases publish the results of their research. Coincidentally, the scientific community is in the throes of a vigorous self-examination of its practice of "peer review." The court system, however, does not seem to be aware of the journal editors' debate over peer review and conflicts of interest, or of the role of recently established Uniform Requirements for Manuscripts Submitted to Biomedical Journals.
Manufacturers involved in products litigation will need to be alert for plaintiff-generated science in their cases. We briefly discuss below the expected growth in litigation-generated science and the means by which product manufacturers can defend against it.
The Growth of Litigation-Generated Research
Plaintiffs' health-case experts have typically relied on existing research, usually performed by academia, industry, or government agencies in the course of regulatory approval. This type of research is expensive and requires considerable expertise to perform, at least if done properly. Except in certain instances of clear-cut causation, registration-based studies usually are not particularly helpful for plaintiffs (products ordinarily do not obtain registration if their underlying studies demonstrate significant health risks), and the plaintiffs' experts often have to stretch credible scientific approaches to support their causation theories.
In several instances, courts applying Daubert have taken issue with the proffered expert's failure to provide peer-reviewed scientific authority supporting the expert's methodology or theory. In the Bendectin cases, for instance, none of the existing epidemiology studies found any association between Bendectin and birth defects. Undeterred, some of the Bendectin plaintiffs' experts "reanalyzed" these studies and argued that they in fact showed the opposite. Daubert v. Merrell Dow Pharm., Inc., 43 F.3d 1311, 1314 (9th Cir. 1995). Other experts based their opinion on animal studies and the chemical structure of Bendectin. Id. Plaintiffs' experts, however, did not conduct independent epidemiology or animal research to support their theories. Furthermore, despite their frequent publication on other topics, plaintiffs' epidemiology experts never submitted their "reanalysis" methodology for peer review and acceptance by the scientific community. Id. at 1318 n. 9. On remand from the Supreme Court, the Ninth Circuit Court of Appeals rejected plaintiffs' expert theories as derived from an unscientific methodology. Id. at 1319.
Because of decisions like those in the Bendectin litigation, the source of the science used in toxic tort and products liability litigation is likely to change. Plaintiffs' counsel are likely to conclude that they can increase the chances for survival of their novel scientific theories under Daubert's reliability requirement if those theories are backed by original, peer-reviewed research.
In a recent birth defects litigation in which Crowell & Moring represented a major agricultural product manufacturer, plaintiffs' expert, disregarding existing research which did not support his theories, arranged for several new in vitro (test tube) experiments on fetal and animal cell tissue in an attempt to establish an extremely low effect level. He claimed that by extrapolating from these in vitro studies, he could conclude that the product causes eye defects in humans. The results of two of those in vitro studies (but not the expert's extrapolation theory) were submitted to a medical journal that purported to utilize peer review, and the article was subsequently published. Plaintiffs contended that both the expert's theories and his studies were thus valid and appropriately peer reviewed. (The expert's methodology of extrapolating from in vitro studies was rejected by a Florida court of appeals under a Frye standard. Castillo v. E.I. du Pont de Nemours and Company, Nos. 96-2486, 96-2489, 1999 Fla. App. LEXIS 1447 (Fla. Ct. App. Feb. 17, 1999)).
Discovery revealed significant flaws with this research, including improper controls, the use of unqualified college students to perform the work, failure to characterize the applied substance, and the lack of validation of this particular test (neurite induction) as a proper or scientifically accepted screening methodology for determining whether a substance causes birth defects. More importantly, the publication of the article occurred without the plaintiffs' researchers alerting the journal or its peer reviewers that the studies had been funded by the plaintiffs' law firm and arranged by their testifying expert. The journal also failed to maintain any records of the peer reviewers' work, and it was unclear what level or extent of external, independent review took place. None of this information was apparent from the article itself.
This particular incident is a good example of the type of science that defense counsel will increasingly encounter. A few courts to date have faced litigation-driven research and have, to some degree, seen the wisdom of peeking behind the "peer review" curtain to see what is really there. See Black v. Rhone-Poulenc, Inc., 19 F. Supp. 2d 592, 600 (S.D. W.V. 1998); National Bank of Commerce v. Dow Chem. Co., 965 F. Supp. 1490 (E.D. Ark. 1996). This is a difficult area for courts, however, because the mechanics of peer review are not well known outside the medical and scientific publishing community.
Although questions about bias, funding, validity, and even fraud sometimes appear in the world of scientific and medical research, litigation-based research by its very nature makes those issues particularly critical. Ferreting out bad or improperly reported studies will be difficult for courts, especially if the studies have achieved some purported level of "peer review" through publication or other scientific discourse. In fact, the most important aspect of peer review occurs after publication when experts in the field attempt to replicate the theories espoused in the article. See Valentine v. Pioneer Chlor Alkali Co., 921 F. Supp. 666, 675 (D. Nev. 1996).
The World of Peer Review
Much of the effort in investigating litigation-generated science must focus on the nature and quality of any actual peer review that the research and its derivative theories have undergone. Litigants cannot conduct this investigation without a fairly thorough understanding of the nature of scientific peer review and its varieties of manifestations.
Although peer review has been around since the early 1700s, it did not become the sine qua non of publication until after World War II, when a sufficient quantity of articles became available to allow journals the luxury of rejecting submissions. Since then, peer review has developed sporadically -- journals basically went their own way, and even today the practices vary considerably. Some of these variations can have a significant effect on the quality of the review - e.g., the use of inside versus outside editors; the level of scrutiny and "cycling" between reviewers and authors required; the willingness to reject articles as seen in the rejection rate of the journal; the failure to arrange for independent epidemiological or statistical review of articles; and the disclosures required of authors. Indeed, the "peer review" that takes place at some journals amounts to nothing more than a few hours of grammatical editing and citation checking See Valentine v. Pioneer Chlor Alkali Co., 921 F. Supp. 666, 675 (D. Nev. 1996).
Disclosure of conflicts of interest in particular is a source of widely varying practices among journals. Financial and conflict disclosure is now being required by most of the major journals, but that practice is still not widespread. The journal that published the article referenced in the eye defect case above, for instance, did not include any requirement that its authors disclose even such severe appearances of conflict as litigation funding and the author's role as a paid expert witness.
In part to correct the great disparity among journal peer review practices, a group of leading editors convened the First International Congress on Peer Review in Biomedical Publication in 1989 to solicit research on peer review. That initial congress led to two others, and a separate group of journal editors ultimately drafted the Uniform Requirements for Manuscripts Submitted to Biomedical Journals. Whether a journal that publishes litigation-based science engages in full disclosure and thorough external peer review should be a critical issue for Daubert review.
Alternatives and Follow-Up to Publication Peer Review
The type of peer review most familiar to us - that conducted by scientific journals - is neither the end of "peer review" in the broader sense nor is it the only form of review that can provide sufficient reliability for Daubert. Unfortunately, courts may well make the mistake of accepting publication in a "peer-reviewed" journal as the seal of scientific approval. In fact, other forms of peer review are often even more important.
In some cases, particularly for research that is not intended for publication, primarily proprietary research, or that is not novel (and thus not publishable) these alternatives are important means of verifying the quality of the work. For instance, much of the research conducted to support approval of drugs or registration of agricultural products is not published in journals. Instead, this type of research is conducted by the manufacturers according to stringent standards established by regulatory agencies (e.g., the FDA or EPA) and submitted to those agencies for review. Agency review is often a more rigorous exercise than publication peer-review because of the scrutiny generated in the registration process and the penalties that face registrants who falsify data or otherwise fail to conduct proper studies and report them accurately. Grant-funding (e.g., by the National Institutes of Health or the National Science Foundation) is another source of potential scientific review since funding requests are often carefully scrutinized by specialists before funding is awarded.
In other instances, alternatives to publication peer review are important to consider because of the minimal value of inadequately-performed journal peer review. If the journal's practices are weak, it is important to determine whether the research and/or theories derived from it have been presented at scientific association meetings, critiqued, responded to, and otherwise run through the gauntlet of scientific debate. Even a well-conducted journal peer review has limitations in the scope of its review and ability to critique work that make the broader back-and-forth within the scientific world an important consideration. This issue becomes more critical if the theory that the expert derives from the research is in fact a far cry from the content of the article that was actually peer reviewed.
Conducting Discovery of Litigation-Generated Research
Litigation-based research typically makes its first public appearance in the course of expert discovery. The report of plaintiffs' causation expert may identify a new study - the expert may or may not be one of the authors of the study - and will probably indicate that the study has been or shortly will be published in a scientific journal, thus making it ostensibly "peer-reviewed." Defense counsel then faces a potential obstacle - one of the primary elements of a Daubert attack may have been substantially undermined.
In fact, however, neither the new research nor its publication may meet the elements of reliability under Daubert. Defense counsel will never know this, however, without the proper discovery, because the information required is ordinarily not provided in the publication itself. The deposition of the testifying expert will often prove unsatisfying, because the expert may well insulate himself or herself from the original research by deferring to the lead researcher and pleading ignorance of the details. To understand the real workings of the research, the lead researcher must be deposed, through a non-party subpoena if necessary. That deposition will focus on the source of funding of the study, the researcher's relationship with the testifying expert and his or her control over the study, the litigation etiology of the study, and the manner in which the study design and results were handled.
Other questions directed to the quality of the work itself can prove useful if they indicate improper procedures. Such flaws might include the failure to use good laboratory practices, to ensure fully blinded measurements and evaluations (thereby introducing bias), to establish proper control groups (thereby rendering the results meaningless or subject to alternative explanations), or to use study parameters that are relevant to the exposures in the litigation. It is also essential to subpoena the researcher's original records, because the testifying expert almost never has those records in his or her possession.
The defendant may also need to direct additional discovery against the journal that published the article. The simple assertion by a journal that its articles are "peer-reviewed" is nearly meaningless without further inquiry. Moreover, most journals do not provide specifics regarding their peer review practices in their published materials, thus requiring a deposition to uncover the actual practices used. Depositions of the peer reviewers themselves could also be informative, but most journals do not reveal the names of reviewers to protect their independence and confidentiality. There will likely be court battles over this practice if the quality of review of a particular study is important enough to the litigation.
Impacts on the Scientific and Product Manufacturing Community
Litigators and courts are not the only entities who will struggle with litigation-generated science. When the scientific community realizes that its sphere is about to be invaded by lawyer-paid researchers, the debate over disclosures, funding sources, and conflicts of interest will escalate. Litigation-driven science may push some editors to suggest that such research be published only after an independent review of underlying data, with explicit disclaimers by the journal, or not at all.
Companies engaged in product research should be prepared for the fallout. On the one hand, they may be forced to engage in their own litigation research to fend off the studies performed by plaintiffs' firms. Such research will be scrutinized to the limit, and the lawyers conducting the litigation must be careful in their approach. In addition, company researchers, who presently publish their research fairly infrequently, may have to step up their submission of studies for publication to keep up with the publication of plaintiff-generated research.
Daubert will continue to be one of the defendants' most potent weapons for keeping bad science out of the courtroom. To the extent litigation-generated science plays a future role in Daubert cases, companies who face product and toxic tort litigation will need to ensure the Daubert inquiry remains an educated and forceful one that is not compromised by unproven theories hiding behind marginal publication.
This material is made available on the Crowell & Moring LLP web site for information purposes only, and should not be relied upon to resolve specific legal questions. If you have such a question, you should consult with legal counsel.