Abstract
Despite extensive research, no therapeutic interventions have been shown to prevent AKI, accelerate recovery of AKI, or reduce progression of AKI to CKD in patients. This failure in translation has led investigators to speculate that the animal models being used do not predict therapeutic responses in humans. Although this issue continues to be debated, an important concern that has not been addressed is whether improvements in preclinical study design can be identified that might also increase the likelihood of translating basic AKI research into clinical practice using the current models. In this review, we have taken an evidence-based approach to identify common weaknesses in study design and reporting in preclinical AKI research that may contribute to the poor translatability of the findings. We focused on use of N-acetylcysteine or sodium bicarbonate for the prevention of contrast-induced AKI and use of erythropoietin for the prevention of AKI, two therapeutic approaches that have been extensively studied in clinical trials. On the basis of our findings, we identified five areas for improvement in preclinical study design and reporting. These suggested and preliminary guidelines may help improve the quality of preclinical research for AKI drug development.
Severe AKI requiring dialysis affects >90,000 patients in the United States each year, and milder, nondialysis–dependent AKI affects >1.5 million per year.1–3 The underlying causes of AKI are frequently multifactorial, most commonly arising from ischemic, obstructive, toxic, and infectious insults.2 There is a bidirectional relationship between AKI and CKD. Mild AKI may lead to CKD, and there is marked increased risk of ESRD in patients with severe AKI requiring dialysis.4–10 Conversely, random sampling of a large integrated health care delivery system suggests that CKD also predisposes to AKI.11 Despite this, no therapeutic interventions have been proven to prevent AKI, improve the rate of renal recovery, or prevent postinjury CKD or ESRD after AKI.10,12 The lack of effective drug therapies is surprising, because there has been extensive basic research into the pathogenesis of AKI and preclinical testing of numerous therapies to prevent and treat AKI.
There is a number of reasons for this failure to translate from the bench to the bedside in AKI research. Failure of translation is, in part, because of problems with clinical trial design that include underpowering, late patient enrollment, and a lack of sensitive and accurate techniques to quantify the severity of injury and recovery in patients with AKI.13 However, an additional response to this lack of clinical translation is to conclude that the animal models of AKI are not predictive of therapeutic responses in patients. This has triggered a vigorous debate, and many excellent reviews on this topic have already been written.14–17 However, the question whether it is the design and execution of the AKI preclinical studies that are the root causes in the translational uncertainty has had significantly less review and debate in nephrology. The goal of this review, therefore, is to evaluate whether improvements in preclinical study design and statistical rigor can be identified that will increase the likelihood of translating basic research using the animal models that we are currently using into care of patients with AKI.
Common Concerns about Preclinical Study Design and Reporting That Affect Data Reproducibility
A key concern of the scientific community is that investigators cannot reproduce many of the published basic and preclinical research studies. It is estimated that 51%–89% of published research cannot be reproduced in other laboratories.18–22 This problem has been highlighted by a number of recent, high–profile commentaries from the pharmaceutical industry,18,19,23 funding agencies, including the National Institutes of Health,24,25 scientific journals,26,27 veterinarians,28 academics, and biostatisticians.29–36 This is of particular importance when animal models are used to advance the development of clinical therapeutics. Not only does this level of uncertainty potentially waste millions of funding dollars,37 but there are also genuine ethical concerns regarding the clinical evaluation of agents with supporting rationale that is solely on the basis of experimental evidence derived from studies in animal models. Problems with data reproducibility can arise at any of the different stages of scientific discovery ranging from exploratory, hypothesis–generating experiments to validation studies designed to evaluate therapeutic interventions. Systematic reviews have identified recurring methodologic weaknesses in preclinical studies.18,19,23,24,34,35 Vulnerabilities include errors in statistical design (inadequate power, ad hoc, interim, and retrospective end point analysis), lack of randomization and blinding, and incomplete methods reporting. In addition, lack of transparency in data reporting allows investigators to selectively report positive results without including information about experiments that fail to support the desired effect. This contrasts with the stringent requirements for study design, data collection, and reporting required for clinical trials. There has also been interest in promoting the analysis of continuous rather than dichotomous measures in clinical trials and preclinical research.38 In AKI, power analysis would be on the basis of the anticipated changes in actual creatinine values rather than setting an artificial dichotomy for what is deemed to be a clinically important effect (such as a 50% reduction in eGFR). This would increase the statistical power of a study without requiring larger numbers. These concerns are, in part, being addressed by scientific journals, many of which allow additional space to report complete methodology.26–28 Other key concerns include persistent bias in the publication of positive data and the lack of suitable forums for publication of intrinsically well conducted studies with outcomes that did not support the key hypothesis under evaluation (i.e., negative studies). Research that is selectively reported or cannot be reproduced ultimately hinders long–term therapeutic developments by increasing costs and causing delays when attempts are made to replicate preclinical studies. A recent analysis performed on the basis of a conservative estimate that 50% of scientific research is not reproducible, concluded that as much as $28 billion each year are spent in the United States on basic and preclinical research that is not reproducible.37 Despite these important concerns, potential limitations in the design and reporting of preclinical studies of AKI have received scant attention from the nephrology community.
To address concerns about preclinical study design and reporting in AKI, we have evaluated the quality of a selection of preclinical studies that have been used to support clinical trials in AKI, focusing specifically on issues related to data reproducibility and scientific and statistical rigor. For this, we have focused on two AKI–specific therapeutic approaches that have been extensively studied in clinical trials: (1) use of N-acetylcysteine (NAC) or sodium bicarbonate (NaHCO3) for the prevention of contrast-induced AKI (CI-AKI) and (2) use of erythropoietin (EPO) for the prevention of AKI. We chose these as examples of situations in which a lack of preclinical research may account for failures to translate into the clinical arena (NAC and NaHCO3 in CI-AKI) or a large number of preclinical studies have been performed, but many of them are poorly designed and of uncertain significance (EPO in AKI). On the basis of our findings, we have identified a number of areas for improvement in preclinical study design and reporting. These preliminary guidelines will help improve the quality of preclinical research for AKI drug development.
Therapeutic Intervention Studies in CI-AKI
A large number of therapeutic intervention studies has been performed to prevent the development of AKI in patients exposed to intravascular radiocontrast agents, often in the context of percutaneous cardiac interventions.39 To date, however, no intervention, other than preprocedure volume expansion, has been conclusively shown to reduce the incidence and severity or improve outcome in CI-AKI.40 In a recent meta-analysis, 11,071 study participants in 55 randomized controlled trials (RCTs) using NAC or NaHCO3 were included for analysis for the prevention of CI-AKI; 23 studies involving 2980 participants reported positive results, whereas 32 studies involving 8091 participants reported negative results. With a single exception (the Acetylcysteine for Contrast-Induced Nephropathy Trial, which failed to show therapeutic benefit of NAC for the prevention of coronary and peripheral vascular angiography–induced AKI41), these studies were small, with 500 study participants or less. In addition, all of these studies evaluated early changes in kidney function as primary end points and did not track long–term patient outcomes (such as death, dialysis, or long-term CKD). Because of the equivocal nature of these studies, a large, definitive study of 8680 participants is now being conducted to evaluate the efficacy of NAC and/or NaHCO3 for prevention of CI-AKI (the Prevention of Serious Adverse Events following Angiography [PRESERVE] Trial).40 Irrespective of the outcome of the PRESERVE Trial, the performance of these 55 clinical trials represents a significant investment of time and resources by clinical investigators, study sponsors, and study participants but has not yielded a definitive answer. To determine whether this could be attributed to the quality of preclinical research, we evaluated preclinical research that was used to support the use of NaHCO3 and NAC as therapeutic interventions for CI-AKI.
The scientific justification for clinical trials of NaHCO3 to prevent CI-AKI was on the basis of the observation that there was increased toxicity of contrast agents in cultured cells exposed to acidic environments and the idea that alkalinizing the urine might reduce free radical production by injured tubular epithelial cells.42 The first clinical study reporting a positive effect of NaHCO3 for the prevention of CI-AKI was published in 2004.43 In this study, 1 of 60 patients receiving NaHCO3 developed AKI, whereas 8 of 59 patients receiving normal saline developed AKI (P=0.02). Unfortunately, it has been persuasively argued that too few patients were enrolled to reliably exclude the possibility of a false-positive result.40 Notably, at the time that this clinical study was published, there were no preclinical animal studies supporting the use of NaHCO3 for the prevention of CI-AKI. Two studies had been reported in rats undergoing ischemia-reperfusion–induced AKI (IR-AKI), one of which showed beneficial short–term effects on serum creatinine,44 whereas the other showed no effect on GFR after injury.45 Since then, a number of studies have failed to show beneficial effects of NaHCO3 in rodent models of CI-AKI.46,47 Only one preclinical study has reported beneficial effects of NaHCO3 in CI-AKI.48 This study was performed in rats that had been water deprived and treated with a loop diuretic before treatment with the contrast agent. In addition, this study did not include a normal saline control to correct the effects of volume expansion resulting from the intravenous (iv) NaHCO3 infusion.48 Thus, despite the large number of clinical trials examining the use of NaHCO3 in patients with CI-AKI, there are negligible preclinical data to support the use of NaHCO3 as a therapeutic intervention in CI-AKI, and published supporting data used a model of CI-AKI that does not reflect clinical practice. If there had been clear published data showing that NaHCO3 did not improve the incidence or severity of AKI in clinically relevant preclinical models of CI-AKI at the time that these clinical trials were being performed, many of these early, underpowered clinical studies might have been avoided.
Like NaHCO3, there were few preclinical data supporting the use of NAC for the prevention of CI-AKI at the time of the first clinical study. NAC is an antioxidant that may improve renal function in AKI by inhibiting the generation of reactive oxygen species (ROS), thus limiting cell injury and inflammation. Widespread interest in the use and study of NAC for the prevention of CI-AKI began after the publication of a prospective RCT of 83 patients in 2000.49 In this study, 600 mg oral NAC was administered twice a day on the day of and day after contrast administration; one patient in the NAC-treated group developed CI-AKI, whereas nine in the placebo group developed CI-AKI. The rationale for the use of NAC was on the basis of animal evidence that ROS plays a role in toxicity (e.g., gentamicin and myoglobin)50 and CI-AKI.50–52 Evidence for the role of ROS in CI-AKI included animal data that ROS were increased in the kidney after iv contrast administration53 and that free radical scavengers, such as superoxide dismutase or catalase, improved renal function in CI-AKI.51,52 Remarkably, no study of NAC had been performed in an animal model of CI-AKI before 2000; iv NAC had been studied in IR-AKI, and improvement in GFR but not tubular necrosis occurred within 24 hours.54,55 Thus, although preclinical data supported the hypothesis that ROS may contribute to kidney injury in CI-AKI, preclinical studies supporting the use of NAC for the prevention of CI-AKI were inadequate. Particularly notable is that preclinical studies examined iv NAC just before kidney injury, whereas human trials examined oral NAC. Missing were key studies testing the half-life, dose-response, and duration of antioxidant effects of oral NAC in the kidney during injury.
Therapeutic Intervention Studies with EPO in AKI
Unlike studies on NaHCO3 and NAC for CI-AKI, a large number of preclinical studies supports the use of EPO to prevent AKI. The rationale supporting the use of EPO as a preventive treatment for AKI is that EPO has been shown to reduce cell death, reduce renal inflammation, and increase cellular repair after AKI.56 We identified 36 preclinical studies published over the last 20 years that evaluated the use of EPO in AKI (Table 1). The majority of studies used EPO as pretreatment (before the procedure used to induce AKI). In the remainder, treatment was initiated within a few hours of AKI induction; 34 of 36 studies showed protection against AKI assessed by reduced serum creatinine, reduced BUN, or increased GFR. A wide variety of animal models of AKI has been tested, including cisplatin AKI (10 studies), IR-AKI (12 studies), sepsis-associated AKI (SA-AKI; 7 studies), aortic occlusion (1 study), cardiopulmonary bypass (2 studies), perinatal asphyxia (1), iv contrast (1 study), amikacin (1 study), extreme exercise (1 study), and rhabdomyolysis (iv glycerol; 1 study). Thus, preclinical studies from numerous investigators using multiple models of AKI support the study of EPO for the prevention but not the treatment of AKI in a wide variety of clinical settings wherein patients may be at risk for AKI.
Summary of preclinical EPO efficacy studies in AKI
EPO has been evaluated in nine RCTs in a variety of clinical scenarios associated with risk of AKI, including AKI associated with cardiac surgery, aortic surgery with hypothermic cardiac arrest, renal transplantation, and medical and surgical intensive care unit (Table 2). Two studies examined EPO after the inciting event causing AKI and did not show a benefit. However, seven of nine RCTs appropriately used EPO to prevent AKI, although only two of these showed a positive effect.57,58 Notably, however, the number of patients in these prevention trials was small, ranging between 39 and 100 patients. As previously discussed, prevention trials in AKI require a large number of patients to be adequately powered to detect a plausible reduction in the rate of AKI; depending on the rate of AKI and the expected benefit, >500 patients would be needed.13 Thus, clinical trial data to date are inadequate to reach conclusions regarding the potential benefit of EPO to prevent AKI. Because the clinical trial data are inconclusive, we further examined the strength of the preclinical studies to identify gaps in research that might aid in the development of future clinical trials. We specifically evaluated issues relating to preclinical study design and reporting as well as comorbidities, sex, and genetic heterogeneity in the animal models being tested.
Summary of completed clinical EPO intervention studies in patients with AKI
Preclinical Study Design and Reporting
On the basis of our analysis of 36 published preclinical studies on the effects of EPO in AKI (Table 1), a number of common weaknesses was identified in study design and reporting, which is similar to those reported in the non-AKI literature.18,19,23,24,34,35
Only one study described prespecified end points (primary outcome measures) and used them to perform a power analysis and determine the sample size required to detect a specified effect size.59 Designing adequately powered animal studies is just as important as in clinical studies, because underpowered studies may lead to both false-negative and false-positive results. Because the number of animals needed to adequately power a study is dependent on variability in the primary outcome measures and because there may be significant inconsistency in primary outcome measures inherent to different models used in different laboratories, power needs to be determined on the basis of preliminary experience in each laboratory using each model. For example, if a mouse model of AKI was characterized by a serum creatinine increase of 1.6 mg/dl with an SD of 0.7 mg/dl, then 22 mice per group would be needed to detect a 40% reduction in serum creatinine (to 1.0 mg/dl) with an 80% power to detect a difference if one exists and a probability of detecting a difference by chance (α-error rate) of <5%. If, however, the SD for creatinine values was 0.4 mg/dl, only seven mice would be required per group to detect a 40% reduction in serum creatinine. In the EPO studies that we have reviewed, 24 of 36 studies used ≤10 animals per group, which may be too few to produce a result that is not caused by chance alone.
In 35 of 36 studies, there was no documentation that the investigators were fully blinded to the treatment groups over the whole course of the study (often this was only for histologic analyses).
Animal mortality was only reported in 4 of 36 studies,60–63 despite the fact that most of the models used (including cisplatin AKI, IR-AKI, and SA-AKI) have recognized mortality rates. These observations are likely to reflect the lack of transparency and accountability in preclinical study data reporting.
Only three studies reported negative outcomes.64–66 Of these, one study reported a positive outcome with another treatment being compared with EPO,65 and one study reported contrasting positive effects of EPO using a different model of AKI.64 These data likely reflect the strong and persistent publication bias for reporting only positive outcomes of studies throughout the medical literature.
Although a variety of doses and timing intervals for EPO treatment were included in the different studies, only one study evaluated dose-response effects of EPO on AKI.67 Importantly, only one study attempted to confirm target engagement and the mechanism of action of EPO through the β-common receptor in a preclinical SA-AKI model.68 This study showed that the β-common receptor mediates the beneficial effects of EPO in SA-AKI. Because the β-common receptor has lower affinity for EPO than the erythropoietin receptor (which mediates EPO-dependent erythropoiesis),69 one additional possibility is that the dose of EPO used in AKI may not be the same as that needed to stimulate erythropoiesis. This suggests that many of the preclinical studies, which used doses of EPO normally used to stimulate erythropoiesis, may have been using submaximal therapeutic doses.
Heterogeneity of Animals Used to Evaluate Therapeutic Responses
There are three major areas of heterogeneity in human AKI that need to be considered in animal models of AKI: multifactorial causes of AKI, which may have different pathobiologic bases; genetic and sex heterogeneity of affected patients; and common confounding comorbidities. Preclinical studies on EPO have included a diverse range of toxin, ischemic, and SA-AKI models, and although it can be argued that individual studies using single models may not be representative of specific human AKI pathophysiologies, involvement of multiple models showing similar beneficial effects provides a strong argument for potential translatability. In addition, the fact that studies were performed in not only genetically inbred mice but also, pigs, macaques, and outbred (genetically heterogeneous) rat strains (e.g., Sprague–Dawley rats) suggests that the efficacy signal for EPO use in animal models is likely to extend to human disease. However, two key confounding issues that have not been addressed in these studies may account for the current failure to translate EPO efficacy from animal models of AKI into humans: (1) modeling common clinical comorbidities (unlike clinical scenarios in which the majority of patents are elderly or have CKD and/or diabetes,1 none of the preclinical studies were performed in diabetic animals, older animals, or animals with baseline impaired kidney function) and (2) sex heterogeneity. The majority of studies were conducted in males, with 5 of 36 studies in females. Only two studies compared responses in males and females,70,71 with one of these studies showing a sex–dependent EPO response.71 Thus, the majority of preclinical AKI EPO studies do not reflect clinical comorbidities or sex-dependent variability in therapeutic responses that result from the inherent heterogeneity of most human study populations. Introduction of these variables into preclinical study design would increase variability and expense of these experiments; however, early investment of time and resources is likely to result in significant savings in the long run.
Recommendations, Implementation, and Challenges
On the basis of this review of the preclinical AKI EPO literature, we suggest five areas for improvement in study design that will increase the probability that preclinical research is translated into clinical practice.
Randomization and blinding to treatment.
Statistical rigor, particularly determination of sample size on the basis of defined, predetermined categorical or (better) continuous measurements of responses to therapy.
Publication bias favoring the publication of positive results. This is associated with lack of data transparency and accountability in reporting published preclinical data.
Lack of sex heterogeneity and lack of modeling for common clinical comorbidities. Studies performed in young, male, inbred mice are more reproducible than studies in old, mixed sex, outbred populations but do not reflect the true heterogeneity of human population pathobiology and therapeutic responses.
Lack of pharmacokinetic and pharmacodynamics, studies including dose-response studies to evaluate efficacy and methods to show efficient target engagement (therefore, adequate dosing).
Although we recognize that these weaknesses in preclinical AKI research are shared with other areas of preclinical research that have already been extensively reported,18,19,23,24,34,35 there remain significant challenges to implementing change. Despite the large number of publications on deficiencies in preclinical research design in other areas of medicine, an expectation gap remains between the academic research community and pharmaceutical industry. Underlying this are fundamental differences in practices: although academic faculty are under constant pressure to publish positive findings to sustain their careers, the pharmaceutical industry faces increasing costs as new molecular entities advance along the development pipeline. Thus, there is strong financial pressure to abandon a therapy if success seems in jeopardy. The first practice pattern leads to unjustified optimism for particular therapies and may squander resources by advancing clinical trials that are ultimately destined to be negative studies. The second pattern risks prematurely abandoning therapies with genuine promise to alleviate disease. Therefore, the key to success in practice modification is the involvement of all of the stakeholders with their varied and often conflicting expectations and demands in an effort to develop guidelines that can be implemented by everyone. Participants who will benefit from the success of this effort include:
Academic investigators who generate preclinical data used to support clinical trials in AKI;
Project leaders from pharmaceutical companies who are using published and in–house preclinical AKI research to support the development of costly, sponsored clinical trials;
Clinical scientists and statisticians who conduct and monitor clinical trials in AKI;
Food and Drug Administration reviewers who evaluate investigational new drug applications on the basis of preclinical AKI research; and
Patients who might benefit from intervention studies.
Other interested parties, including journal editors who publish preclinical AKI research as well as the funding bodies supporting preclinical AKI research, might also be involved. We anticipate that, through these deliberations, the goals of different stakeholders will be more clearly understood and aligned to ultimately improve preclinical studies and enhance the likelihood of successful AKI clinical trials. Examples of issues that will need to be discussed and addressed include (1) mechanisms to improve communication between preclinical investigators and investigators involved in clinical AKI research, (2) the ability of published preclinical data to be reproduced by industry, (3) improvements in preclinical study design, and (4) appropriateness of the application of preclinical data to human clinical trials.
The solutions to some of these problems should be relatively straightforward. For example, there should be wide agreement that animal experiments involving a test compound should be blinded, that power calculations should be performed with predefined primary outcomes, and that all outcomes, including unanticipated experimental mortality, should be reported. However, implementation of even these relatively straightforward changes will present significant logistical and financial challenges to many laboratories, and therefore, practical challenges to implementation would need to be addressed. For example, investigators may be concerned that additional costs incurred by performing properly powered preclinical research studies may make these experiments prohibitively expensive, particularly in this era of financial constraints. However, alternative approaches for statistical analysis on the basis of the use of continuous versus dichotomous end point variables (such as creatinine) could be used to mitigate the effect of these requirements of research costs.38 Other problems will be even more difficult to address, and potential solutions will be less obvious. For example, calling for the publication of negative results is unquestionably good for the science community at large but consumes an individual scientist’s time and resources with limited academic reward. Alternative strategies for complete data reporting might be considered, such as the development of secure, web–based data portals. These could be developed as repositories for pharmaceutical industry and academic research used to support new therapeutic applications in humans. All data could be sealed until the studies are completed and/or published but would be open to the public thereafter. This could also be used to enable data monitoring and accountability and could provide a centralized and secure system to ensure new standards in research design (such as identification of prespecified end points and power analysis). It is also important to recognize that many of the solutions may be costly, a significant issue in our current era of declining federal support for biomedical research. Notwithstanding this, it is also clear that our current strategies have failed to deliver new therapies that are urgently needed.10,12 We would argue that as a significant part of this process, many of the preclinical study design and reporting methods that are currently in use are in immediate need of change.
Disclosures
M.d.C. performs consultancy work for NephroGenex (Raleigh, NC). No financial support was utilized for this report.
Footnotes
Published online ahead of print. Publication date available at www.jasn.org.
See related editorial, “Models of Human AKI: Resemblance, Reproducibility, and Return on Investment,” on pages 2891–2893.
- Copyright © 2015 by the American Society of Nephrology
References
In this issue
Jump to section
More in this TOC Section
Up Front Matters
Cited By...
- Systematic review of guidelines for internal validity in the design, conduct and analysis of preclinical biomedical experiments involving laboratory animals
- Single Cell Sequencing and Kidney Organoids Generated from Pluripotent Stem Cells
- Long-Term Outcomes in Patients with Acute Kidney Injury
- Characterizing Pharmacokinetic-Pharmacodynamic Relationships and Efficacy of PI3K{delta} Inhibitors in Respiratory Models of TH2 and TH1 Inflammation
- Overcoming Translational Barriers in Acute Kidney Injury: A Report from an NIDDK Workshop
- Artificial Intelligence in Cardiology
- Selective Nanoparticle Targeting of the Renal Tubules