Skip to main content

Main menu

  • Home
  • Content
    • Published Ahead of Print
    • Current Issue
    • Archives
    • Saved Searches
    • Topics by Section
  • Authors
    • Submit a Manuscript
    • Author Resources
    • Reprints
  • Editorial Board
  • Subscriptions
  • More
    • About JASN
    • Advertising
    • Feedback
    • JASN Relaunch
    • Editorial Fellowship Program
  • Other
    • CJASN
    • Kidney News Online
    • In the Loop
    • American Society of Nephrology

User menu

  • Register
  • Subscribe
  • My alerts
  • Log in
  • My Cart

Search

  • Advanced search
American Society of Nephrology
  • Other
    • CJASN
    • Kidney News Online
    • In the Loop
    • American Society of Nephrology
  • Register
  • Subscribe
  • My alerts
  • Log in
  • My Cart
American Society of Nephrology

Advanced Search

  • Home
  • Content
    • Published Ahead of Print
    • Current Issue
    • Archives
    • Saved Searches
    • Topics by Section
  • Authors
    • Submit a Manuscript
    • Author Resources
    • Reprints
  • Editorial Board
  • Subscriptions
  • More
    • About JASN
    • Advertising
    • Feedback
    • JASN Relaunch
    • Editorial Fellowship Program
  • Follow JASN on Twitter
  • Visit ASN on Facebook
  • Follow JASN on RSS
  • Community Forum
Up Front MattersPerspectives
Open Access

Translational Methods in Nephrology: Individual Treatment Effect Modeling

F. Perry Wilson and Chirag R. Parikh
JASN November 2018, 29 (11) 2615-2618; DOI: https://doi.org/10.1681/ASN.2018060629
F. Perry Wilson
Program of Applied Translational Research, Department of Medicine, Yale University School of Medicine, New Haven, Connecticut; and
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Chirag R. Parikh
Program of Applied Translational Research, Department of Medicine, Yale University School of Medicine, New Haven, Connecticut; andDivision of Nephrology, Johns Hopkins School of Medicine, Baltimore, Maryland
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Data
  • Info & Metrics
  • PDF
Loading
  • outcomes
  • uplift
  • personalized medicine
  • marketing
  • randomized controlled trials
  • machine-learning

When we treat a patient, we are guided by the ethical principle of beneficence: to do good for them. How often do we achieve that goal? The best available data on interventions tested in randomized trials suggest that, even for medications that are efficacious on a population level, our interventions often make no difference on an individual level.

Number Needed to Treat: A Double-Edged Sword

This concept can best be understood using the number needed to treat (NNT).1 If a novel drug has an NNT of ten—a fairly robust effect by most standards—we are implying that we would need to treat ten patients to prevent one from experiencing the outcome of interest. In other words, 90% of individuals who received this highly efficacious drug would actually derive no benefit. Are we meeting our ethical obligation to this 90%? Do physicians owe our patients an explanation of this phenomenon when we prescribe a medication? Should we say that “this medication probably won’t make a difference in your life, but there is a chance that it may. I suggest you take it.”?

The NNT is a particularly meaningful metric in situations where a substantial relative treatment effect exists, but the occurrence of outcomes is low. For example, the Systolic Blood Pressure Intervention Trial was stopped early due to a marked observed treatment effect, with a 5.2% occurrence of the composite cardiovascular end point in the intensive BP control group compared with 6.8% in the control group (P<0.001).2 Although the relative effect size [1−(5.2/6.8)=24%] was large, the absolute effect size (6.8−5.2=1.6%) led to an NNT of around 60, because the vast majority of patients do not experience any of the cardiovascular events in the composite outcome.

How do we reconcile these results? The study was “positive” by conventional statistical definitions; however, we realize that we will need to treat 60 people to prevent one cardiovascular outcome. Does the 1.6% chance of (substantial) benefit outweigh the costs of treating 60 individuals (where costs can be literal dollars or complications and adverse effects of treatment)?

Broadly, the NNT of several common interventions for chronic diseases is depressing. However, if we broaden the perspective to the general population, treatments with high NNTs might still have significant benefits, particularly if the disease is common. Warfarin (versus aspirin) to prevent stroke in atrial fibrillation has an NNT of 80.3 Yet, we rarely tell patients that, except for in 1.25% of people, this drug (which requires extensive monitoring) will perform no better than aspirin. In the United States, however, hospitalizations with atrial fibrillation as the primary diagnosis number roughly 470,000 per year. This suggests that universal warfarin treatment would lead to >5875 fewer strokes per year—a clear societal benefit, although most treated patients would not experience that benefit individually.

Looking at the NNT shows how our obligations, as physicians, to our patients are potentially in conflict with our obligation to society at large. Can this issue be fixed? Broadly, this is the goal of personalized medicine—to identify the patients who will benefit the most from an intervention. To date, personalized medicine efforts have focused on genetic and biomarker measurement, but clinical data can be used as well, although there are significant hurdles to be crossed.

Risk-Based Targeting of Interventions

The first step is a recognition that the NNT is not uniform across all populations. It depends strongly on the baseline event rate, which may differ on a patient-to-patient level on the basis of a variety of other factors (such as severity of illness, family support, cointerventions, etc.). To use our warfarin example again, among those with a congestive heart failure, hypertension, age, diabetes, stroke (CHADS2) score of one, the NNT of warfarin versus aspirin to prevent stroke is 53. Among those with a CHADS2 score of six, the NNT is eight, suggesting that targeting warfarin to the subpopulation of patients with a high CHADS2 score (as is the common practice) may provide the most overall benefit.

The above approach to target a subgroup is achieved by prognostic modeling, and it is used frequently. Prognostic models, which are often refined into risk scores and calculators, estimate the probability of a given outcome. Prognostic models have the advantage of being able to be developed on observational data, but they have a significant weakness: not all interventions work better on those at higher risk for future events. This finding was driven home in a paper examining prognostic stratification in 32 large clinical trials. Although the absolute risk reduction was usually higher in the high-risk subgroups, the relative risk reduction was rarely different.4 In other words, prognostic targeting does not seem to identify “responders”—individuals who will disproportionately benefit from the intervention—but rather, it identifies those at high baseline risk. Efficient targeting of therapies requires maximizing the absolute risk reduction by identifying both those at high baseline risk and those in whom the relative risk reduction is largest.

A case in point is the use of tamoxifen in women with breast cancer. The drug is uniquely effective among those with hormone receptor–positive breast cancer, a group that, in fact, is at lower risk of breast cancer mortality overall.5 Targeting the use of tamoxifen to women with breast cancer at high risk of death would be ineffective (those at highest risk are hormone receptor negative). This is a case where the relative risk of an outcome is not constant as baseline risk increases.

Individual Treatment Effect Targeting of Therapies

By contrast, individual treatment effect (ITE) modeling attempts to predict who will benefit from a drug or intervention by identifying characteristics associated with greater than average relative risk reduction at the individual level.6,7 Broadly speaking, this is the goal of personalized medicine—where individual patient characteristics are used to predict therapeutic response. Typically, this is presented in a genetic framework, but it is conceivable that any collection of unique patient-level data can be used to identify this subgroup.

A perfect ITE model would allow targeting of an intervention such that the NNT would be one—all patients so targeted would experience the outcome without the intervention, and none would experience it if they received the intervention. However, with current statistical approaches, ITE models cannot be created outside a randomized trial. A comparison of the two modeling strategies is in Table 1. There are multiple statistical approaches to ITE modeling that can be broadly classified into one- or two-equation models. In a typical two-equation model, the first equation models the risk of outcome under control conditions, and the second equation models the risk of outcome under treatment conditions. The difference in predicted outcome is the ITE, where higher values predict a better treatment response. Single-equation models include several machine-learning algorithms, such as the “uplift random forest,”8 as well as models that attempt to group trial participants into “good targets” versus “bad targets” (also known as a class variable transformation).9

View this table:
  • View inline
  • View popup
Table 1.

Prognostic versus individual treatment effect modeling

Saving the Most Lives May Entail Treating Fewer People

Our goal in identifying a subpopulation for targeted therapy is not to maximize the NNT. The goal is to maximize the total events avoided (or lives saved) in the entire population regardless of treatment status. The concept is illustrated by a series of curves (known as uplift curves) in Figure 1. Imagine that we have an intervention that, if given to everyone in a population, would save 1000 lives. We can then posit what might happen if we choose (or are forced) to limit who we treat (Figure 1). If we treat randomly, then our population benefit decreases as we give the intervention to fewer and fewer people. If, instead, we treat prognostically—allocating treatment to the sickest individuals—we may save more lives than expected by chance but may miss some individuals with low baseline risk who nevertheless may have been saved by the intervention. However, if we target with ITE modeling, we achieve our maximum population benefit and can even exceed 1000 lives saved by excluding from treatment those who would be killed by the intervention.

Figure 1.
  • Download figure
  • Open in new tab
  • Download powerpoint
Figure 1.

Uplift curves demonstrate improvement of outcomes above expected when more narrow populations are targeted. Three “uplift curves” that illustrate (using hypothetical data) the number of lives saved under various targeting paradigms. With random targeting of an intervention to a population, the number of lives saved decreases linearly. Prognostic targeting may improve on random targeting, but it may fail to treat the individuals with low baseline risk who, nevertheless, would die without the intervention. Individual treatment effect (ITE) models maximize the area under the uplift curve by targeting those most likely to benefit from an intervention. This graph posits a “perfect” ITE model, which is unlikely to ever be created, but nevertheless, it shows how this approach may be superior to prognostic modeling when it comes to population benefit. Dashed lines highlight outcomes when 90% and 10% of the eligible population are targeted. ITE models may improve on a “treat everyone” approach by excluding individuals who may be harmed by the intervention, and they may preserve performance over narrower target ranges by excluding individuals whose outcomes will not be affected by the intervention.

Aside from the requirement for randomization, ITE models suffer from a low signal-to-noise ratio, because factors not related to the randomized intervention (like patient age and comorbidities) are often much more strongly related to the ultimate outcome than the intervention itself. The risk of “learning” noise is thus greater than what is seen in prognostic modeling. This means that proactive steps to avoid model overfitting are critical. Steps that are useful include ensuring adequate (and multiple) internal crossvalidations and mechanisms to combine sparse data into aggregate metrics (such as denoising autoencoders, a common machine-learning technique).10

ITE modeling can incorporate many covariates into a single score. Rather than noting that a treatment seems to uniquely benefit patients with diabetes, for example, one could include diabetes, congestive heart failure, or any other collected baseline variables (even gene variants) into the model. This avoids the multiple comparisons issue that can arise when multiple subgroups analyses are conducted. Beyond that, it would allow for true individualization of therapy, and it would allow a doctor to inform a patient of his or her individual chance of success with the drug, thereby balancing patient preferences with clinical benefits.

Finally, ITE modeling does not necessarily incorporate harms distinct from the outcome of interest. For example, an ITE model trained to identify which patients would benefit from an intervention in terms of avoiding initiation of dialysis would, by default, identify those for whom the intervention would hasten dialysis, but experiencing the primary outcome is rarely the only harm from an intervention. As such, ITE models should be targeted to outcomes (like death), where there is tolerance for potential harm provided that benefit is maximized.

The Future of ITE Modeling

ITE modeling will be most useful in situations where the costs of an intervention (in terms of potential patient harms or the broader cost to society) are significant or where resources are limited. In nephrology, ITE modeling could be used to improve allocation strategies to better select patients for kidney transplant, determine when (and if) dialysis should be initiated in AKI or CKD, or optimize BP treatment. ITE modeling seeks to minimize the population-level exposure to an intervention while maximizing the population benefit; most interventions would benefit from such precision targeting.

The future is personalized medicine. We now can capture wide-ranging genomic, metabolomic, proteomic, and phenomic data on individual patients. ITE modeling provides an opportunity to personalize treatment on the basis of routinely measured covariates. The critical next step is to validate these models prospectively—to show that ITE targeting can increase both the absolute risk reduction and the relative risk reduction of a specific intervention in the target population. Our group will be conducting such a trial, targeting electronic alerts for AKI on the basis of an ITE model, in the near future (clinicaltrials.gov NCT02786277). If ITE modeling proves to successfully target therapies—preserving population benefit while maximizing individual benefit—we are ethically obligated to use it.

Disclosures

None.

Acknowledgments

The authors would like to thank Melissa Martin for her editorial assistance.

F.P.W. is supported by National Institutes of Health (NIH) grant R01DK113191. C.R.P. is supported by NIH grants R01HL-085757 and R01-DK082185.

Footnotes

  • Published online ahead of print. Publication date available at www.jasn.org.

  • Copyright © 2018 by the American Society of Nephrology

References

  1. ↵
    1. Cook RJ,
    2. Sackett DL
    : The number needed to treat: A clinically useful measure of treatment effect. BMJ 310: 452–454, 1995
    OpenUrlFREE Full Text
  2. ↵
    1. Wright JT Jr.,
    2. Williamson JD,
    3. Whelton PK,
    4. Snyder JK,
    5. Sink KM,
    6. Rocco MV, et al
    .: SPRINT Research Group: A randomized trial of intensive versus standard blood-pressure control. N Engl J Med 373: 2103–2116, 2015
    OpenUrlCrossRefPubMed
  3. ↵
    1. Hart RG,
    2. Pearce LA,
    3. Aguilar MI
    : Meta-analysis: Antithrombotic therapy to prevent stroke in patients who have nonvalvular atrial fibrillation. Ann Intern Med 146: 857–867, 2007
    OpenUrlCrossRefPubMed
  4. ↵
    1. Kent DM,
    2. Nelson J,
    3. Dahabreh IJ,
    4. Rothwell PM,
    5. Altman DG,
    6. Hayward RA
    : Risk and treatment effect heterogeneity: Re-analysis of individual participant data from 32 large clinical trials. Int J Epidemiol 45: 2075–2088, 2016
    OpenUrl
  5. ↵
    1. Davies C,
    2. Godwin J,
    3. Gray R,
    4. Clarke M,
    5. Cutter D,
    6. Darby S, et al
    .: Early Breast Cancer Trialists’ Collaborative Group (EBCTCG): Relevance of breast cancer hormone receptors and other factors to the efficacy of adjuvant tamoxifen: Patient-level meta-analysis of randomised trials. Lancet 378: 771–784, 2011
    OpenUrlCrossRefPubMed
  6. ↵
    Jaskowski M, Jaroszewicz S: Uplift modeling for clinical trial data. Presented at the ICML Workshop on Clinical Data Analysis, Edinburgh, Scotland, June 30, 2012
  7. ↵
    1. Guelman L,
    2. Guillén M,
    3. Pérez-Marín AM
    : Random forests for uplift modeling: An insurance customer retention case. In: Modeling and Simulation in Engineering, Economics and Management, edited by Engemann KJ, Gil-Lafuente AM, and Merigo JM, Berlin, Springer, 2012, pp 123–133
  8. ↵
    1. Rzepakowski P,
    2. Jaroszewicz S
    : Decision trees for uplift modeling with single and multiple treatments. Knowl Inf Syst 32: 303–327, 2012
    OpenUrl
  9. ↵
    Zaniewicz Ł, Jaroszewicz S: Support vector machines for uplift modeling. Presented at the Data Mining Workshops (ICDMW) at the 2013 IEEE 13th International Conference, Dallas, TX, December 7–10, 2013
  10. ↵
    1. Biswas A,
    2. Parikh CR,
    3. Feldman HI, et al
    .: Identification of patients expected to benefit from electronic alerts for acute kidney injury. Clin J Am Soc Nephrol 3: 842–849, 2018
    OpenUrl
PreviousNext
Back to top

In this issue

Journal of the American Society of Nephrology: 29 (11)
Journal of the American Society of Nephrology
Vol. 29, Issue 11
November 2018
  • Table of Contents
  • Table of Contents (PDF)
  • About the Cover
  • Index by author
View Selected Citations (0)
Print
Download PDF
Article Alerts
Sign In to Email Alerts with your Email Address
Email Article
Thank you for your help in sharing the high-quality science in JASN.
Enter multiple addresses on separate lines or separate them with commas.
Translational Methods in Nephrology: Individual Treatment Effect Modeling
(Your Name) has sent you a message from American Society of Nephrology
(Your Name) thought you would like to see the American Society of Nephrology web site.
Citation Tools
Translational Methods in Nephrology: Individual Treatment Effect Modeling
F. Perry Wilson, Chirag R. Parikh
JASN Nov 2018, 29 (11) 2615-2618; DOI: 10.1681/ASN.2018060629

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Request Permissions
Share
Translational Methods in Nephrology: Individual Treatment Effect Modeling
F. Perry Wilson, Chirag R. Parikh
JASN Nov 2018, 29 (11) 2615-2618; DOI: 10.1681/ASN.2018060629
del.icio.us logo Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like

Jump to section

  • Article
    • Number Needed to Treat: A Double-Edged Sword
    • Risk-Based Targeting of Interventions
    • Individual Treatment Effect Targeting of Therapies
    • Saving the Most Lives May Entail Treating Fewer People
    • The Future of ITE Modeling
    • Disclosures
    • Acknowledgments
    • Footnotes
    • References
  • Figures & Data
  • Info & Metrics
  • PDF

More in this TOC Section

Up Front Matters

  • Therapeutic Inhibition of VEGF Signaling and Associated Nephrotoxicities
  • Producing Purer Podocytes
  • Readmissions Metrics in Hemodialysis: Do the Specifics Matter?
Show more Up Front Matters

Perspectives

  • Evaluating the Evidence behind Policy Mandates in US Dialysis Care
  • Are Current Strategies for Building a Human Kidney Misguided? Speculative Alternatives
  • Persistent Underrepresentation of Kidney Disease in Randomized, Controlled Trials of Cardiovascular Disease in the Contemporary Era
Show more Perspectives

Cited By...

  • No citing articles found.
  • Google Scholar

Similar Articles

Related Articles

  • No related articles found.
  • Scopus
  • PubMed
  • Google Scholar

Keywords

  • outcomes
  • uplift
  • personalized medicine
  • marketing
  • randomized controlled trials
  • machine-learning

About

  • JASN
  • ASN Journals
  • ASN Podcasts
  • About ASN
  • JASN Relaunch

Author Information

  • Manuscript Submission
  • Information for Authors
  • Reuse/Reprint Policy

More Information

  • Advertise
  • Subscribe
  • Email Alerts

© 2019 American Society of Nephrology

Print ISSN - 1046-6673 Online ISSN - 1533-3450

Powered by HighWire