Biomarker Score in Risk Prediction: Beyond Scientific Evidence and Statistical Performance

Heejung Bang

doi:10.4093/dmj.2020.0073

Articles

Page Path: HOME > Diabetes Metab J > Volume 44(2); 2020 > Article

Editorial Biomarker Score in Risk Prediction: Beyond Scientific Evidence and Statistical Performance: Heejung Bang^1,2; Diabetes & Metabolism Journal 2020;44(2):245-247.
DOI: https://doi.org/10.4093/dmj.2020.0073
Published online: April 23, 2020

4,345 Views
63 Download
3 Web of Science
3 Crossref
3 Scopus

¹Division of Biostatistics, Department of Public Health Sciences, School of Medicine, University of California, Davis, CA, USA.

²Clinical and Translational Science Center & Center for Healthcare Policy and Research, Davis School of Medicine, University of California, Sacramento, CA, USA.

Corresponding author: Heejung Bang. Division of Biostatistics, Department of Public Health Sciences, School of Medicine, University of California, One Shields Avenue, Med Sci 1C, Davis, CA 95616, USA. hbang@ucdavis.edu

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

prev next

Full Article

Download PDF

ACKNOWLEDGMENTS
NOTES
REFERENCES
About this article

Common objectives in research are to (1) explain and (2) predict [1 2]. In practice, these are generally inter-dependent; for example, sound prediction relies upon clear explanation (e.g., causal mechanism), and comprehensible explanation is informed by predictors such as correlation and association.

From clinical and public health perspectives, type 2 diabetes mellitus has emerged as a critical concern in all regions of the world, including Asia. Current theory is that diabetes in Asian populations presents somewhat differently compared with Western populations, e.g., “Asian body mass index (BMI)” [3 4].

Notably, multiple studies, including randomized controlled trials, have demonstrated that it may be possible to prevent or delay diabetes onset. While prevention may be considered the highest goal, it could be costly and time-consuming at the population level; benefits are in the future and costs are immediate [5]. Diabetes is often accompanied by other conditions or adverse events that can be used for purposes of early prediction and treatment. Since diabetes or prediabetes may be present but undiagnosed, research methods to identify or screen ‘current’ at-risk individuals are as needed as those aimed at predicting and preventing ‘future’ cases. In fact, the question of how to identify those at high risk for diabetes has been studied for a few decades, resulting in publication of a number of prediction models and risk or screening scores. These tools may be tailored to accommodate specific populations or subgroups; for example, American Diabetes Association diabetes risk test, Centers for Disease Control and Prevention prediabetes screening test, Korean diabetes risk score, among others [6-8]. However, given the volume of models/scores and wide range of predictors, clinicians and patients may find it difficult to select the best tool or they ignore them altogether, citing “research for research sake,” perceived lack of cooperation among investigators and with users, or frequently changing and often inconsistent results as reasons [9 10 11 12]. For example, more than 350 models for chronic obstructive pulmonary disease and for cardiovascular disease have been reported [11 12].

In this issue of the journal, Wang et al. [13] report on a proposed biomarker risk score to predict diabetes using the Singapore Chinese Health Study, a population-based cohort study, with a nested matched case-control design, using standard risk factors and (scientifically supported) biomarkers as predictors. The study employed conditional logistic regression for data analysis, area under the curve (AUC) P value and other information and re-classification statistics as evaluation measures, and internal validation and sensitivity analyses; these approaches are well accepted standards in research practice, despite some controversies in some measures [14]. No external validation, data limitations (e.g., no waist), uncertain cost implications, and future research direction (e.g., determination of cut-off values) were correctly discussed as potential limitations, indicating the authors' attempts at a balanced report. Here, I share several observations and viewpoints for readers pursuing similar lines of research.

First, the authors rightly mentioned cost and cost-effectiveness issues. Decisions involve trade-offs. While development of a biomarker test may create jobs, identify patients at risk and improve the public health, it will also incur costs (and possible other side effects). Despite the potential value of a statistically significant observation, cost might outweigh benefit; this will be particularly relevant in resource-limited settings. Despite expensive cutting-edge, high-tech or high-dimensional methods or best-fitting models, clinical or practical improvements may be disappointing [15 16 17] or even found to be no better than well known, easily obtained information at low or no cost, such as family history, BMI, or age. In that case, we may become “cost boosters” despite good intentions and the scientific knowledge gained.

Second, beyond prediction, what is the next step? After obtaining a set of predictors, odds/risk ratio (beta coefficient), risk estimate, or possibly risk status (say, high, or low), “so what?” A clinician (or patient) might use the information to recommend medication or additional tests, modified diet, increased exercise, or for emotional or financial preparedness. This point may best explain why so many prediction models are not used in the real world, in that there is no clear next step. Moreover, people tend not to worry much (or over-worry) about future events, and the importance of probability is perceived differently among individuals; for example, whether a 15% risk over 5 to 10 years is low or high [15 18].

Third, there are many biomarkers, and they are generally correlated, and levels of a variety of biomarkers would increase prior to the manifestation of glucose intolerance. It is entirely possible that different patient populations will be characterized by dissimilar sets of biomarkers, say, different “final four.” Understanding and use of these biomarkers or anything novel requires expert clinical interpretation (vs. patient's self-assessment or learning). In implementation, the usual issues related to measurement and variable definitions come into play: different units (e.g., mg/dL vs. mg/L vs. mmol/L); continuous vs. categorical variables (e.g., yes/no or high/middle/low); not applicable or missing/censored/mismeasured (randomly or informatively) data. In addition, some biomarkers (such as ferritin) may be unknown to most non-experts, so that even a single variable may be a barrier or deal breaker in real world adoption of a model. Model developers and clinicians should ask if biomarker A is really better than biomarker B (e.g., glycosylated hemoglobin), considering the pros and cons of each. Models can compete, evolve, or be upgraded naturally as with scientific knowledge, but there may be unintended consequences related to confusion and mistrust. Nevertheless, these could be opportunities for patient-clinician communication toward shared decision making and patient empowerment. Some clinicians view diabetes as a metabolic disease versus glucose disease and may treat biomarkers accordingly without need of a score. Clinicians may be called upon to explain biomarkers to patients—what they are, what their levels signify, why a given medication is needed, or even why disease definition can differ by sex/race/age/location. A patient's health education, including disease awareness and behavioral changes, may actually be stymied by highly technical and complex information. Also, which variables to use and how to use them in a risk model/score is an art, beyond science. While traditional regression may be well suited to education and simple and explainable models, black-box or push-the-button approaches (e.g., machine learning, artificial intelligence) can certainly have advantages in prediction or diagnostics. They should not compete [2 18 19 20].

Fourth, the terms “predictable,” “preventable,” “modifiable,” and “actionable” are not synonymous. While most biomarkers are modifiable or may serve as a mediator or surrogate marker, medical history and socioeconomic status are essentially unmodifiable despite high predictive value. As Hume (1748) stated, “The only immediate utility of all the sciences is to teach us how to control and regulate future events through their causes.” Although journal editors and reviewers of prediction models tend to be too focused on AUC, too high AUC may imply “too late,” such that predictor and outcome are virtually the same thing, or that a predictor must be just another (correlated) outcome or early marker of disease onset. Context and purpose always matter, beyond statistical performance (e.g., higher AUC, larger odds ratio, lower P value, largest sample size) in the prediction race [2 18 19].

Finally, the number #1 reason for any invention is need. Why need, for whom, when, etc. My personal motto for prediction modeling or method development as a statistician with heath economics/services and epidemiology training is: “As long as some use or are willing to pay, a model can be a success.” Researchers developing future models should emphasize this ‘translational’ aspect and practical use; perhaps a reasonable short-term goal could be intent to implement at the authors' own institution(s). The study of risk prediction is widespread in medicine and public health, with ever increasing availability of data and powerful computer and easy statistical modeling [11], but it should encompass fundamental principles of business/marketing and engineering. Without a consumer or user, a risk score is just another regression model, supplier of a bag of odds/risk/hazard ratios, or one more paper in curriculum vitae (including my own!).

Regarding prediction modeling, Breiman stated: “My attitude toward new and/or complicated methods is pragmatic. Prove that you've got a better mousetrap and I'll buy it. But the proof had better be concrete and convincing” [2 20]. My hope is that we can maintain a disciplined, focused but pragmatic approach to conducting studies that will impart real knowledge, impact, use or value to diabetes research worldwide.

Acknowledgements

ACKNOWLEDGMENTS

The author thanks Ms. Caron Modeas for English editing service, and Drs. Thamer, Spence, Kaufman, Franks, Jaffe, and Kashner for useful advice and comments. The author is partly supported by the National Institutes of Health through grant UL1 TR001860 and R01 AR076088.

NOTES

CONFLICTS OF INTEREST: No potential conflict of interest relevant to this article was reported.

REFERENCES

1. Shmueli G. To explain or to predict? Statist Sci 2010;25:289-310.Article
2. Breiman L. Statistical modeling: the two cultures. Statist Sci 2001;16:199-231.
3. WHO Expert Consultation. Appropriate body-mass index for Asian populations and its implications for policy and intervention strategies. Lancet 2004;363:157-163. Article PubMed
4. Matsushita K, Tang O, Selvin E. Addressing challenges and implications of national surveillance for racial/ethnic disparities in diabetes. JAMA 2019;322:2387-2388. Article PubMed PMC
5. Haddix AC, Teutsch SM, Corso PS. Prevention effectiveness: a guide to decision analysis and economic evaluation. Oxford: Oxford University Press; 1996.
6. Poltavskiy E, Kim DJ, Bang H. Comparison of screening scores for diabetes and prediabetes. Diabetes Res Clin Pract 2016;118:146-153. Article PubMed PMC
7. Ha KH, Lee YH, Song SO, Lee JW, Kim DW, Cho KH, Kim DJ. Development and validation of the Korean diabetes risk score: a 10-year national cohort study. Diabetes Metab J 2018;42:402-414. Article PubMed PMC PDF
8. Lee YH, Bang H, Kim HC, Kim HM, Park SW, Kim DJ. A simple screening score for diabetes for the Korean population: development, validation, and comparison with other scores. Diabetes Care 2012;35:1723-1730. PubMed PMC
9. Wyatt JC, Altman DG. Prognostic models: clinically useful or quickly forgotten? BMJ 1995;311:1539.Article PMC
10. Liao L, Mark DB. Clinical prediction models: are we building better mousetraps? J Am Coll Cardiol 2003;42:851-853. PubMed
11. Adibi A, Sadatsafavi M, Ioannidis JPA. Validation and utility testing of clinical prediction models: time to change the approach. JAMA 2020 3 05 [Epub].
12. Damen JA, Hooft L, Schuit E, Debray TP, Collins GS, Tzoulaki I, Lassale CM, Siontis GC, Chiocchia V, Roberts C, Schlussel MM, Gerry S, Black JA, Heus P, van der Schouw YT, Peelen LM, Moons KG. Prediction models for cardiovascular disease risk in the general population: systematic review. BMJ 2016;353:i2416. Article PubMed PMC
13. Wang Y, Koh WP, Sim X, Yuan JM, Pan A. Multiple biomarkers improved prediction for the risk of type 2 diabetes mellitus in Singapore Chinese men and women. Diabetes Metab 2020;44:295-306.Article PDF
14. Kerr KF, Wang Z, Janes H, McClelland RL, Psaty BM, Pepe MS. Net reclassification indices for evaluating risk prediction instruments: a critical review. Epidemiology 2014;25:114-121. Article PubMed PMC
15. Chow SC. Chapter 95, Cost-effectiveness analysis. Encyclopedia of biopharmaceutical statistics. 3rd ed. Boca Raton: CRC Press; 2010.
16. Fojo T, Grady C. How much is life worth: cetuximab, non-small cell lung cancer, and the $440 billion question. J Natl Cancer Inst 2009;101:1044-1048. Article PubMed PMC
17. Ellis LM, Bernstein DS, Voest EE, Berlin JD, Sargent D, Cortazar P, Garrett-Mayer E, Herbst RS, Lilenbaum RC, Sima C, Venook AP, Gonen M, Schilsky RL, Meropol NJ, Schnipper LE. American Society of Clinical Oncology perspective: raising the bar for clinical trials by defining clinically meaningful outcomes. J Clin Oncol 2014;32:1277-1280. Article PubMed
18. Lee YH, Bang H, Kim DJ. How to establish clinical prediction models. Endocrinol Metab (Seoul) 2016;31:38-44. Article PubMed PMC PDF
19. Goldstein DG, Gigerenzer G. Fast and frugal forecasting. Int J Forecast 2009;25:760-772.Article
20. Raper S. Leo Breiman's “two cultures”. Significance 2020;17:34-37.Article PDF

Figure & Data

References

Citations

Citations to this article as recorded by

Imatinib adherence prediction using machine learning approach in patients with gastrointestinal stromal tumor
Li Liu, Ze Yu, Hefen Chen, Zhujun Gong, Xiao Huang, Linhua Chen, Ziying Fan, Jinyuan Zhang, Jiannan Yan, Hongkun Tian, Xiangyu Zeng, Zhiliang Chen, Peng Zhang, Hong Zhou
Cancer.2024;[Epub] CrossRef
Performance of Diabetes and Kidney Disease Screening Scores in Contemporary United States and Korean Populations
Liela Meng, Keun-Sang Kwon, Dae Jung Kim, Yong-ho Lee, Jeehyoung Kim, Abhijit V. Kshirsagar, Heejung Bang
Diabetes & Metabolism Journal.2022; 46(2): 273. CrossRef
Unified framework for brain connectivity-based biomarkers in neurodegenerative disorders
Sung-Woo Kim, Yeong-Hun Song, Hee Jin Kim, Young Noh, Sang Won Seo, Duk L. Na, Joon-Kyung Seong
Frontiers in Neuroscience.2022;[Epub] CrossRef

PubReader

Cite this Article

Cite this Article: export Copy Download Format; Close

Download Citation

Download a citation file in RIS format that can be imported by all major citation management software, including EndNote, ProCite, RefWorks, and Reference Manager.

Format:

RIS — For EndNote, ProCite, RefWorks, and most other reference management software
BibTeX — For JabRef, BibDesk, and other BibTeX-specific software

Include:

Citation for the content below
Citation and abstract for the content below

Biomarker Score in Risk Prediction: Beyond Scientific Evidence and Statistical Performance

Diabetes Metab J. 2020;44(2):245-247. Published online April 23, 2020

DOI: https://doi.org/10.4093/dmj.2020.0073

XML Download

Biomarker Score in Risk Prediction: Beyond Scientific Evidence and Statistical Performance

About this article

Bang H. Biomarker Score in Risk Prediction: Beyond Scientific Evidence and Statistical Performance. Diabetes Metab J. 2020;44(2):245-247.

DOI: https://doi.org/10.4093/dmj.2020.0073.

Download citation ↓