Técnicas estadísticas utilizadas en la comparación de métodos cuantitativos de medición

Federico Nave; Jorge Nave

doi:10.36829/63CTS.v10i1.1490

Authors

Federico Nave Instituto de Investigaciones Químicas y Biológicas, Facultad de Ciencias Químicas y Farmacia, Universidad de San Carlos de Guatemala https://orcid.org/0000-0002-9981-3268
Jorge Nave Facultad de Ciencias Químicas y Biológicas, Universidad Mariano Gálvez de Guatemala

DOI:

https://doi.org/10.36829/63CTS.v10i1.1490

Keywords:

Reproducibility, reliability, concordance, agreement, Bland-Altman

Abstract

The methods comparison for quantitative measurements is an important practice both in the health and scientific-technological areas, as well as in any case in which measurement procedures are carried out; it allows the analysis of samples by the method whose analytical performance is desired and by the method of reference or another that has proven effectiveness, to determine if the results can be alike. An adequate sampling procedure should be considered, using a paired design, in which each sample must be analyzed by both methods and establish that the variable measured meets the requirements to apply adequate statistics for quantitative variables. For the analysis of these comparisons, the reproducibility of the methods should be evaluated in terms of reliability and agreement, as well as a regression analysis that will provide useful information on the proportional, constant, and random errors through the analysis of the slope, the intercept and the standard deviation of the residuals, respectively. The most appropriate techniques for comparing quantitative methods should consider a combination that includes the Bland and Altman procedure for concordance, the intraclass correlation coefficient or concordance correlation coefficient to assess reliability, and at least one regression technique (ordinary linear, Deming, or Passing-Bablok regressions), all the elements necessary for its interpretation must be reported and thus have the best statistical information to take decisions on the reproducibility and interchangeability of the methods; under no circumstances the Pearson's correlation coefficient should be used.

Downloads

Download data is not yet available.

References

Abu-Arafeh, A., Jordan, H., & Drummond, G. (2016). Reporting of methods comparison studies: A review of advice, an assessment of current practice, and specific suggestions for future reports. British Journal of Anaesthesia, 117(5), 569-575. https://doi.org/10.1093/bja/aew320

Akoglu, H. (2018). User's guide to correlation coefficients. Turkish Journal of Emergency Medicine, 18(3), 91-93. https://doi.org/10.1016/j.tjem.2018.08.001

Altman, D. G. (2009). Assessing new methods of clinical measurement. British Journal of General Practice, 59(563), 399-400. https://doi.org/10.3399/bjgp09X420905

Altman, D. G., & Bland, J. M. (1983). Measurement in medicine: The analysis of method comparison studies. The Statistician, 32(3), 307-317. https://doi.org/10.2307/2987937

Barnhart, H. X., Haber, M. J., & Lin, L. I. (2007). An overview on assessing agreement with continuous measurements. Journal of Biopharmaceutical Statistics, 17(4), 529-569. https://doi.org/10.1080/10543400701376480

Bartko, J. J. (1994). General methodology II. Measures of agreement: A single procedure. Statistics in Medicine, 13(5-7), 737-745. https://doi.org/10.1002/sim.4780130534

Bartlett, J. W., & Frost, C. (2008). Reliability, repeatability and reproducibility: Analysis of measurement errors in continuous variables. Ultrasound in Obstetrics & Gynecology, 31(4), 466-475. https://doi.org/10.1002/uog.5256

Baumdicker, F., & Hölker, U. (2020). Method comparison with repeated measurements – Passing-Bablok regression for grouped data with errors in both variables. Statistics and Probability Letters, 164, Artículo 108801. https://doi.org/10.1016/j.spl.2020.108801

Berthelsen, P. G., & Nilsson, L. B. (2006). Researcher bias and generalization of results in bias and limits of agreement analyses: A commentary based on the review of 50 Acta Anesthesiologica Scandinavica papers using the Altman-Bland approach. Acta Anesthesiologica Scandinavica, 50(9), 1111-1113. https://doi.org/10.1111/j.1399-6576.2006.01109.x

Bilić-Zulle, L. (2011). Comparison of methods: Passing and Bablok regression. Biochemia Medica, 21(1), 49-52. https://doi.org/10.11613/bm.2011.010

Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet, 1(8476), 307-310. https://doi.org/10.1016/S0140-6736(86)90837-8

Bland, J. M., & Altman, D. G. (1990). A note on the use of the intraclass correlation coefficient in the evaluation of agreement between two methods of measurement. Computers in Biology and Medicine, 20(5), 337-340. https://doi.org/10.1016/0010-4825(90)90013-f

Bland, J. M., & Altman, D. G. (1995a). Comparing methods of measurement: Why plotting difference against standard method is misleading. Lancet, 346(8982), 1085-1087. https://doi.org/10.1016/s0140-6736(95)91748-9

Bland, J. M., & Altman, D. G. (1995b). Comparing two methods of clinical measurement: A personal history. International Journal of Epidemiology, 24(Suppl. 1), S7-S14. https://doi.org/10.1093/ije/24.supplement_1.s7

Bland, J. M., & Altman, D. G. (1999). Measuring agreement in method comparison studies. Statistical Methods in Medical Research, 8(2), 135-160. https://doi.org/10.1177/096228029900800204

Bland, J. M., & Altman, D. G. (2003). Applying the right statistics: Analyses of measurement studies. Ultrasound in Obstetrics & Gynecology, 22(1), 85-93. https://doi.org/10.1002/uog.122

Bland, J. M., & Altman, D. G. (2007). Agreement between methods of measurement with multiple observations per individual. Journal of Biopharmaceutical Statistics, 17(4), 571-582. https://doi.org/10.1080/10543400701329422

Bland, J. M., Altman, D. G., & Warner, D. S. (2012). Agreed statistics: Measurement method comparison. Anesthesiology, 116(1), 182-185. https://doi.org/10.1097/ALN.0b013e31823d7784

Bruton, A., Conway, J. H., & Holgate, S. T. (2000). Reliability: What is it, and how is it measured? Physiotherapy, 86(2), 94-99. https://doi.org/10.1016/S0031-9406(05)61211-4

Bunce, C. (2009). Correlation, agreement, and Bland-Altman analysis: Statistical analysis of method comparison studies. American Journal of Ophthalmology, 148(1), 4-6. https://doi.org/10.1016/j.ajo.2008.09.032

Camacho-Sandoval, J. (2008). Coeficiente de concordancia para variables continuas. Acta Médica Costarricense, 50(4), 211-212.

Cardemil, F. (2017). Análisis de comparación y aplicaciones del método de Bland-Altman: ¿Concordancia o correlación? Medwave, 16(1), Artículo e6852. https://doi.org/10.5867/medwave.2017.01.6852

Carkeet, A., & Teng Goh, Y. (2018). Confidence and coverage for Bland-Altman limits of agreement and their approximate confidence intervals. Statistical Methods in Medical Research, 27(5), 1559-1574. https://doi.org/10.1177/0962280216665419

Carrasco, J. L., & Jover, L. (2004). Métodos estadísticos para evaluar la concordancia. Medicina Clínica (Barc), 122(Supl 1), 28-34. https://www.elsevier.es/es-revista-medicina-clinica-2-articulo-metodos-estadisticos-evaluar-concordancia-13057543

Chen, L. A., & Kao, C. L. (2021). Parametric and nonparametric improvements in Bland and Altman’s assessment of agreement method. Statistics in Medicine, 40(9), 2155-2176. https://doi.org/10.1002/sim.8895

Chhapola, V., Kanwal, S. K., & Brar, R. (2015). Reporting standards for Bland-Altman agreement analysis in laboratory research: A cross-sectional survey of current practice. Annals of Clinical Biochemistry, 52(3), 382-386. https://doi.org/10.1177/0004563214553438

Cornbleet, P. J., & Gochman, N. (1979). Incorrect least-squeres regression coefficients in method-comparison analysis. Clinical Chemistry, 25(3), 432-438. https://doi.org/10.1093/clinchem/25.3.432

Cortés-Reyes, E., Rubio-Romero, J. A., & Gaitán-Duarte, H. (2010). Métodos estadísticos de evaluación de la concordancia y la reproducibilidad de pruebas diagnósticas. Revista Colombiana de Obstetricia y Ginecología, 61(3), 247-255. https://doi.org/10.1093/clinchem/25.3.432

Costa-Santos, C., Bernardes, J., Ayres-de-Campos, D., Costa, A., & Costa, C. (2011). The limits of agreement and the intraclass correlation coefficient may be inconsistent in the interpretation of agreement. Journal of Clinical Epidemiology, 64(3), 264-269. https://doi.org/10.1016/j.jclinepi.2009.11.010

Dhakal, C. P. (2018). Regression invented as statistics. International Journal of Interdisciplinary Research and Innovations, 6(2), 1-5.

de Vet, H. C. W., Terwee, C. B., Knol, D. L., & Bouter, L. M. (2006). When to use agreement versus reliability measures. Journal of Clinical Epidemiology, 59(10), 1033-1039. https://doi.org/10.1016/j.jclinepi.2005.10.015

Dewitte, K., Fierens, C., Stöckl, D., & Thienpont, L. M. (2002). Application of the Bland-Altman plot for interpretation of method-comparison studies: A critical investigation of its practice. Clinical Chemistry, 48(5), 799-801. https://doi.org/10.1093/clinchem/48.5.799

Doğan, N. Ö. (2018). Bland-Altman analysis: A paradigm to understand correlation and agreement. Turkish Journal of Emergency Medicine, 18(4), 139-141. https://doi.org/10.1016/j.tjem.2018.09.001

Downing, S. M. (2004). Reliability: On the reproducibility of assessment data. Medical Education, 38(9), 1006-1012. https://doi.org/10.1111/j.1365-2929.2004.01932.x

Fleiss, J. L., Levin, B., & Cho Paik, M. (2003). Statistical methods for rates and proportions (3rd ed.). John Wiley & Sons.

Galton, F. (1886a). Adress by Francis Galton, M.A., F.R.S., President of the Anthropological Institute, President of the Section. Report of the fifty-fifth meeting of the British Association for the Advancement of Science held at Aberdeen in September 1885 (pp. 1206-1214). John Murray.

Galton, F. (1886b). Regression towards mediocrity in hereditary stature. Journal of the Anthropological Institute of Great Britain and Ireland, 15, 246-263. https://doi.org/10.2307/2841583

Galton, F. (1889). I. Co-relations and their measurement, chiefly from anthropometric data. Proceedings of the Royal Society of London, 45(273-279), 135-145. https://doi.org/10.1098/rspl.1888.0082

Gerke, O. (2020a). Reporting standards for a Bland-Altman agreement analysis: A review of methodological reviews. Diagnostics, 10(5), 334. https://doi.org/10.3390/diagnostics10050334

Gerke, O. (2020b). Nonparametric limits of agreement in method comparison studies: A simulation study on extreme quantile estimation. International Journal of Environmental Research and Public Health, 17(22), 8330. https://doi.org/10.3390/ijerph17228330

Gerke, O., & Möller, S. (2021). Bland-Altman limits of agreement from a Bayesian and frequentist perspective. Stats, 4(4), 1080-1090. https://doi.org/10.3390/stats4040062

Gerke, O., Pedersen, A. K., Debrabant, B., Halekoh, U., & Möller, S. (2022). Sample size determination in method comparison and observer variability studies. Journal of Clinical Monitoring and Computing, 36, 1241-1243. https://doi.org/10.1007/s10877-022-00853-x

Giavarina, D. (2015). Understanding Bland Altman analysis. Biochemia Medica, 25(2), 141-151. https://doi.org/10.11613/BM.2015.015

Giraudeau, B. (1996). Negative values of the intraclass correlation coefficient are not theoretically possible. Journal of Clinical Epidemiology, 49(10), 1205-1206. https://doi.org/10.1016/0895-4356(96)00053-4

Gulliford, M. C., Adams, G., Ukoumunne, O. C., Latinovic, R., Chinn, S., & Campbell, M. J. (2005). Intraclass correlation coefficient and outcome prevalence are associated in clustered binary data. Journal of Clinical Epidemiology, 58(3), 246-251. https_//doi.org/10.1016/j.jclinepi.2004.08.012

Han, H., Ma, Y., & Zhu, W. (2015). Galton's family heights data revisited.

arXiv:1508.02942v1 [stat.AP]. https://doi.org/10.48550/arXiv.1508.02942

Hernaez, R. (2015). Reliability and agreement studies: A guide for clinical investigators. Gut, 64(7), 1018-1027. https://doi.org/10.1136/gutjnl-2014-308619

Houston, W. J. (1983). The analysis of errors in orthodontic measurements. American Journal of Orthodontics, 83(5), 382-390. https://doi.org/10.1016/0002-9416(83)90322-6

Hsu, H., & Lachenbruch, P. A. (2005). Paired t test. Encyclopedia of Biostatistics, Online. John Wiley & Sons. https://doi.org/10.1002/0470011815.b2a15112

Jan, S. L., & Shieh, G. (2018). The Bland-Altman range of agreement: Exact interval procedure and sample size determination. Computers in Biology and Medicine, 100(1), 247-252. https://doi.org/10.1016/j.compbiomed.2018.06.020

Jensen, A. L., & Kjelgaard-Hansen, M. (2006). Method comparison in the clinical laboratory. Veterinary Clinical Pathology, 35(3), 276-286. https://doi.org/10.1111/j.1939-165x.2006.tb00131.x

Kalaria, T., Fenn, J., Sanders, A., Ford, C., & Gama, R. (2022). Clinical concordance assessment should be an integral component for laboratory method comparison studies: A regression transference of routine clinical data approach. Clinical Biochemistry, 103, 25-28. https://doi.org/10.1016/j.clinbiochem.2022.02.008

Kalra, A. (2017). Decoding the Bland-Altman plot: Basic review. Journal of the Practice of Cardiovascular Sciences, 3(1), 36-38. https://doi.org/10.4103/jpcs.jpcs_11_17

Kim, J., & Lee, J. H. (2022). A novel graphical evaluation of agreement. BMC Medical Research Methodology, 22(1), Artículo 51. https://doi.org/10.1186/s12874-022-01532-w

Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155-163. https://doi.org/10.1016/j.jcm.2016.02.012

Kotinkaduwa, L. N., & Choudhary, P. K. (2020). A segmented measurement error model for modeling analysis of method comparison data. Statistics in Medicine, 39(25), 3491-3502. https://doi.org/10.1002/sim.8677

Kottner, J., Audigé, L., Brorson, S., Donner A., Gajewski, B. J., Hróbjartsson, A., Roberts, C., Shoukri, M., & Streiner, D. L. (2011). Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. Journal of Clinical Epidemiology, 64(1), 96-106. https://doi.org/10.1016/j.jclinepi.2010.03.002

Kusunoki, T., Matsuoka, J., Ohtsu, H., Kagimura, T., & Nakamura, H. (2009). Relationship between intraclass and concordance correlation coefficients: Similarities and differences. Japanese Journal of Biometrics, 30(1), 35-53. https://doi.org/10.5691/jjb.30.35

Liao, J. J., Capen, R. C., & Schofield, T. L. (2006). Assessing the reproducibility of an analyitical method. Journal of Chromatographic Science, 44(3), 119-122. https://doi.org/10.1093/chromsci/44.3.119

Liljequist, D., Elfving, B., & Skavberg Roaldsen, K. (2019). Intraclass correlation - A discussion and demonstration of basic features. PLoS ONE, 14(7), Artículo e0219854. https://doi.org/10.1371/journal.pone.0219854

Lin, L. I. (1989). A concordance correlation coefficient to evaluate reproducibility. Biometrics, 45(1), 255-268. https://doi.org/10.2307/2532051

Lin, L. I. (1992). Assay validation using the concordance correlation coefficient. Biometrics, 48(2), 599-604. https://doi.org/10.2307/2532314

Linnet, K. (1993). Evaluation of regression procedures for methods comparison studies. Clinical Chemistry, 39(3), 424-432. https://doi.org/10.1093/clinchem/39.3.424

Linnet, K. (1999). Necessary sample size for method comparison studies based on regression analysis. Clinical Chemistry, 45(6), 882-894. https://doi.org/10.1093/clinchem/45.6.882

Liu, J., Tang, W., Chen, G., Lu, Y., Feng, C., & Tu, X. M. (2016). Correlation and agreement: Overview and clarification of competing concepts and measures. Shangai Archives of Psychiatry, 28(2), 115-120. https://doi.org/10.11919/j.issn.1002-0829.216045

Ludbrook, J. (2010). Confidence in Altman-Bland plots: A critical review of the method of differences. Clinical and Experimental Pharmacology and Physiology, 37(2), 143-149. https://doi.org/10.1111/j.1440-1681.2009.05288.x

Mandeville, P. B. (2005). Tema 9: El coeficiente de correlación intraclase (ICC). Ciencia UANL, 8(3), 414-416.

Mansournia, M. A., Waters, R., Nazemipour, M., Bland, M., & Altman, D. G. (2021). Bland-Altman methods for comparing methods of measurement and response to criticisms. Global Epidemiology, 3, Artículo 100045. https://doi.org/10.1016/j.gloepi.2020.100045

Manterola, C., Grande, L., Otzen, T., García, N., Salazar, P., & Quiroz, G. (2018). Confiabilidad, precisión o reproducibilidad de las mediciones. Métodos de valoración, utilidad y aplicaciones en la práctica clínica. Revista Chilena de Infectología, 35(6), 680-688. http://doi.org/10.4067/S0716-10182018000600680

Mantha, S., Roizen, M. F., Fleisher, L. A., Thisted, R., & Foss, J. (2000). Comparing methods of clinical measurement: Reporting standards for Bland and Altman analysis. Anesthesia & Analgesia, 90(3), 593-602. http://doi.org/10.1097/00000539-200003000-00018

Martelli Filho, J. A., Ávila Maltagliati, L., Trevisan, F., & Lopes de Alcântara Gil, C. T. (2005). Novo método estadístico para análise da reprodutibilidade. Revista Dental Press de Ortodontia e Ortopedia Facial, 10(5), 122-129. https://doi.org/10.1590/S1415-54192005000500012

Martínez Curbelo, G., Cortés Cortés, M. E., & Pérez Fernández, A. d. (2016). Metodología para el análisis de correlación y concordancia en equipos de mediciones similares. Universidad y Sociedad, 8(4), 65-70.

McBride, G. B. (2005). A proposal for strength-of-agreement criteria for Lin's concordance correlation coefficient. National Institute of Water & Atmospheric Research.

McDemid, R. (2021). Statistics in medicine. Anaesthesia and Intensive Care Medicine, 22(7), 454-462. https://doi.org/10.1016/j.mpaic.2021.05.009

Merriman, M. (1877). On the history of the method of least squares. The Analyst, 4(2), 33-36. https://doi.org/10.2307/2635472

Mishra, P., Singh, U., Pandey, C. M., Mishra, P., & Pandey, G. (2019). Application of Student’s t-test, analysis of variance, and covariance. Annals of Cardiac Anaesthesia, 22(4), 407-411. https://doi.org/10.4103/aca.ACA_94_19

Morgan, C. J., & Aban, I. (2016). Methods for evaluating the agreement between diagnostic tests. Journal of Nuclear Cardiology, 23(3), 511-513. https://doi.org/10.1007/s12350-015-0175-7

Müller, R., & Büttner, P. (1994). A critical discussion of intraclass correlation coefficients. Statistics in Medicine, 13(23-24), 2465-2476. https://doi.org/10.1002/sim.4780132310

Myles, P. S., & Cui, J. (2007). Using the Bland-Altman method to measure agreement with repeated measures. British Journal of Anaesthesia, 99(3), 309-311. https://doi.org/10.1093/bja/aem214

National Committee for Clinical Laboratory Standards. (2002). Method comparison and bias estimation using patient samples, approved Guideline (EP9-A2, 2nd

ed., Vol. 22, No. 19). Clinical and Laboratory Standards Institute.

Nickerson, C. A. (1997). A note on "A concordance correlation coefficient to evaluate reproducibility". Biometrics, 53(4), 1503-1507. https://doi.org/10.2307/2533516

Oldham, P. D. (1962). A note on the analysis of repeated measurements of the same subjects. Journal of Chronic Diseases, 15(10), 969-977. https://doi.org/10.1016/0021-9681(62)90116-9

Oleson, J. J., Brown, G. D., & McCreery, R. (2019). The evolution of statistical methods in speech, language, and hearing sciences. Journal of Speech, Language, and Hearing Research, 62, 498-506. https://doi.org/10.1044/2018_JSLHR-H-ASTM-18-0378

Pandis, N. (2021). Why using a paired t test to assess agreement is problematic? American Journal of Orthodontics and Dentofacial Orthopedics, 160(5), 767-768. https://doi.org/10.1016/j.ajodo.2021.07.001

Passing, H., & Bablok, W. (1983). A new biometrical procedure for testing the equality of measurements from two different analytical methods. Journal of Clinical Chemistry and Clinical Biochemistry, 21, 709-720. https://doi.org/10.1515/cclm.1983.21.11.709

Payne, R. B. (1997). Method comparison: Evaluation of least squares, Deming and Passing/Bablok regression procedures using computer simulation. Annals of Clinical Biochemistry, 34(3), 319-320. https://doi.org/10.1177/000456329703400317

Pearson, K. (1896). Mathematical contributions to the theory of evolution. -III. Regression, hereditym, and panmixia. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 187, 253-318. https://doi.org/10.1098/rsta.1896.0007

Pearson, K. (1930). The life, letters and labours of Francis Galton (Vol. IIIA Correlation, personal identification and eugenics). Cambridge at the University Press.

Rojas, C. M., Puerta, J., Gomez, J., & Calvache, J. A. (2016). Reproducibilidad de las mediciones clínicas. Revista Facultad de Salud, 8(1), 42-47. https://doi.org/10.25054/rfs.v8i1.1335

Schober, P., Boer, C., & Schwarte, L. A. (2018). Correlation coefficients: Appropriate use and interpretation. Anesthesia & Analgesia, 126(5), 1763-1768. https://doi.org/10.1213/ANE.0000000000002864

Senn, S., & Richardson, W. (1994). The first t-test. Statistics in Medicine, 13(8), 785-803. https://doi.org/10.1002/sim.4780130802

Smith, M. W., Ma, J., & Stafford, R. S. (2010). Bar charts enhance Bland-Altman plots when value ranges are limited. Journal of Clinical Epidemiology, 63(2), 180-184. https://doi.org/10.1016/j.jclinepi.2009.06.001

Stevens, N. T., Steiner, S. H., & MacKay, R. J. (2017). Assessing agreement between two measurement systems: An alternative to the limits of agreement approach. Statistical Methods in Medical Research, 26(6), 2487-2504. https://doi.org/10.1177/0962280215601133

Stöckl, D., Dewitte, K., & Thienpont, L. M. (1998). Validity of linear regression in method comparison studies: Is it limited by the statistical model or the quality of the analytical input data? Clinical Chimestry, 44(11), 2340-2346. https://doi.org/10.1093/clinchem/44.11.2340

Student. (1908). The probable error of a mean. Biometrika, 6(1), 1-25. https://doi.org/10.2307/2331554

Taffé, P. (2018). Effective plots to assess bias and precision in method comparison estudies. Statistical Methods in Medical Research, 27(6), 1650-1660. https://doi.org/10.1177/096228021666666

Taffé, P. (2019). Assessing bias, precision, and agreement in method comparison studies. Statistical Methods in Medical Research 29(3), 778-796. https://doi.org/10.1177/0962280219844535

Taffé, P. (2021). When can the Bland & Altman limits of agreement method be used and when it should not be used. Journal of Clinical Epidemiology, 137, 176-181. https://doi.org/10.1016/j.jclinepi.2021.04.004

Taffé, P. (2023). Use of clinical tolerance limits for assessing agreement. Statistical Methods in Medical Research, 32(1), 195-206. https://doi.org/10.1177/09622802221137743

Taffé., P., Halfon, P., & Halfon, M. (2020). A new statistical methodology overcame the defects of the Bland & Altman method. Journal of Clinical Epidemiology, 124, 1-7. https://doi.org/10.1016/j.jclinepi.2020.03.018

Temel, G., & Erdogan, S. (2017). Determining the sample size in agreement studies. Marmara Medical Journal, 30, 101-112. https://doi.org/10.5472/marumj.344822

Ungerer, J. P. J., & Pretorius, C. J. (2018). Method comparison - a practical approach based on error identification. Clinical Chemistry and Laboratory Medicine, 56(1), 1-4. https://doi.org/10.1515/cclm-2017-0842