Original Article| Volume 68, ISSUE 11, P1251-1260, November 2015

Using a matrix-analytical approach to synthesizing evidence solved incompatibility problem in the hierarchy of evidence

  • Harald Walach
    Corresponding author. Institute for Transcultural Health Studies, European University Viadrina Frankfurt (Oder), Grosse Scharrnstrasse 59, 15230 Frankfurt (Oder), Germany. Tel.: +49-335-5534-2380; fax: +49-335-5534-2748.
    Institute of Transcultural Health Studies, European University Viadrina, Frankfurt (Oder), Germany
    Search for articles by this author
  • Martin Loef
    Institute of Transcultural Health Studies, European University Viadrina, Frankfurt (Oder), Germany
    Search for articles by this author



      The hierarchy of evidence presupposes linearity and additivity of effects, as well as commutativity of knowledge structures. It thereby implicitly assumes a classical theoretical model.

      Study Design and Setting

      This is an argumentative article that uses theoretical analysis based on pertinent literature and known facts to examine the standard view of methodology.


      We show that the assumptions of the hierarchical model are wrong. The knowledge structures gained by various types of studies are not sequentially indifferent, that is, do not commute. External validity and internal validity are at least partially incompatible concepts. Therefore, one needs a different theoretical structure, typical of quantum-type theories, to model this situation. The consequence of this situation is that the implicit assumptions of the hierarchical model are wrong, if generalized to the concept of evidence in total.


      The problem can be solved by using a matrix-analytical approach to synthesizing evidence. Here, research methods that produce different types of evidence that complement each other are synthesized to yield the full knowledge. We show by an example how this might work. We conclude that the hierarchical model should be complemented by a broader reasoning in methodology.

      Key Words

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Sackett D.L.
        Evidence based medicine: how to practice and teach EBM.
        Churchill Livingstone, New York1997
        • Aickin M.
        Randomization, balance, and the validity and efficiency of design-adaptive allocation methods.
        J Stat Plan Inference. 2001; 94: 97-119
        • Sacks H.
        • Chalmers T.C.
        • Smith H.
        Randomized versus historical controls for clinical trials.
        Am J Med. 1982; 72: 233-240
        • Schulz K.F.
        • Chalmers I.
        • Hayes R.J.
        • Altman D.G.
        Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials.
        JAMA. 1995; 273: 408-412
        • Berger V.W.
        • Exner D.V.
        Detecting selection bias in randomized clinical trials.
        Control Clin Trials. 1999; 20: 319-327
        • Investigators WGfWsHI
        Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results from the Women's Health Initiative randomized controlled trial.
        JAMA. 2002; 288: 321-333
        • Chalmers T.C.
        • Celano P.
        • Sacks H.S.
        • Smith H.
        Bias in treatment assignment in controlled clinical trials.
        N Engl J Med. 1983; 309: 1358-1361
        • Collins R.
        • Gray R.
        • Godwin J.
        • Peto R.
        Avoidance of large biases and large random errors in the assessment of moderate treatment effects: the need for systematic overviews.
        Stat Med. 1987; 6: 245-250
        • Chalmers T.C.
        • Levin H.
        • Sacks H.S.
        • Reitman D.
        • Berrier J.
        • Nagalingam R.
        Meta-analysis of clinical trials as a scientific discipline. I: control of bias and comparison with large co-operative trials.
        Stat Med. 1987; 6: 315-325
        • Howick J.
        The philosophy of evidence-based medicine.
        Wiley-Blackwell, Chichester2011
        • Guyatt G.H.
        • Oxman A.D.
        • Vist G.E.
        • Kunz R.
        • Falck-Ytter Y.
        • Alonse-Coello P.
        • et al.
        GRADE: an emerging consensus on rating quality of evidence and strength of recommendations.
        BMJ. 2008; 336: 924
        • Rothwell P.M.
        External validity of randomised controlled trials: “to whom do the results of this trial apply?”.
        Lancet. 2005; 365: 82-93
        • Abel U.
        • Koch A.
        The mythology of randomization.
        in: Abel U. Koch A. Nonrandomized comparative clinical studies. Symposion Publishing, Düsseldorf1998: 27-40
        • Black N.
        Why we need observational studies to evaluate the effectiveness of health care.
        BMJ. 1996; 312: 1215-1218
        • Mant D.
        Can randomised trials inform clinical decisions about individual patients?.
        Lancet. 1999; 353: 743-746
        • Cook T.D.
        • Campbell D.T.
        Quasi-experimentation design and analysis issues for field settings.
        Rand McNally, Chicago1979
        • Halpern S.D.
        Evaluating preference effects in partially unblinded randomized clinical trials.
        J Clin Epidemiol. 2003; 56: 109-115
        • Howard K.I.
        • Cox W.M.
        • Saunders S.M.
        Attrition in substance abuse comparative treatment research: the illusion of randomization.
        in: Onken L.S. Blaine J.D. Psychotherapy and counseling in the treatment of drug abuse. National Institute of Drug Abuse, Rockville1990: 66-79
        • Llewellyn-Thomas H.A.
        • McGreal M.J.
        • Thiel Ec
        • Fine S.
        • Erlichman C.
        Patients' willingness to enter clinical trials: measuring the association with perceived benefit and preference for decision participation.
        Soc Sci Med. 1991; 32: 35-42
        • Brewin C.R.
        • Bradley C.
        Patient preferences and randomized clinical trials.
        BMJ. 1989; 299: 313-315
        • Feinstein A.R.
        Problems of randomized trials.
        in: Abel U. Koch A. Nonrandomized comparative clinical studies. Symposion Publishing, Düsseldorf1998: 1-13
        • Lloyd-Williams F.
        • Mair F.
        • Shiels C.
        • Hanratty B.
        • Goldstein P.
        • Beaton S.
        • et al.
        Why are patients in clinical trials of heart failure not like those we see in everyday practice?.
        J Clin Epidemiol. 2003; 56: 1157-1162
        • Moore M.J.
        • O'Sullivan B.
        • Tannock I.F.
        How expert physicians would wish to be treated if they had genitourinary cancer.
        J Clin Oncol. 1988; 6: 1736-1745
        • van der Lem R.
        • de Wever W.W.H.
        • van der Wee N.J.A.
        • van Veen T.
        • Cuijpers P.
        • Zitman F.G.
        The generalizability of psychotherapy efficacy trials in major depressive disorder: an analysis of the influence of patient selection in efficacy trials on symptom outcome in daily practice.
        BMC Psychiatry. 2012; 12: 192
        • Van Spall H.G.
        • Toren A.
        • Kiss A.
        • Fowler R.A.
        Eligibility criteria of randomized controlled trials published in high-impact general medical journals.
        JAMA. 2007; 297: 1233-1240
        • Rush J.A.
        • Trivedi M.H.
        • Wisniewski S.R.
        • Nierenberg A.A.
        • Stewart J.W.
        • Warden D.
        • et al.
        Acute and longer-term outcomes in depressed outpatients requiring one or several treatment steps: a STAR*D report.
        Am J Psychiatry. 2006; 163: 1905-1917
        • Pigott H.E.
        • Leventhal A.M.
        • Alter G.S.
        • Boren J.J.
        Efficacy and effectiveness of antidepressants: current status of research.
        Psychother Psychosom. 2010; 79: 267-279
        • Fava G.A.
        • Tomba E.
        • Grandi S.
        The road to recovery from depression—don't drive today with yesterday's map.
        Psychother Psychosom. 2007; 76: 260-265
        • Lilford R.J.
        • Jackson J.
        Equipoise and the ethics of randomization.
        J R Soc Med. 1995; 88: 552-559
        • Miller F.G.
        • Joffe S.
        Equipoise and the dilemma of randomized clinical trials.
        N Engl J Med. 2011; 364: 476-480
        • von Rohr E.
        • Pampallona S.
        • van Wegberg B.
        • Hürny C.
        • Bernhard J.
        • Heusser P.
        • et al.
        Experiences in the realisation of a research project on anthroposophical medicine in patients with advanced cancer.
        Schweiz Med Wochenschr. 2000; 130: 1173-1184
        • McCambridge J.
        • Kypri K.
        • Elbourne D.
        In randomization we trust? There are overlooked problems in experimenting with people in behavioral intervention trials.
        J Clin Epidemiol. 2014; 67: 247-253
        • Goyal M.
        • Singh S.
        • Sibinga E.M.
        • Gould N.F.
        • Rowland-Seymour A.
        • Sharma R.
        • et al.
        Meditation programs for psychological stress and well-being: a systematic review and meta-analysis.
        JAMA. 2014; 174: 357-368
        • Walach H.
        • Esch T.
        • Schmidt S.
        Meditation is no medication (Letter).
        JAMA Intern Med. 2014; 17: 1193-1194
        • Ney P.G.
        • Collings C.
        • Spensor C.
        Double-blind: double talk or are there ways to do better research.
        Med Hypotheses. 1986; 21: 119-126
        • Glasziou P.
        When are randomised trials unnecessary? Picking signal from noise.
        BMJ. 2007; 334: 349-351
        • Fonnebo V.
        • Grimsgaard S.
        • Walach H.
        • Ritenbaugh C.
        • Norheim A.J.
        • MacPherson H.
        • et al.
        Researching complementary and alternative treatments—the gatekeepers are not at home.
        BMC Med Res Methodol. 2007; 7: 7
        • Howick J.
        • Friedemann C.
        • Tsakok M.
        • Watson R.
        • Tsakok T.
        • Thomas J.
        • et al.
        Are treatments more effective than placebos? A systematic review and meta-analysis.
        PLoS One. 2013; 8: e62599
        • Kleijnen J.
        • de Craen A.J.M.
        • Van Everdingen J.
        • Krol L.
        Placebo effect in double-blind clinical trials: a review of interactions with medications.
        Lancet. 1994; 344: 1347-1349
        • Walach H.
        • Sadaghiani C.
        • Dehm C.
        • Bierman D.J.
        The therapeutic effect of clinical trials: understanding placebo response rates in clinical trials—a secondary analysis.
        BMC Med Res Methodol. 2005; 5: 26
      1. Meissner K. Kohls N. Colloca L. Placebo effects in medicine: mechanisms and clinical implications. Royal Society, London2011
        • Vase L.
        • Riley J.L.
        • Price D.D.
        A comparison of placebo effects in clinical analgesic trials versus studies of placebo analgesia.
        Pain. 2002; 99: 443-452
        • Hróbjartsson A.
        • Gøtzsche P.C.
        Is the placebo powerless? An analysis of clinical trials comparing placebo with no treatment.
        N Engl J Med. 2001; 344: 1594-1602
        • Kaptchuk T.J.
        • Kelley J.M.
        • Conboy L.A.
        • Davis R.B.
        • Kerr C.E.
        • Jacobson E.E.
        • et al.
        Components of placebo effect: randomised controlled trial in patients with irritable bowel syndrome.
        BMJ. 2008; 336: 999-1003
        • Moerman D.E.
        • Jonas W.B.
        Deconstructing the placebo effect and finding the meaning response.
        Ann Intern Med. 2002; 136: 471-476
        • Walach H.
        Placebo controls: historical, methodological and general aspects.
        Philos Trans R Soc Lond B Biol Sci. 2011; 366: 1870-1878
        • Walach H.
        • Falkenberg T.
        • Fonnebo V.
        • Lewith G.
        • Jonas W.
        Circular instead of hierarchical—methodological principles for the evaluation of complex interventions.
        BMC Med Res Methodol. 2006; 6: 29
        • Walach H.
        The efficacy paradox in randomized controlled trials of CAM and elsewhere: beware of the placebo trap.
        J Altern Complement Med. 2001; 7: 213-218
        • Turner E.H.
        • Matthews A.M.
        • Linardatos E.
        • Tell R.A.
        • Rosenthal R.
        Selective publication of antidepressant trials and its influence on apparent efficacy.
        N Engl J Med. 2008; 358: 252-260
        • Kirsch I.
        • Deacon B.J.
        • Huedo-Medina T.B.
        • Scoboria A.
        • Moore T.J.
        • Johnson B.T.
        Initial severity and antidepressant benefits: a meta-analysis of data submitted to the food and drug administration.
        PLoS Med. 2008; 5: e45
        • Rief W.
        • Nestoriuc Y.
        • Weiss S.
        • Welzel E.
        • Barsky A.J.
        • Hofmann S.G.
        Meta-analysis of the placebo response in antidepressant trials.
        J Affect Disord. 2009; 118: 1-8
        • Scharf H.-P.
        • Mansmann U.
        • Streitberger K.
        • Witte S.
        • Krämer J.
        • Maier C.
        • et al.
        Acupuncture and knee osteoarthritis.
        Ann Intern Med. 2006; 145: 12-20
        • Haake M.
        • Muller H.H.
        • Schade-Brittinger C.
        • Basler H.D.
        • Schäfer H.
        • Maier C.
        • et al.
        German Acupuncture Trials (GERAC) for chronic low back pain: randomized, multicenter, blinded, parallel-group trial with 3 groups.
        Arch Intern Med. 2007; 167: 1892-1898
        • Diener H.C.
        • Kronfeld K.
        • Boewing G.
        • Lungenhausen M.
        • Maier C.
        • Molsberger A.
        • et al.
        Efficacy of acupuncture for the prophylaxis of migraine: a multicentre randomised controlled clinical trial.
        Lancet Neurol. 2006; 5: 310-316
        • Howard R.
        • McShane R.
        • Lidesay J.
        • Ritchie C.
        • Baldwin A.
        • Barber R.
        • et al.
        Donepezil and memantine for moderate-to-severe Alzheimer's disease.
        N Engl J Med. 2012; 366: 893-903
        • Petticrew M.
        • Bell R.
        • Hunter D.
        Influence of psychological coping on survival and recurrence in people with cancer: systematic review.
        BMJ. 2002; 325: 1066
        • Smedslund G.
        • Ringdal G.I.
        Meta-analysis of the effects of psychosocial interventions on survival time in cancer patients.
        J Psychosom Res. 2004; 57: 123-131
        • Mays N.
        • Pope C.
        • Popay J.
        Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field.
        J Health Serv Res Policy. 2005; 10: 6-20
        • Pawson R.
        Evidence-based policy: in search of a method.
        Evaluation. 2002; 8: 157-181
        • Petticrew M.
        • Rehfuss E.
        • Noyes J.
        • Higgins J.P.
        • Mayhew A.
        • Pantoja T.
        • et al.
        Synthesizing evidence on complex interventions: how meta-analytical, qualitative, and mixed-method approaches can contribute.
        J Clin Epidemiol. 2013; 66: 1230-1243
        • Sager F.
        • Andereggen C.
        Dealing with complex causality in realist synthesis: the promise of qualitative comparative analysis.
        Am J Eval. 2012; 33: 60-78
        • Greenhalgh T.
        • Wong G.
        • Westhrop G.
        • Pawson R.
        Protocol—realist and meta-narrative evidence synthesis: evolving standards (RAMESES).
        BMC Med Res Methodol. 2011; 11: 115
        • Noyes J.
        • Gough D.
        • Lewin S.
        • Mayhew A.
        • Michie S.
        • Pantoja T.
        • et al.
        A research and development agenda for systematic reviews that ask complex questions about complex interventions.
        J Clin Epidemiol. 2013; 66: 1262-1270
        • Atmanspacher H.
        • Römer H.
        • Walach H.
        Weak quantum theory: complementarity and entanglement in physics and beyond.
        Foundations Phys. 2002; 32: 379-406
        • Atmanspacher H.
        • Filk T.
        • Römer H.
        Weak quantum theory: formal framework and selected applications.
        in: Khrennikov A. Quantum theory: reconsiderations of foundations—American Institute of Physics, Conference Proceedings. Melville, New York, USA2006
        • Filk T.
        • Römer H.
        Generalized quantum theory: overview and latest developments.
        Axiomathes. 2011; 21: 211-220
        • Walach H.
        • von Stillfried N.
        Generalizing quantum theory—approaches and applications.
        Axiomathes. 2011; 21: 185-371
        • Walach H.
        • Stillfried Nv
        Generalised quantum theory—basic idea and general intuition: a background story and overview.
        Axiomathes. 2011; 21: 185-209
        • Pothos E.M.
        • Busemeyer J.R.
        Can quantum probability provide a new direction for cognitive modeling.
        Behav Brain Sci. 2013; 36: 255-327
        • Evans D.
        Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions.
        J Clin Nurs. 2003; 12: 77-84
        • Hinterberger T.
        • von Stillfried N.
        The concept of complementarity and its role in quantum entanglement and generalized entanglement.
        Axiomathes. 2013; 23: 443-459
        • Bohr N.
        Causality and complementarity.
        Philos Sci. 1937; 4: 289-298
        • Concato J.
        • Horwitz R.I.
        Beyond randomised versus observational studies.
        Lancet. 2004; 363: 1660-1661
        • Concato J.
        • Lawler E.V.
        • Lew R.A.
        • Gaziano J.M.
        • Aslan M.
        • Huang G.D.
        Observational methods in comparative effectiveness research.
        Am J Med. 2010; 123: e16-e23
        • Linde K.
        • Scholz M.
        • Melchart D.
        • Willich S.N.
        Should systematic reviews include non-randomized and uncontrolled studies? The case of acupuncture for chronic headache.
        J Clin Epidemiol. 2002; 55: 77-85
        • Petticrew M.
        • Anderson L.
        • Elder R.
        • Grimshaw J.
        • Hopkins D.
        • Hahn R.
        • et al.
        Complex interventions and their implications for systematic reviews: a pragmatic approach.
        J Clin Epidemiol. 2013; 66: 1209-1214
        • Reilly D.
        • Taylor M.A.
        The evidence profile. The multidimensional nature of proof.
        Complement Ther Med. 1993; 1: 11-12
        • Jonas W.B.
        Building an evidence house: challenges and solutions to research in complementary and alternative medicine.
        Forsch Komplementrmed Klass Naturheilkd. 2005; 12: 159-167
        • Averill J.B.
        Matrix analysis as a complementary analytic strategy in qualitative inquiry.
        Qual Health Res. 2002; 12: 855-866
        • Clarke B.
        • Gillies D.
        • Illari P.
        • Russo F.
        • Williamson J.
        The evidence that evidence-based medicine omits.
        Prev Med. 2013; 57: 745-747
        • Bingham J.W.
        • Quinn D.C.
        • Richardson M.G.
        • Miles P.V.
        • Gabbe S.G.
        Using a healthcare matrix to assess patient care in terms of aims for improvement and core competencies.
        J Qual Patient Saf. 2005; 31: 98-105
        • Loef M.
        • Mendoza L.F.
        • Walach H.
        Lead (Pb) and the risk of Alzheimer's disease or cognitive decline: a systematic review.
        Toxin Rev. 2011; 30: 103-114
        • Loef M.
        • Walach H.
        Copper and iron in Alzheimer's disease: a systematic review and its dietary implications.
        Br J Nutr. 2012; 107: 7-19
        • Loef M.
        • Walach H.
        Fruit, vegetables, and prevention of cognitive decline or dementia: a systematic review of cohort studies.
        J Nutr Health Aging. 2012; 16: 626-630
        • Loef M.
        • Walach H.
        The omega-6/omega-3 ratio and dementia or cognitive decline: a systematic review on human studies and biological evidence.
        J Nutr Gerontol Geriatr. 2013; 32: 1-23
        • Mutter J.
        • Curth A.
        • Naumann J.
        • Deth R.
        • Walach H.
        Does inorganic mercury play a role in Alzheimer's disease? A systematic review and an integrated molecular mechanism.
        J Alzheimers Dis. 2010; 22: 357-374
        • Vickers A.J.
        Reducing systematic reviews to a cut and paste.
        Forsch Komplementrmed. 2010; 17: 303-305

      Linked Article

      • The hierarchy of evidence and quantum theory
        Journal of Clinical EpidemiologyVol. 72
        • Preview
          Medicine is complex, and evidence regarding the effects of interventions is subject to multiple sources of bias. The dominant view is that the “best available evidence” for the effects of interventions comes from randomized trials or systematic reviews of randomized trials. For instance, evidence-based medicine provides a hierarchy of evidence for assessing the benefits of treatments, which places well-conducted randomized trials and systematic reviews of randomized trials above other sources of evidence including observational studies.
        • Full-Text
        • PDF
      • Portability of therapeutic effects (letter commenting: J Clin Epidemiol. 2015;68(11):1251--1260.)
        Journal of Clinical EpidemiologyVol. 75
        • Preview
          Figure 1 used by Wallach and Loef [1] suggests that the worse the methodological quality of a clinical study the better a therapeutic effect found in a study could be transferred to future patients (external validity).
        • Full-Text
        • PDF
      • There is no reverse hierarchy (letter commenting: J Clin Epidemiol. 2015;68(11):1251--1260.)
        Journal of Clinical EpidemiologyVol. 75
        • Preview
          I am neither an expert in non-Abelian algebra nor in quantum theory, so I do not feel competent to judge the corresponding arguments of the authors [1]. The basis of these arguments, however, is obviously wrong. There is no reverse hierarchy of internal and external validity, as is suggested in Figure 1 and explicitly stated in the text: “The more internal validity is emphasized in a study, the lower external validity tends to become.”
        • Full-Text
        • PDF