alexa Comparative Effectiveness Research: Are The Methods Being Used Correctly? | Open Access Journals
ISSN: 2471-268X
Health Economics & Outcome Research: Open Access
Like us on:
Make the best use of Scientific Research and information from our 700+ peer reviewed, Open Access Journals that operates with the help of 50,000+ Editorial Board Members and esteemed reviewers and 1000+ Scientific associations in Medical, Clinical, Pharmaceutical, Engineering, Technology and Management Fields.
Meet Inspiring Speakers and Experts at our 3000+ Global Conferenceseries Events with over 600+ Conferences, 1200+ Symposiums and 1200+ Workshops on
Medical, Pharma, Engineering, Science, Technology and Business

Comparative Effectiveness Research: Are The Methods Being Used Correctly?

Paula Andrade*, Tatiana Dilla and José A Sacristán

Medical Department, Lilly, Avenida de la Industria, Madrid, Spain

*Corresponding Author:
Paula Andrade
Health Outcomes Scientist
Medical Department
Lilly, Avenida de la Industria, 30, 28108 Alcobendas
Madrid, Spain
Tel: +34 91 623 33 16
Fax: +34 91 663 52 31
E-mail: [email protected]

Received Date: June 02, 2017; Accepted Date: June 19, 2017; Published Date: June 26, 2017

Citation: Andrade P, Dilla T, Sacristán JA (2017) Comparative Effectiveness Research: Are The Methods Being Used Correctly?. Health Econ Outcome Res 3:132.doi: 10.4172/2471-268X.1000132

Copyright: © 2017 Andrade P, et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Visit for more related articles at Health Economics & Outcome Research: Open Access


Rationale, aims and objectives: Comparative Effectiveness Research (CER) seeks to identify what health care interventions work best for improving health at both the individual and population level. The objective of this study was to determine whether published comparative effectiveness research studies adhere to accepted methodological principles.

Methods: Structured literature search of CER articles published in high-impact general medicine journals between 2009 and 2015, and assessment of their adherence to five methodological principles.

Results: 93 articles were retrieved from the search and 40 articles were finally selected. All of the studies included active comparators, 35% of the studies did not evaluate safety, 97% did not evaluate costs, 95% of the studies did not included patient perspectives, and 60% did not use any procedure to determine the heterogeneity of the response.

Conclusion: The sample of CER papers examined did not meet the recommended requisites for this type of studies. Our findings suggest that the majority of CER studies may not be useful to guide physicians, purchasers, and policy makers to make informed decisions that improve health care at both the individual and population levels.


Health care; CER studies; Clinical setting; Heterogeneity; Medical interventions; Placebo; Safety analysis; Quality journals; Health outcomes


Comparative Effectiveness Research (CER) studies compare the relative effectiveness of different medical interventions in the diagnosis or treatment of a specific disease. In recent years, CER has garnered the advocacy and support of the US Institute of Medicine (IOM), numerous government healthcare agencies, and international organizations [1-3]. According to the formal definition by the IOM, CER is “the generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor or improve the delivery of care”. CER has the potential of providing information “to assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve health care at both the individual and population levels” [4]. The two main characteristic aspects of CER studies are the comparison of active treatments (active comparator studies) rather than the comparison of an active treatment with a placebo or the absence of a comparator, and the evaluation of the effects of interventions in a real-world setting. CER studies should contrast alternatives that are already used in the clinical setting but for which effectiveness data are not available or are insufficient to demonstrate superiority over other treatments [5-7]. The comparison interventions in CER may include medications, procedures, medical and assistive devices and technologies, behavioural change strategies, and delivery systems.

Apart from these two characteristics, the definition of CER suggests that the study of effectiveness cannot be the only objective of the programmes. Aspects such as safety and cost are relevant elements as well [7]. Additionally, CER studies should not only assess the best treatment option for the average patient but also analyse the best option for each individual patient [8]. Since the available evidence may be incomplete or does not apply to certain patient populations, such as children or the elderly, organisations such as the Patient-Centred Outcomes Research Institute (PCORI) promotes CER studies to generate new evidence in different health care interventions and across a broad range of medical conditions and patient populations [9,10].

Despite the extensive amount of CER literature, little is known about the characteristics of published CER studies and how they fit into the CER concept. Therefore, the goal of this work was to characterize relevant CER studies to analyse if they met comparative effectiveness major objectives.


The MEDLINE and EMBASE databases were searched using the search terms “Comparative AND effectiveness [Title] AND ("2009/05/18" [PDat]: "2015/05/18" [PDat])”. All original articles with a title containing the words "comparative" and "effectiveness" were selected. The search was limited to articles published between 2009 and 2015 in general-medicine journals with the highest 2013 impact factor (NEJM, IF: 54.42; Lancet, IF: 39.20; JAMA-J Am Med Assoc, IF: 30.38; Brit Med J, IF: 16.37; Ann Intern Med, IF: 16.10; PLOS Med, IF: 14.00; Arch Intern Med, IF: 13.24; BMC Med, IF: 7.27; Cochrane DB Syst Rev, IF: 5.93). After the search and the elimination of duplicates, 93 articles were obtained. These articles were manually reviewed to eliminate Opinions and Editorials. Finally, 40 articles met the requirements. These articles were double-checked by two independent investigators to extract the relevant data included in this analysis. Disagreement was resolved by consensus.

The following variables were analysed in each of the selected articles: use and type of treatment (active or placebo), safety as primary or secondary objective, cost assessment, inclusion of Patient Reported Outcomes (PROs), analysis of the heterogeneity in the response, type of intervention evaluated, study design, therapeutic area, and funding source. The variables were analysed using descriptive statistics.


Study characteristics of the sample of 40 CER publications included in this review are presented in Table 1. Of these, 25 were comparative studies of medicines and 15 of other types of medical intervention. Independently of the intervention evaluated, and in relation to the most important features of CER studies, we found that active comparators were used in all of the studies, and 20% of them also used a placebo. 62.5% described comparisons between medications, while 37.5% studied other types of medical interventions (e.g., behavioural, surgical devices, diagnostic strategies).

  Medications (N=25) Other interventions (N=15) Total (N=40)
Reference number [29-53] [54-68] -
Active vs. inactive comparators
Active 25 15 40 (100%)
Placebo 8 0 8 (20%)
Type of comparator
Different drugs 24 2 26 (65%)
Non-pharmacological interventions 3 10 13 (33%)
Different strategies 1 4 5 (13%)
Safety (adverse effects)
Yes (primary or secondary) 17 9 26 (65%)
No 8 6 14 (35%)
Cost analysis
Yes 1 0 1 (3%)
No 24 15 39 (97%)
Inclusion of PROs
Yes 2 0 2 (5%)
No 23 15 38 (95%)
Assess the heterogeneity in the response
Yes 10 6 16 (40%)
Subgroup analysis 9 5 14
Propensity score 4 4 8
Risk adjustment 0 0 0
No 15 9 24 (60%)
Study design
Randomised clinical trials (RCT) 2 4 6 (15%)
Observational (prospective or retrospective) 4 5 9 (23%)
Meta-analysis/Systematic review 17 5 22 (55%)
Model 2 1 3 (8%)
Therapeutic area
Pain 1 0 1
Cardiovascular 7 6 13
Infectious 4 0 4
Arthritis 1 0 1
Diabetes 4 3 7
Glaucoma 1 0 1
Osteoporosis 1 0 1
Chronic obstructive pulmonary disease 1 0 1
Central nervous system 2 1 3
Oncology 2 3 5
Others 1 2 3
Public 18 12 30 (75%)
Private 7 3 10 (25%)

Table 1: Characteristics of the CER studies analysed.

The safety of the interventions was evaluated in 65% of the studies as primary or secondary objective. Cost analyses were performed in 3% of the papers. Respect to the "patient centricity” of the studies, PROs were included in 5% of the publications and 40% of them used some procedure to determine the heterogeneity of the response.

The most common CERs were meta-analyses or systematic reviews (55%), followed by observational studies (23%) and Randomized Clinical Trials (RCTs) (15%). With regard to the source of funding, 75% of the publications received public support. In our sample, the most represented medical field is the cardiovascular area, followed by diabetes.


The results of this study show that the CER articles analysed did not meet the criteria commonly accepted for CER studies. Although all papers included active-drug comparators, 20% also included comparisons with placebo. As previously stated, comparison groups in CER studies should reflect clinical choices in real world practice. The comparison of interventions with a clinically meaningful alternative is usually a better choice from a methodological perspective than the comparison with an untreated group. The option of “no treatment” may not meet usual standards of care where multiple therapeutic options are available and should be limited only to certain clinical situations. Our finding about the use of placebo in CER studies can be partially explained by the methodology used in many of the studies analysed, which were meta-analyses and systematic reviews that included RTCs with different treatment comparison arms including placebo.

Safety analysis was included in 65% of the publications. Since the ultimate goal of CER is to provide decision makers with accurate and scientifically-rigorous information for comparing alternative clinical options, CER studies should systematically include not just the effectiveness but also an evaluation of safety. We observe an imbalance in the measuring of the treatment efficacy over treatment risk and safety. Although the methodology to assess the safety of medical interventions is less standardized than the assessment of their efficacy, we believe than CER studies should focus on the assessment of the risk-benefit of the compared interventions [11].

Regarding cost comparisons, an economic analysis of the alternatives was performed in only 3% of the studies. Although CER should allow making clinical decisions in an environment of limited resources, this is a controversial point with which not all stakeholders agree [12]. According to the detractors, if economic considerations were included, there would be many interventions with small but positive health benefits that would not be compensated by the additional cost and that, therefore, would not be recommended. However, as some medical organizations have recently highlighted, performing Cost-Effectiveness Analysis (CEA) is important to compare and contextualize the price of any intervention [12-14].

With regard to patients’ perspectives, 5% of the analysed studies collected PROs. A fully informative CER should incorporate patients’ perspectives standardised in the form of PROs [15]. Patients’ perspectives can generate valuable information for clinical health professionals in their decision-making processes [16-18]. The importance of incorporating PROs has been emphasised by regulatory (i.e., the U.S. Food and Drug Administration) and National Health Agencies (i.e., the U.S. National Institutes of Health) [19] as well as by organisations such as PCORI, with the objective of developing patientcentred medicine [3]. However, our analysis highlights the fact that few studies take patient perspectives into consideration.

Regarding the assessment of the heterogeneity of the response in patient subgroups, the goal of CER is to identify interventions that work best in specific subtypes of patients. The majority of the studies included in this study (60%) did not use any procedures directed at identifying uniqueness in the response of patient subgroups. In order to increase the generalizability of the results, CER studies usually use wide selection criteria. However, it is increasingly evident that although most therapeutic decisions rely on information obtained based on the average patient, the one-size-fits-all approach is not the ideal to optimise healthcare as it does not cover the singularities of the individual patient [20,21]. CER programmes should include patientoriented research methods to assess the responses in individuals and subgroups with the objective of "particularising" the results [8,22-24]. Therefore, current research falls short of the ambitious goal of improving “health care at both the individual and population levels”, as stated in the CER definition.

Although pragmatic RCTs could be considered as an adequate design for the evaluation of CER, other designs such as observational studies or mixed treatment comparisons (MTCs) [25], systematic reviews, meta-analyses, or decision analysis models [26] are also recommended. In our review, most of the CER studies were systematic reviews or meta-analyses (55%), followed by observational studies (23%) and clinical trials (15%) (Of which half were pragmatic RCTs). The low number of pragmatic RCTs agrees with the results of other reviews [27,28]. The implementation of CER based on pragmatic RCTs and observational studies should be encouraged.

This study has a number of limitations. The most important one is the potential information bias caused by the inclusion of only highimpact- factor general medicine journals. Our search selected only those articles that included the words "comparative" and "effectiveness" in the title. However, the biases introduced by the selection of journals and articles support our hypothesis because by including CER studies of high quality, one would expect that they would better comply with the theoretical requirements of these studies. It is foreseeable that the identified problems will be even greater with less stringent criteria, including lower quality journals and articles that do not include the CER concept in their titles.


Our study shows that most of the CERs analysed did not adhere to the five basic requirements that these studies should fulfil. Among other issues, they often included comparisons with placebo, did not evaluate the safety or the cost of the interventions, did not usually include PROs, and did not take into account heterogeneity in the response. In our opinion, it is necessary to encourage true CER studies primarily based on pragmatic RCTs or observational studies carried out under conditions of actual clinical practise. Only in this way it will be possible to achieve higher quality healthcare systems capable of improving the health outcomes of individual patients.


Select your language of interest to view the total content in your interested language
Post your comment

Share This Article

Relevant Topics

Article Usage

  • Total views: 210
  • [From(publication date):
    June-2017 - Sep 22, 2017]
  • Breakdown by view type
  • HTML page views : 171
  • PDF downloads :39

Post your comment

captcha   Reload  Can't read the image? click here to refresh

Peer Reviewed Journals
Make the best use of Scientific Research and information from our 700 + peer reviewed, Open Access Journals
International Conferences 2017-18
Meet Inspiring Speakers and Experts at our 3000+ Global Annual Meetings

Contact Us

Agri, Food, Aqua and Veterinary Science Journals

Dr. Krish

[email protected]

1-702-714-7001 Extn: 9040

Clinical and Biochemistry Journals

Datta A

[email protected]

1-702-714-7001Extn: 9037

Business & Management Journals


[email protected]

1-702-714-7001Extn: 9042

Chemical Engineering and Chemistry Journals

Gabriel Shaw

[email protected]

1-702-714-7001 Extn: 9040

Earth & Environmental Sciences

Katie Wilson

[email protected]

1-702-714-7001Extn: 9042

Engineering Journals

James Franklin

[email protected]

1-702-714-7001Extn: 9042

General Science and Health care Journals

Andrea Jason

[email protected]

1-702-714-7001Extn: 9043

Genetics and Molecular Biology Journals

Anna Melissa

[email protected]

1-702-714-7001 Extn: 9006

Immunology & Microbiology Journals

David Gorantl

[email protected]

1-702-714-7001Extn: 9014

Informatics Journals

Stephanie Skinner

[email protected]

1-702-714-7001Extn: 9039

Material Sciences Journals

Rachle Green

[email protected]

1-702-714-7001Extn: 9039

Mathematics and Physics Journals

Jim Willison

[email protected]

1-702-714-7001 Extn: 9042

Medical Journals

Nimmi Anna

[email protected]

1-702-714-7001 Extn: 9038

Neuroscience & Psychology Journals

Nathan T

[email protected]

1-702-714-7001Extn: 9041

Pharmaceutical Sciences Journals

John Behannon

[email protected]

1-702-714-7001Extn: 9007

Social & Political Science Journals

Steve Harry

[email protected]

1-702-714-7001 Extn: 9042

© 2008-2017 OMICS International - Open Access Publisher. Best viewed in Mozilla Firefox | Google Chrome | Above IE 7.0 version