alexa Methods for Analysis of Pre-Post Data in Clinical Research: A Comparison of Five Common Methods | Open Access Journals
ISSN: 2155-6180
Journal of Biometrics & Biostatistics
Like us on:
Make the best use of Scientific Research and information from our 700+ peer reviewed, Open Access Journals that operates with the help of 50,000+ Editorial Board Members and esteemed reviewers and 1000+ Scientific associations in Medical, Clinical, Pharmaceutical, Engineering, Technology and Management Fields.
Meet Inspiring Speakers and Experts at our 3000+ Global Conferenceseries Events with over 600+ Conferences, 1200+ Symposiums and 1200+ Workshops on
Medical, Pharma, Engineering, Science, Technology and Business

Methods for Analysis of Pre-Post Data in Clinical Research: A Comparison of Five Common Methods

Nathaniel S. O'Connell, Lin Dai, Yunyun Jiang, Jaime L. Speiser, Ralph Ward, Wei Wei, Rachel Carroll and Mulugeta Gebregziabher*

Department of Public Health Sciences, Medical University of South Carolina, Charleston, South Carolina, USA

*Corresponding Author:
Gebregziabher M
Department of Public Health Sciences
Medical University of South Carolina
135 Cannon St, MSC 835, Charleston
SC 29425, USA
Tel: 843-876-1112
Fax: 843-876-1126
E-mail: [email protected]

Received Date: Janaury 23, 2017; Accepted Date: February 21, 2017; Published Date: February 24, 2017

Citation: O'Connell NS, Dai L, Jiang Y, Speiser JL, Ward R, et al. (2017) Methods for Analysis of Pre-Post Data in Clinical Research: A Comparison of Five Common Methods. J Biom Biostat 8: 334. doi:10.4172/2155-6180.1000334

Copyright: © 2017 O'Connell NS, et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Visit for more related articles at Journal of Biometrics & Biostatistics

Abstract

Often repeated measures data are summarized into pre-post-treatment measurements. Various methods exist in the literature for estimating and testing treatment effect, including ANOVA, analysis of covariance (ANCOVA), and linear mixed modeling (LMM). Under the first two methods, outcomes can either be modeled as the post treatment measurement (ANOVA-POST or ANCOVA-POST), or a change score between pre and post measurements (ANOVACHANGE, ANCOVA-CHANGE). In LMM, the outcome is modeled as a vector of responses with or without Kenward- Rogers adjustment. We consider five methods common in the literature, and discuss them in terms of supporting simulations and theoretical derivations of variance. Consistent with existing literature, our results demonstrate that each method leads to unbiased treatment effect estimates, and based on precision of estimates, 95% coverage probability, and power, ANCOVA modeling of either change scores or post-treatment score as the outcome, prove to be the most effective. We further demonstrate each method in terms of a real data example to exemplify comparisons in real clinical context.

Keywords

Analysis of variance; Analysis of covariance; Linear mixed model; Pre-post; Rrandomized trial; Repeated measures

Abbreviations

ANCOVA: Analysis of covariance; ANCOVACHANGE: Analysis of covariance using change score as the outcome; ANCOVA-POST: Analysis of covariance using post-treatment as the outcome; ANOVA: Analysis of variance; ANOVA-CHANGE: Analysis of variance using change score as the outcome; ANOVA-POST: Analysis of variance using post-treatment as the outcome; CI: Confidence interval; HCS: Heterogeneous compound symmetry; KR: Kenward- Rogers; LMM: Linear mixed model; MAR: Missing at random: MCAR Missing completely at random; MNAR: Missing not at random; REML: Restricted maximum likelihood; SD Standard deviation.

Introduction

In clinical research, it is common to record repeated measurements for subject responses across multiple occasions. There exist a variety of analysis methods, including repeated measures analysis of variance (RANOVA), multivariate ANOVA (MANOVA) and linear mixed modeling (LMM). Researchers often simplify repeated measures data by using summary data to quantify pre-and post-treatment outcomes, allowing for a more intuitive and easier interpretation of treatment comparisons [1]. When appropriately applied, simplifying repeated measure outcomes to only two time measurements in the design phase improves efficiency and cost effectiveness, particularly in circumstances when responses are expensive to measure. In many instances this is done because the response of patients at a certain final time point is more clinically relevant than trends over time. Rather than comparing trends over time within each treatment group, the pre-post treatment summary method also simplifies data analysis to standard t-test procedures.

Decades of literature exists exploring and comparing methods for pre-post analysis, in both theory and application. The goal of this paper is not in developing new methods of analysis, but to review and succinctly tie together existing literature into a cohesive comparison of common methods often discussed and employed. We revisit and review the basic methods of pre-post data analysis discussed in the literature, and then exemplify the results through simulation and real data examples to corroborate existing knowledge. The rest of the paper is structured as follows: we first provide a review of key literature in prepost analysis. We then outline the models to be compared and set up a simple simulation study to demonstrate the comparison of methods, and discuss simulated results in conjunction with the theoretical expectations of variance and related implicit measures. A real data example is used to exemplify the difference in methods in practice, and highlight the importance of a proper analysis method. We finish with a discussion of results and further present ideas for future avenues of research in the area of pre-post data analysis.

Review of Literature

Frison and Pocock [2] discuss three methods for analyzing data from pre-post designs: a) ANOVA with the post measurement as the response variable (ANOVA-POST), b) ANOVA with the change from pre-treatment to post-treatment as the response variable (ANOVACHANGE), and c) ANCOVA with the post measurement as the response variable (ANCOVA-POST), adjusting for the pre-treatment measurement. Brogan and Kutner [3] compare the use of ANOVACHANGE with RANOVA. However, Huck and McLean [4] criticize the latter method due to its frequent misinterpretation in practice. Furthermore, they note the F-test in an RANOVA interaction is equivalent to the F-test in change score analysis. RANOVA provides the same conclusion as ANOVA-CHANGE, but use of ANOVACHANGE is simpler and more accurately interpreted compared to RANOVA. These conclusions are defended by Jennings [5], who asserts RANOVA is not recommended for pre-post analysis given the simpler alternatives presented.

Among the methods, ANCOVA-POST is generally regarded as the preferred approach, given that it typically leads to unbiased treatment effect estimate with the lowest variance relative to ANOVAPOST or ANOVA-CHANGE [1-6]. However, ANCOVA has been criticized as being biased in the case of unequal pre-treatment mean measurements between groups [7,8]. This conundrum, known as “Lord’s Paradox,” was first documented in 1967 Lord [9], and has been discussed in the literature extensively. Among a detailed examination of various methods of repeated measures data analysis for pre-post outcomes, Liang and Zeger [10] note in the simple case with only two responses (i.e. pre- and post-treatment measurements), ANCOVAPOST produces an unbiased estimate only in the case of equal pretreatment measurements, whereas ANOVA-CHANGE leads to unbiased estimates that are only slightly less efficient than ANCOVAPOST. Senn [11] discusses these criticisms at length, providing various conditions for which these claims do not hold, ultimately concluding ANCOVA should be used with caution in the case of unequal pretreatment measurements, but ANOVA-CHANGE is not impervious to bias either.

A recent simulation study by Egbewale, Lewis, and Sim [12] over varying degrees of pre-treatment imbalance and pre-post treatment correlations, demonstrates that a comparison of the methods is not straightforward in the presence of unequal pre-treatment measures between groups. They recommend ANCOVA when pre-treatment measurements are equal in expectation across groups, as should be the case in properly designed randomized trials [12]. ANOVA-POST has a larger variance because it allows for possible random baseline imbalance for which it cannot adjust. ANCOVA allows adjustment for baseline differences and thus has a smaller variance than ANOVA. In further support for ANCOVA, Vickers and Altman [13] note that ANCOVA achieves the greatest power relative to ANOVA-CHANGE or ANOVA-POST, but the power of ANOVA-CHANGE approaches ANCOVA as correlation between pre-post measures approaches one.

Combining analysis of change scores with adjustments for pre-treatment measures, Laird [14] offers a slight modification to ANCOVA, in which the change score is incorporated as the outcome and pre-treatment measures as covariate. Compared with traditional ANCOVA, this ANCOVA-CHANGE leads to equal results in terms of variance of treatment effect, although Laird [14] asserts the latter method allows one to assess whether change occurred in individual treatment groups. Despite this possible advantage, this appears less frequently used or discussed in the literature.

In the rest of the paper, we will discuss and compare results between the five common methods ANOVA and ANCOVA modeling both the post-treatment response only and the change score, along with a linear mixed model (LMM) modeling the pre-post treatment response vector, Yij. We use simulations over a range of sample sizes and pre-post measurement correlations to corroborate the comparison of methods with the existing literature and theoretical expectations of variance

Methods

To set up the modeling framework, let Yi be the continuous response variable from a randomized trial, for i=1,…,n, patient responses from samples n1 and n2 from each treatment group. Let the group assignment be designated by the indicator, Xi, such that for the ith patient, Xi=1 for the active treatment and Xi=0 in the control/placebo group. Responses for each treatment are each sampled from a Gaussian distribution with mean μx and variance σ2, where μx01 Xi. To distinguish between post-treatment and change score measures as outcomes, let Y[p] i represent the post treatment response and Y[c] i represent the change from pre-treatment to post-treatment measurements. In the case of ANCOVA, let Y0i be the pre-treatment measurement for which the model is adjusted for. Lastly, let εi represent the random error terms for each of the models. Maximum likelihood is used to estimate the parameters corresponding to each model except for the variance in LMM which are estimated using restricted maximum likelihood (REML). Derivations of variance for each of the estimators from the different methods are given in the Appendix.

Method 1: ANOVA-POST

Method 1 uses linear regression to compare treatment effects. Formally, the model is as follows:

equation

It is assumed that εi are independently and identically normally distributed with mean 0 and variance σ2. β[p] 1 is interpreted as the difference in the post-treatment mean between treatment groups. The variance of the estimated treatment effect is given by:

equation

Method 2: ANOVA-CHANGE

Similar to ANOVA-POST, ANOVA-CHANGE employs a simple ANOVA framework, but instead models the outcome, Y[c] i without adjustment for pre-treatment values. Formally, the model with εi assumed to be ndependently and identically normally distributed with mean 0 and variance σ2 is given as follows:

equation

Here β[ p] i is interpreted as the difference in the change score mean of the treatment groups. Under an unstructured covariance structure assumption, the variance ofequation is given by:

equation

Under a compound symmetry assumption, where pre- and posttreatment

variance is assumed to be equal, the variance is given by:

equation

Method 3: ANCOVA-CHANGE

Method 3 employs an ANCOVA model to analyze the change score as an outcome, adjusting for the pre-treatment values. Essentially, ANCOVA-CHANGE is equivalent to ANOVA-CHANGE, with an added adjustment for the pre-treatment measurement for every patient. Formally, the model is as follows:

equation

It is assumed that εi are independently and identically normally distributed with mean 0 and variance σ2. β[c2] 1 is interpreted as the difference in the change score mean of the treatment groups, given the pre-treatment measurement and the variance of its estimator is given by

equation

Which, as the sample size increases, simplifies to:

equation

Method 4: ANCOVA-POST

Method 4 employs an ANCOVA model to analyze the posttreatment measurements as the outcome, adjusting for the pretreatment values. In the context of previous methods, ANCOVAPOST is essentially ANOVA-POST (method 1) with pre-treatment measurement included as a covariate. Formally, the model is as follows:

equation

It is assumed that εi are independently and identically normally distributed with mean 0 and variance σ2. β[a] 1 is interpreted as the difference in the post treatment score mean of the treatment groups, given the pre-treatment measurement. Since this method is equivalent to method 3 Laird [14], results including the variance of the estimated treatment effect β[a] 1 is the same.

Method 5: LMM

Method 5 consists of employing a linear mixed model (LMM) to analyze a vector of the pre-and post-measurements as the outcome. Yij denotes the jth measure of the ith subject. Formally, the model is as follows:

equation

Where tij is an indicator for pre-treatment measurement (coded 0) or post-treatment measurement (coded 1). In LMM, it is assumed that εij are bivariate normally distributed with means 0 and heterogeneous compound symmetric (HCS) covariance matrix and correlation, ρ. The term β[b] 1+ β[b] 3 is interpreted as the mean difference between treatment groups post-treatment, and β[b] 1 is the mean difference between treatment groups pre-treatment. LMM allows for pretreatment mean differences between the groups.

The variance of equation under HCS covariance matrix for the error term is given by:

equation

If assuming compound symmetry (CS) is assumed, the variance is:

equation

The LMM was evaluated under REML estimation in PROC MIXED (SAS 9.3, SAS Institute Inc, Cary, NC). Acknowledging that many degree of freedom adjustments may be employed in mixed models, we choose to evaluate this approach with the conservative, and widely used Kenward and Rogers (KR) adjustment, as well as an unadjusted model. The KR adjustment Kenward and Roger [15]; Schaalje, McBride and Fellingham [16]; Senn [17] appropriately inflates the variancecovariance matrix, along with an adjusted degree of freedom estimate (KR degree of freedom adjustment) when making inference on fixed effects which rely on asymptotic distributions that can lead to biased variance estimates when sample sizes are small. No adjustment, according to Senn [17], leads to negligible difference with the first scenario as sample size grows reasonably large (e.g. n>40).

Simulation Study

Data are simulated using SAS 9.3. Simulations are designed to represent a variety of situations which are plausible in pre-post studies. Using 1000 repetitions, we compare models under three sample sizes (n=50, 100, and 200), under three pre-post correlations (ρ=0.1, 0.5, and 0.8), and six β1 coefficients for treatment effect (β1=- 1.5, -0.1, -1.0, 0.1, 1.0, and 1.5). Covariates are generated assuming X~uniform (0,1), and Y0~N(0,1); the post treatment response Y1 is generated using: Y1=10+1.5*{X ≥ 0.5} +1.5* Y0 +ε, such that {X ≥ 0.5} represents treatment 1 (i.e. Xi=1) and {X<0.5} represents control/ placebo (i.e. Xi=0). To generate correlation between pre- (Y0) and post- (Y1) treatment measures, we use the relationship between correlation and slope: equation , where σy0 and σy1 are the standard deviations for pre- and post-treatment responses, respectively. β is fixed at 1.5 and σy1 is calculated for each combination of σy0 and ρ. Random errors are generated such that ε ~N(0, σ2). The corresponding residual variance is calculated using the relationship between σy1 and the variance of ε for different β1 coefficients: equation . There are a total of 108 simulated scenarios among the combinations of n, ρ, and β1. Estimates for the parameter (equation) , its variance, bias, power, and nominal 95% coverage probability are computed for each simulation scenario, and the results are compared across the five methods. Code used to implement this may be found in supplementary material available online.

Simulation Results

In this section, findings from the simulations are discussed and compared to expected theoretical results. Treatment effect parameter estimates and associated standard deviations are reported in Table 1, while bias and power are presented in Tables 2 and 3 shows the 95% coverage probability. The results, consistent with theoretical expectations, show that all methods produce equally unbiased estimates of the treatment effect across equivalent combinations of ρ, β1, and n, with accuracy of the estimates improving with sample size. In general, as the number of observations increases, the bias converges to zero for each of the methods across all of the simulated scenarios. When ρ increases, the bias decreases, regardless of sample size or true β1.

Simulated β1 Values across methods ρ=0.1 ρ=0.5 ρ=0.8
n=50 n=100 n=200 n=50 n=100 n=200 n=50 n=100 n=200
0.1 ANOVA-POST 0.095(4.240) 0.013(3.065) 0.206(2.121) 0.114(0.856) 0.117(0.601) 0.110(0.447) 0.110(0.532) 0.096(0.386) 0.102(0.268)
ANOVA-CHANGE 0.095(4.220) 0.018(3.055) 0.198(2.115) 0.107(0.768) 0.110(0.537) 0.105(0.396) 0.104(0.343) 0.096(0.246) 0.101(0.177)
ANCOVA-CHANGE 0.110(4.261) 0.022(3.075) 0.201(2.119) 0.106(0.767) 0.106(0.535) 0.102(0.387) 0.101(0.312) 0.095(0.222) 0.100(0.159)
ANCOVA-POST 0.110(4.261) 0.022(3.075) 0.201(2.119) 0.106(0.767) 0.106(0.535) 0.102(0.387) 0.101(0.312) 0.095(0.222) 0.100(0.159)
LMM 0.095(4.240) 0.013(3.065) 0.206(2.121) 0.114(0.856) 0.117(0.601) 0.110(0.447) 0.110(0.532) 0.096(0.386) 0.102(0.268)
1.0 ANOVA-POST 1.104(4.424) 0.997(3.197) 0.912(2.075) 1.038(0.833) 1.007(0.598) 1.006(0.424) 1.028(0.504) 0.992(0.351) 1.006(0.256)
ANOVA-CHANGE 1.105(4.411) 1.008(3.178) 0.917(2.066) 1.031(0.723) 1.000(0.525) 1.005(0.379) 1.010(0.311) 0.988(0.219) 1.002(0.159)
ANCOVA-CHANGE 1.098(4.448) 1.022(3.188) 0.913(2.059) 1.025(0.712) 0.997(0.519) 1.004(0.373) 0.999(0.278) 0.986(0.199) 0.999(0.143)
ANCOVA-POST 1.098(4.448) 1.022(3.188) 0.913(2.059) 1.025(0.712) 0.997(0.519) 1.004(0.373) 0.999(0.278) 0.986(0.199) 0.999(0.143)
LMM 1.104(4.424) 0.997(3.197) 0.912(2.075) 1.038(0.833) 1.007(0.598) 1.006(0.424) 1.028(0.504) 0.992(0.351) 1.006(0.256)
1.5 ANOVA-POST 1.557(4.295) 1.668(3.048) 1.577(2.094) 1.525(0.831) 1.520(0.578) 1.522(0.412) 1.503(0.504) 1.492(0.347) 1.487(0.242)
ANOVA-CHANGE 1.567(4.277) 1.650(3.039) 1.579(2.083) 1.526(0.724) 1.518(0.508) 1.526(0.361) 1.502(0.290) 1.494(0.199) 1.489(0.140)
ANCOVA-CHANGE 1.556(4.313) 1.638(3.057) 1.586(2.086) 1.528(0.718) 1.516(0.502) 1.530(0.354) 1.500(0.250) 1.496(0.170) 1.491(0.119)
ANCOVA-POST 1.556(4.313) 1.638(3.057) 1.586(2.086) 1.528(0.718) 1.516(0.502) 1.530(0.354) 1.500(0.250) 1.496(0.170) 1.491(0.119)
LMM 1.557(4.295) 1.668(3.048) 1.577(2.094) 1.525(0.831) 1.520(0.578) 1.522(0.412) 1.503(0.504) 1.492(0.347) 1.487(0.242)
 

Table 1: Mean Parameter and standard deviation estimates across 1000 simulations for positive β1 values.

Simulatedβ1 Values across methods   ρ=0.1 ρ=0.5 ρ=0.8
n=50 n=100 n=200 n=50 n=100 n=200 n=50 n=100 n=200
0.1 ANOVA-POST -0.005 -0.087 0.106 0.014 0.017 0.010 0.010 -0.004 0.002
0.056 0.062 0.049 0.053 0.060 0.070 0.056 0.060 0.069
ANOVA-CHANGE -0.005 -0.082 0.098 0.007 0.010 0.005 0.004 -0.004 0.001
0.056 0.061 0.049 0.051 0.061 0.066 0.058 0.064 0.091
ANCOVA-CHANGE 0.010 -0.078 0.101 0.006 0.006 0.002 0.001 -0.005 0.000
0.057 0.062 0.045 0.054 0.060 0.060 0.071 0.068 0.107
ANCOVA-POST 0.010 -0.078 0.101 0.006 0.006 0.002 0.001 -0.005 0.000
0.057 0.062 0.045 0.054 0.060 0.060 0.071 0.068 0.107
LMM -0.005 -0.087 0.106 0.014 0.017 0.010 0.010 -0.004 0.002
0.056 0.062 0.049 0.053 0.060 0.070 0.056 0.060 0.069
1.0 ANOVA-POST 0.104 -0.003 -0.088 0.038 0.007 0.006 0.028 -0.008 0.006
0.058 0.067 0.088 0.211 0.382 0.644 0.490 0.794 0.976
ANOVA-CHANGE 0.105 0.008 -0.083 0.031 0.000 0.005 0.010 -0.012 0.002
0.059 0.066 0.086 0.267 0.470 0.756 0.856 0.988 1.000
ANCOVA-CHANGE 0.098 0.022 -0.087 0.025 -0.003 0.004 -0.001 -0.014 -0.001
0.060 0.065 0.085 0.275 0.479 0.772 0.920 0.999 1.000
ANCOVA-POST 0.098 0.022 -0.087 0.025 -0.003 0.004 -0.001 -0.014 -0.001
0.060 0.065 0.085 0.275 0.479 0.772 0.920 0.999 1.000
LMM 0.104 -0.003 -0.088 0.038 0.007 0.006 0.028 -0.008 0.006
0.058 0.067 0.088 0.211 0.382 0.644 0.490 0.794 0.976
1.5 ANOVA-POST 0.057 0.168 0.077 0.025 0.020 0.022 0.003 -0.008 -0.013
0.058 0.094 0.102 0.412 0.713 0.949 0.848 0.987 1.000
ANOVA-CHANGE 0.067 0.150 0.079 0.026 0.018 0.026 0.002 -0.006 -0.011
0.058 0.094 0.105 0.522 0.825 0.985 1.000 1.000 1.000
ANCOVA-CHANGE 0.056 0.138 0.086 0.028 0.016 0.030 0.000 -0.004 -0.009
0.057 0.088 0.107 0.527 0.836 0.987 1.000 1.000 1.000
ANCOVA-POST 0.056 0.138 0.086 0.028 0.016 0.030 0.000 -0.004 -0.009
0.057 0.088 0.107 0.527 0.836 0.987 1.000 1.000 1.000
LMM 0.057 0.168 0.077 0.025 0.020 0.022 0.003 -0.008 -0.013
0.058 0.094 0.102 0.412 0.713 0.949 0.848 0.987 1.000

Table 2: Power and bias estimates from 1000 simulations for true values of β1.

Simulated β1Values Across Methods ρ=0.1 ρ=0.5 ρ=0.8
n=50 n=100 n=200 n=50 n=100 n=200 n=50 n=100 n=200  
0.1 ANOVA-POST 0.939 0.949 0.952 0.957 0.960 0.939 0.959 0.943 0.953  
ANOVA-CHANGE 0.936 0.950 0.950 0.949 0.947 0.946 0.940 0.936 0.956  
ANCOVA-CHANGE 0.935 0.950 0.949 0.947 0.950 0.952 0.944 0.942 0.948  
ANCOVA-POST 0.935 0.950 0.949 0.947 0.950 0.952 0.944 0.942 0.948  
LMM 0.939 0.949 0.952 0.957 0.960 0.939 0.959 0.943 0.953  
1.0 ANOVA-POST 0.937 0.951 0.945 0.944 0.951 0.955 0.954 0.941 0.960  
ANOVA-CHANGE 0.941 0.952 0.947 0.950 0.954 0.951 0.962 0.952 0.959  
ANCOVA-CHANGE 0.945 0.952 0.947 0.943 0.950 0.951 0.955 0.947 0.956  
ANCOVA-POST 0.945 0.952 0.947 0.943 0.950 0.951 0.955 0.947 0.956  
LMM 0.937 0.951 0.945 0.944 0.951 0.955 0.954 0.941 0.96  
1.5 ANOVA-POST 0.954 0.947 0.951 0.957 0.957 0.951 0.964 0.951 0.948  
ANOVA-CHANGE 0.953 0.948 0.952 0.954 0.953 0.954 0.961 0.952 0.950  
ANCOVA-CHANGE 0.950 0.942 0.953 0.953 0.947 0.949 0.960 0.954 0.952  
ANCOVA-POST 0.950 0.942 0.953 0.953 0.947 0.949 0.960 0.954 0.952  
LMM 0.954 0.947 0.951 0.957 0.957 0.951 0.964 0.951 0.948  
-0.1 ANOVA-POST 0.948 0.954 0.959 0.952 0.961 0.952 0.964 0.949 0.944  
ANOVA-CHANGE 0.947 0.953 0.962 0.951 0.961 0.950 0.965 0.952 0.945  
ANCOVA-CHANGE 0.950 0.952 0.959 0.953 0.954 0.950 0.959 0.955 0.944  
ANCOVA-POST 0.950 0.952 0.959 0.953 0.954 0.950 0.959 0.955 0.944  
LMM 0.948 0.954 0.959 0.952 0.961 0.952 0.964 0.949 0.944  
-1.0 ANOVA-POST 0.953 0.954 0.957 0.955 0.959 0.953 0.957 0.959 0.955  
ANOVA-CHANGE 0.953 0.953 0.956 0.955 0.959 0.948 0.957 0.948 0.955  
ANCOVA-CHANGE 0.954 0.951 0.956 0.952 0.948 0.95 0.962 0.950 0.949  
ANCOVA-POST 0.954 0.951 0.956 0.952 0.948 0.95 0.962 0.950 0.949  
LMM 0.953 0.954 0.957 0.955 0.959 0.953 0.957 0.959 0.955  
-1.5 ANOVA-POST 0.958 0.948 0.946 0.956 0.950 0.950 0.954 0.954 0.948  
ANOVA-CHANGE 0.958 0.947 0.944 0.957 0.949 0.952 0.957 0.953 0.950  
ANCOVA-CHANGE 0.958 0.951 0.942 0.96 0.949 0.947 0.955 0.950 0.951  
ANCOVA-POST 0.958 0.951 0.942 0.96 0.949 0.947 0.955 0.950 0.951  
LMM 0.958 0.948 0.946 0.956 0.950 0.950 0.954 0.954 0.948  

Table 3: 95% confidence interval coverage probabilities from 1000 simulations.

Since all methods result in unbiased estimates for the treatment effect, we use variance and other implicit measures of the estimates to compare the five methods (Table 1 and Figure 1). In general, when the correlation between pre-treatment and post-treatment values is high, the variance of the estimates are relatively small, regardless of method and the value of β1. However, for higher correlation values, differences in variability between methods become more apparent when the sample size and value of β1 are fixed. For example, in the scenario when β1=0.1 and n=200, the difference between the largest and smallest standard deviations among the 5 methods with low correlation (ρ=0.1), is 0.006. However, under the same scenario with high correlation (ρ=0.8), the difference between the largest and smallest standard deviations is 0.109. While the individual standard deviation estimates are greater in low correlation scenarios, the difference between estimates of differing methods are much more pronounced in scenarios with high correlation, i.e. the greatest variability in β1 occurrs when ρ is close to 0 (i.e. ρ= 0.1). Furthermore, as sample size increases, variability decreases as expected theoretically. To summarize in terms of correlation and sample size, variability of β1 estimates are greatest when n=50 and ρ=0.1, and they are lowest when n=200 and ρ=0.8.

biometrics-biostatistics-survival

Figure 1: Distribution of treatment effects estimates varied by correlation, sample size and true positive β1 values under Y0~N(0,1).

Comparing the two ANOVA methods specifically, ANOVACHANGE produces approximately equal or less variability compared to ANOVA-POST, the difference of which increases as ρ approaches one. Intuitively, it follows that ignoring pre-treatment observations in ANOVA-POST causes a loss of information which leads to an increase in variance estimates when pre-treatment and post-treatment values are correlated. However, when correlation is low, results are dependent the on pre-post covariance structure. These discrepancies are addressed and explained in detail in following paragraphs. Table 1 displays the difference between variance estimates which grows with increasing correlation, holding sample size and true β1 parameters constant.

The two scenarios of ANCOVA (ANCOVA-POST and ANCOVACHANGE) give identical measures of variability, regardless of differing combinations of ρ, β1 and n. These results can be seen numerically in Table 1 and visually assessed in Figure 1, and are further supported by theoretical derivation of variance. Lastly, the LMM analysis results are equivalent to the ANOVA-POST in terms of estimates and their standard deviations, regardless of the assumed within-subject covariance structure, Ri. The LMM approach is evaluated with and without a KR adjustment, ultimately showing no difference. These results are consistent with the literature, given the smallest simulated sample size was n=50.

Across all methods, ANCOVA models consistently performed best compared to the other methods, regardless of ρ, β1 and n, as has been demonstrated in existing literature [18]. The ANCOVA methods (ANCOVA-CHANGE and ANCOVA-POST) are compared with ANOVA-POST, ANOVA-CHANGE and LMM in terms of variance of the estimate of β1 (Figure 1). Except at the lowest combination of ρ and n values where discrepancies between differing method variances is negligible, ANCOVA had the lowest variability. Similarly, the 95% coverage intervals are smallest under ANCOVA, and widest in ANOVA-POST (Table 4). Compared to ANOVA-CHANGE, ANCOVA models have less variation. These results are also similar when ANCOVA is compared with LMM, except when the correlation is close to zero.

Method equation 95% Confidence Interval SE(equation) P-value
1st Session
ANOVA-POST 0.16 -0.09 0.41 0.123 0.192
ANOVA-CHANGE 0.14 0.05 0.24 0.047 0.005
ANCOVA-CHANGE 0.14 0.05 0.24 0.046 0.004
ANCOVA-POST 0.14 0.05 0.24 0.046 0.004
LMM 0.16 -0.09 0.41 0.123 0.191
4th Session
ANOVA-POST -0.08 -0.28 0.12 0.099 0.409
ANOVA-CHANGE 0.09 -0.02 0.21 0.056 0.107
ANCOVA-CHANGE 0.07 -0.05 0.19 0.059 0.243
ANCOVA-POST 0.07 -0.05 0.19 0.059 0.243
LMM -0.08 -0.28 0.12 0.099 0.409

Table 4: Estimates of toothbrush effect on bacterial plaque index in 1st and 4th sessions.

It is of interest to compare the simulations to theoretical expectations for the variance of the treatment effect estimates. As the correlation between pretreatment and post-treatment observations approaches zero, the variance for ANOVA-POST, ANCOVA-POST, ANCOVA-CHANGE, and LMM should be approximately equivalent, but the variance for ANOVA-CHANGE is two times that of the others. When ρ=0.1 and assuming equal variance, the variance of ANOVACHANGE should theoretically approach 2(1-ρ=0.1), or 1.8 times that of ANOVA-POST, and ANCOVA methods should approach (1-ρ2), or 0.99 of ANOVA-POST. Similarly, for ρ = 0.1, the ratio of ANOVAPOST to ANOVA-CHANGE should be 1 and ANOVA-POST to ANCOVA should be 0.75. Finally, at ρ=0.8, ANOVA-CHANGE and both ANCOVA models should be at 0.4 and 0.36 times ANOVA-POST respectively. However, permitting pre and post measurements to have different variances leads to differing results, particularly apparent when correlation is low, such that the variance of ANOVA-CHANGE approaches the variance of ANOVA-POST as ρ approaches zero and the variance of post-measures becomes increasingly greater than premeasures. Presented simulations are for data simulated assuming this unstructured covariance matrix. For example, the variance of ANOVACHANGE and ANOVA-POST are approximately equal when ρ=0.1; had we assumed equalvariance, ANOVA-CHANGE would be almost twice that of ANOVA-POST. These are made apparent given the following ratio of variances:

As a result, the variance of ANOVA-CHANGE approaches the variance of ANOVA-POST as ρ approaches zero and β1>0.5 Under the given method of simulating pre and post treatment variances where equation and equation, the ratio of the variances is given by:

equation

Since the HCS structure is more conservative in its assumptions (permitting the pre- and post-treatment effects to have different variances), it is used to report the main simulation results of the study. As correlation increases beyond 0.5, results become less sensitive to the pre-post measure covariance structure.

Power for testing β1 = 0 was assessed for the methods under the simulated conditions (Table 2). Under the primary simulation method assuming Y0~N(0,1), power across methods did not vary by a large degree. Observing power across simulated scenarios at the lowest sample size (n=50), there is marginally higher power in ANCOVA and ANOVA-CHANGE methods over ANOVA-POST and LMM. Overall, ANCOVA methods achieve the greatest power, intuitively so given ANCOVA leads to the lowest variability. As correlation between pre-and post-measurements increase, the difference in power between ANCOVA and ANOVA-CHANGE compared to ANOVA-POST and LMM, grows appreciably, while ANOVA-CHANGE nears that of ANCOVA as correlation approaches one. Finally, increases in sample size leads to increased power for detecting a significant treatment effect similarly across methods, meaning that an increase in sample size does not appear to affect any single methods statistical power more than other methods. Additional results from simulations with an increased variance (assuming Y0 ~N(0,9)), are reported.

Data Example

To illustrate the application of the five methods, we consider data from a dental hygiene study characterized by small sample size [19]. There were a total of 32 subjects, randomized to two treatment groups based on type of toothbrush, and effectiveness was measured by reduction in bacterial plaque index over time. Of the original four independent sessions, we analyzed data from the first and last sessions, comparing pre and post treatment outcomes. The results for the first and last session are summarized in Table 4. In the first session, the pretreatment mean (standard error) for treatment group 1 and treatment group 2 are 1.31 (0.35) and 1.33 (0.38), respectively, and similarly for the last session, 1.54 (0.26) and 1.36 (0.27), respectively. The pretreatment measures between groups show no significant difference for either of the sessions. The correlations in the pre and post treatment measures are 0.91 and 0.82 for the first and last sessions, respectively. In line with simulations and theoretical expectations, it follows that under the first session, ANCOVA-CHANGE and ANCOVA-POST performed equally well among methods, presenting the lowest standard error for treatment effect. However, in the last session, ANOVACHANGE actually presented a slightly lower standard error compared to ANCOVA methods (0.0564 compared to 0.0592). Given the high correlation between pre and post measurements in this data set, it follows theoretical expectations that ANOVA-CHANGE is extremely close to that of ANCOVA models. In the case where ρ=0.91, ANOVACHANGE should theoretically produce variance that is 1.05 times ANCOVA. In both sessions, ANOVA and LMM exhibit larger variance than ANCOVA methods. The variance estimates from ANOVA and LMM are 2.7 times greater in the first session and 1.7 times greater in the second session than ANCOVA.

Discussion

This paper compares four traditional approaches (ANOVA-POST, ANOVA-CHANGE, ANCOVA-POST, and ANCOVA-CHANGE) and a more modern approach (LMM) used in the analysis of pre-post data. These five methods are compared via simulated data from a typical clinical trials setting, where pre-treatment groups are assumed equally allocated through proper randomization, and the primary interest is to examine estimates of treatment effect. Comparisons of these methods have been investigated in theoretical framework (Brogan and Kutner [3]; Dimitrov and Rumrill [4]; Frison and Pocock [2]; Huck and McLean [4]; Laird [14]), and in a similar manner as Egbewale et al. [12] we review these methods and discuss them in terms of several simulated circumstances, as well as a real data application.

Overall, all of the five methods in the simulated scenarios yield equally unbiased treatment effect estimate. However, their performance (in terms of variance, type-I error and 95% CI coverage) varies with pretreatment group differences as indicated in previous literature [7,11,12]. For example, LMM is found to be more conservative compared to the ANCOVA methods. Consistent with previous literature, ANCOVA models have the smallest variance, highest power, and nominal 95% confidence interval coverage compared to ANOVA-POST, ANOVACHANGE, and LMM. Similar to conclusions reached by Vickers and Altman [13], in our simulation study as correlation between pre-and post-treatment measures increase, ANOVA-CHANGE approaches ANCOVA in both variance and power. However, in all but the most extreme cases (i.e. when ρ ≈ 0 or 1), ANCOVA methods are the most optimal, achieving the greatest power and lowest variability. Thus, in the case of balanced pre-treatment data, our results are consistent with most existing literature, in that ANCOVA is a preferred method. This may not hold in situations with some degree of imbalance between treatment groups at baseline and different levels of pre-post correlation [12].

We also examine the robustness of these methods when the pre-post measures are simulated under CS versus HCS covariance structures. The data simulated under HCS produces the greatest effect on the ANOVA-CHANGE results, where treatment effect variances are particularly influenced at lower correlations. When pre and post treatment measures have equal variance and low correlation, ANOVAPOST outperforms ANOVA-CHANGE, but as the imbalance between pre-post variance grows, the two methods perform similarly. In practical applications, when one does not have control over preand post- treatment variances, results demonstrate that one could reasonably expect ANOVA-CHANGE to consistently perform better than ANOVA-POST when ρ ≤ 0.5 regardless of equality of variances. When ρ ≤ 0.5, the best performing method will depend on the degree of equality of the variances in pre-and post-measurements. Nevertheless, both methods are still consistently outperformed by ANCOVA.

In our simulation study, the LMM approach performs only as well as ANOVA-POST. However, these simulations assume no missing data. In clinical trials evaluating patients over time, on the other hand, missing data are common (i.e. some patients are lost to follow-up and post-treatment measurements are never recorded). In such cases, the mechanism of missing data as defined by Little and Rubin [20], along with method of analysis are important in reaching unbiased results. When data are missing completely at random (MCAR), the method of analysis makes little difference, i.e. the generalized least squares (GLS) based methods and LMM should provide equivalent results [21]. However, in the case when data are missing at random (MAR), GLS can lead to biased inference on effects, whereas the LMM approach which relies on likelihood provides unbiased results when the withinsubject covariance matrix, Ri, is specified correctly [21]. When data are missing not at random (MNAR), all five methods may lead to biased results. Thus, conclusions made in this study with regards to the LMM approach compared to the other approaches hold only when data are MCAR or not missing. In the case of MAR data, LMM may be more optimal.

Conclusion

Despite decades of long scrutiny of this topic and our extensive simulation study under a wide range of scenarios, there still remain several avenues of future work. For example, in the clinical trials setting, there are often many more important covariates which are included to deal with baseline covariate imbalance. Additionally, this study assumed that the treatment assignment was random, which is usually the case in clinical trials, resulting in equal pre-treatment values among treatment groups. As it was discussed in the introduction, results may differ when pre-treatment measures are unequal, particularly affecting the bias of ANCOVA-POST. Future work is needed to assess how ANCOVA-CHANGE as presented by Laird [14], performs under varying degrees of pre-treatment imbalance as in the simulation study performed by Egbewale [11]. Finally, these methods could be explored under the generalized linear mixed model framework with non- Gaussian pre-post data.

Acknowledgement

The study concept and design and acquisition of data is by MG. Analysis and interpretation of data, drafting of the manuscript, critical revision of the manuscript for important intellectual content and final approval of manuscript is made by all co-authors. The authors thank (names omitted to maintain integrity of the review process) for helpful comments. The manuscript represents the views of the authors. It is a byproduct of group project work by graduate students in an advanced statistical methods course.

References

Select your language of interest to view the total content in your interested language
Post your comment

Share This Article

Relevant Topics

Recommended Conferences

Article Usage

  • Total views: 445
  • [From(publication date):
    February-2017 - Jul 27, 2017]
  • Breakdown by view type
  • HTML page views : 364
  • PDF downloads :81
 

Post your comment

captcha   Reload  Can't read the image? click here to refresh

Peer Reviewed Journals
 
Make the best use of Scientific Research and information from our 700 + peer reviewed, Open Access Journals
International Conferences 2017-18
 
Meet Inspiring Speakers and Experts at our 3000+ Global Annual Meetings

Contact Us

 
© 2008-2017 OMICS International - Open Access Publisher. Best viewed in Mozilla Firefox | Google Chrome | Above IE 7.0 version
adwords