Determining the Probability Distribution and Evaluating Sensitivity and False Positive Rate of a Confounder Detection Method Applied To Logistic Regression

Background: In epidemiologic studies researchers are often interested in detecting confounding (when a third variable is both associated with and affects associations between the outcome and predictors). Confounder detection methods often compare regression coefficients obtained from “crude” models that exclude the possible confounder(s) and “adjusted” models that include the variable(s). One such method compares the relative difference in effect estimates to a cutoff of 10% with differences of at least 10% providing evidence of confounding. Methods: In this study we derive the asymptotic distribution of the relative change in effect statistic applied to logistic regression and evaluate the sensitivity and false positive rate of the 10% cutoff method using the asymptotic distribution. We then verify the results using simulated data. Results: When applied to a logistic regression models with a dichotomous outcome, exposure, and possible confounder, we found the 10% cutoff method to have an asymptotic lognormal distribution. For sample sizes of at least 300 the authors found that when confounding existed, over 80% of models had >10% changes in odds ratios. When the confounder was not associated with the outcome, the false positive rate increased as the strength of the association between the predictor and confounder increased. When the confounder and predictor were independent of one another, false positives were rare (most < 10%). Conclusions: Researchers must be aware of high false positive rates when applying change in estimate confounder detection methods to data where the exposure is associated with possible confounder variables. In practice, it is difficult to determine whether a variable will be a confounder in the population of interest. Data driven confounder detection methods often examine “crude” and “adjusted” estimates of effect size in samples drawn from the population [1,6-8]. One popular method is to evaluate the magnitude of confounding by examining the relative change in estimated effect size before and after adjusting for the possible confounder. The difference in effect size is divided by the crude effect estimate, providing the magnitude of the difference, relative to the unadjusted effect size. If the relative difference is “large” (i.e., > 10%) [2,6,8], investigators conclude that there is confounding [1,6-8]. When researchers detect confounding of the association between the exposure and outcome using this “change in estimate” approach, the model adjusted for the confounder is generally preferred [1,6-8]. Maldonado and Greenland (1993) applied crude and adjusted regression models to simulated data. They selected the preferable model based on the results of several confounder detection methods by evaluating the estimated effect size, the average relative bias, the Robin Bliss1,2, Janice Weinberg3, Thomas Webster1 and Veronica Vieira1,4* Bliss, J Biomet Biostat 2012, 3:4 DOI: 10.4172/2155-6180.1000142 1Department of Environmental Health, Boston University School of Public Health, Boston, MA, USA 2Orthopedic and Arthritis Center for Outcomes Research, Brigham and Women’s Hospital/Harvard Medical School, Boston, MA, USA 3Department of Biostatistics, Boston University School of Public Health, Boston, MA, USA 4Department of Population Health and Disease Prevention, Program in Public Health, University of California Irvine, USA *Corresponding author: Veronica Vieira, University of California Irvine, Irvine, CA, United States, Email:vvieira@uci.edu Received May 02, 2012; Accepted May 21, 2012; Published May 23, 2012 Citation: Bliss R, Weinberg J, Webster T, Vieira V (2012) Determining the Probability Distribution and Evaluating Sensitivity and False Positive Rate of a Confounder Detection Method Applied To Logistic Regression. J Biomet Biostat 3:142. doi:10.4172/2155-6180.1000142 Copyright: © 2012 Bliss R, et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. J o ur na l o f B iometrics & Bistatis t i c s ISSN: 2155-6180 Journal of Biometrics & Biostatistics


Introduction
Detecting confounding is an important part of research. Uncontrolled confounding can cause spurious, magnified, or minimized results and can produce effect sizes comparable to those observed in epidemiologic studies. Confounding is defined as the mixing of effects between an exposure, an outcome, and a third extraneous variable, known as a confounder [1][2][3]. Three standards are often used to describe confounder variables: (1) Confounders must be associated with the exposure of interest, (2) confounders must be associated with the outcome of interest, and (3) confounders must not lie on the causal pathway between the exposure and outcome [1,2]. In other words, a confounder must be a separate and distinct trait that is associated with, but not affected by, the exposure and the outcome of interest.
Causal pathway models and Directed Acyclic Graphs (DAG) are graphical displays that can be used to describe and illustrate relationships between variables, including confounding [4], as shown in Figure 1. Arrows indicate associations between variables with the direction illustrating the causal direction. As there is no path that can be traced from exposure to outcome through, the confounder does not fall on the pathway between E and Y. DAGs are useful tools to describe associations between variables without the need for regression assumptions. When qualitative associations between variables are unknown, DAGs are limited in requiring associations to be stated a priori and are therefore not appropriate for model building scenarios [4,5].
In practice, it is difficult to determine whether a variable will be a confounder in the population of interest. Data driven confounder detection methods often examine "crude" and "adjusted" estimates of effect size in samples drawn from the population [1,[6][7][8]. One popular method is to evaluate the magnitude of confounding by examining the relative change in estimated effect size before and after adjusting for the possible confounder. The difference in effect size is divided by the crude effect estimate, providing the magnitude of the difference, relative to the unadjusted effect size. If the relative difference is "large" (i.e., > 10%) [2,6,8], investigators conclude that there is confounding [1,[6][7][8]. When researchers detect confounding of the association between the exposure and outcome using this "change in estimate" approach, the model adjusted for the confounder is generally preferred [1,[6][7][8]. Maldonado and Greenland (1993) applied crude and adjusted regression models to simulated data. They selected the preferable model based on the results of several confounder detection methods by evaluating the estimated effect size, the average relative bias, the the estimate obtained from the selected model compared to the model underlying the simulated data. The authors focused their evaluation on the precision of estimates produced from single realizations of data. While their results reflect the precision of estimates as the confounder detection methods are applied in practice, Maldonado and Greenland did not evaluate whether the correct model was selected given the simulation parameters [6]. It is important to evaluate the sensitivity and false positive rate (1-specificity) of confounding detection methods to better understand the method reliability in providing accurate information under a variety of scenarios.
As data driven methods generally rely on some change statistic, it is reasonable to assume that the sensitivity and false positive rate of the detection method rely on the probability distribution of the statistic. A better understanding of this distribution may provide inference on the strengths and limitations of such confounder detection methods. To our knowledge, no studies have examined the probability distribution of the relative change in estimate statistic, how frequently the adjusted model is selected when confounding is present (sensitivity) or how often it is falsely detected when confounding is absent (false positive rate; 1-specificity). In this study, we derive the probability distribution of a 10% change in odds ratios rule for logistic regression results and use synthetic data to compare simulated results to derived results. Under a variety of scenarios we determine how often we expect to correctly detect confounding and how often we misclassify a change in estimated effect size observed by chance as confounding.

Derivation of Probability Distribution of the Change in Estimate Statistic
Suppose that we have an observable, dichotomous outcome, Y, such that Y~ Binomial(p) . If Y depends on two observable variables, E and C, the true association between Y, E, and C can be expressed as a function of the parameter p: We can represent the effect estimates of E and C as odds ratios, computed as exp(β 1 ) and exp(β 2 ). After observing Y, E, and C we may be interested in finding estimates for coefficients β 0 , β 1 , and β 2 to better understand the associations between the variables. While no closedform solution exists in general, maximum likelihood estimates for coefficients β 0 , β 1 , and β 2 can be found using the Newton-Raphson Algorithm. As a result, the estimates 0 β , 1 β , and 2 β have asymptotic normal distributions.
In practice, we may wish to determine whether the variable C is a confounder. To do this, we can compare two nested logistic regression models: equation 2 including variable E as the sole predictor variable and equation 3 including both E and C 0 1ˆl Note that if C is a confounder of the association between Y and E then we expect 1 1 θ ≠ γ and can look for evidence of confounding by evaluating the relative difference between effect size estimates. When using logistic regression, we typically compare the relative difference between odds ratios: We compare R to some predetermined cutoff, such as +/-0.10, to evaluate the likelihood of confounding.
We can find the probability density function of R to determine the probability of having evidence of confounding in a given scenario, i.e.

( )
Using the cumulative density function of the lognormal distribution, this can be expressed as: For large sample sizes, we can use simulated data to obtain estimated (empirical) values for θ, γ, τ, σ, and ρ, and compute the expected probability of detecting confounding.
Note that the derivation of the asymptotic distribution of is independent of the distributions of the exposure and confounder variables and the derivation is true for continuous as well as categorical predictor variables.

Simulated Data
Simulated data had a dichotomous outcome (Y), a dichotomous exposure (E), and a dichotomous third variable and possible confounder (C). Probability of exposure was range of effect sizes commonly observed in epidemiologic research. When the confounder C was present, the probability of the outcome was decreased by half, stayed the same, or was doubled: In scenarios where did not affect the probability of disease (k = 1) no confounding was present.
Associations between C and E were defined by a conditional probability.
For example, when k 0.25 = , one in four exposed observations had confounder trait C, while three of four unexposed observations had trait C. For k 0.50 = , one in two observations had trait C and there was no association between C and the exposure variables. No confounding was present.
When there is no association between E and Y (odds ratio = 1.0), there cannot be confounding. While a change in the estimate after adjustment could be misclassified as confounding, this scenario is not evaluated in this study.
The underlying probability of Y for unexposed subjects without C was 10%. Sample sizes were 300, 1000 and 3000. For each set of parameters, 1000 datasets were simulated.
Logistic regression models were applied to simulated data and estimated odds ratios and logodds were recorded for the crude model, predicting Y by E (equation 2), and for the adjusted model, which included the exposure E and the possible confounder C as independent variables (equation 3). All simulations and statistical analyses were performed using R version 2.11.1 [10].

Detecting Confounding using Probability Distributions
For each set of simulation parameters, the mean and standard deviation of the logodds for the effect of exposure E were computed for the crude and adjusted models across the 1000 simulated datasets. We also computed the correlation between the crude and adjusted coefficients. Using the empirical estimates for θ and τ in the crude model, for γ and σ in the adjusted model, and for the correlation ρ, we computed the probability of R having a magnitude greater than 10% based on the probability distribution derived above. We then determined the sensitivity for simulation scenarios where confounding * OR is the odds ratio between exposure and outcome. Confounder Effect is the influence of the presence of the confounder variable on the outcome. The confounder halves the probability of the outcome (P(Y)x0.5), has no effect on the outcome (P(Y)x1), or doubles the probability of the outcome (P(Y)x2). † P(Confounder|Exposure) is the association between the possible confounder and exposure variable. The probability of the confounder among exposed subjects is 0.25, 0.50, 0.67, or 0.75. The probability of confounder among unexposed subjects is 0.75, 0.50, 0.33, and 0.25, respectively. ‡ Row displays False Positive Rate as there is no association between possible confounder and outcome, Y, and therefore no confounding. *OR is the odds ratio between exposure and outcome. Confounder Effect is the influence of the presence of the confounder variable on the outcome. The confounder halves the probability of the outcome (P(Y)x0.5), has no effect on the outcome (P(Y)x1), or doubles the probability of the outcome (P(Y)x2). † P(Confounder|Exposure) is the association between the possible confounder and exposure variable. The probability of the confounder among exposed subjects is 0.25, 0.50, 0.67, or 0.75. The probability of confounder among unexposed subjects is 0.75, 0.50, 0.33, and 0.25, respectively. ‡ Row displays False Positive Rate as there is no association between possible confounder and outcome, Y, and therefore no confounding. truly existed and the false positive rates for scenarios where confounding did not exist.

Detecting Confounding using Simulated Data
The relative change in estimate statistic R was computed using the odds ratios for crude and adjusted models applied to each simulated dataset. Confounding was defined as a change from the crude to the adjusted odds ratio with magnitude exceeding / + − 10%. We determined the observed sensitivity and false positive rates for the simulation scenarios. Table 1 displays the derived sensitivity and false positive rates for a probability of exposure of 50%. Table 2 contains corresponding sensitivity and false positive rate estimates from simulated data. Similarly, Tables 3 and 4 display the computed and simulated sensitivity and false positive rates, respectively, for a 5% probability of exposure.

Results
Across odds ratios between Y and E with 50% exposure, stronger associations between E and C corresponded to increased sensitivity.
For example, for an odds ratio of 3.0 and a sample size of 300, when C doubled the probability of disease and the probability of C was 0.67 among exposed subjects, the sensitivity computed using the lognormal distribution (equation 6) was 0.9681. This is compared to a sensitivity of 0.9915 when the probability of C was 0.75 among exposed subjects. For the same two scenarios, 978 and 994 of the 1000 simulated datasets respectively, had relative differences of the crude and adjusted effect sizes that exceeded 10% (sensitivity of 0.978 and 0.994).
Sensitivity was higher when the probability of Y was doubled in the presence of the confounder than when the probability of Y was reduced by half. The sensitivity for detecting confounding was roughly symmetric for inverse associations between E and C (i.e. the probability of C was 0.75 vs. 0.25 among exposed subjects; Table 1, 2).
When C did not impact the probability of disease but was associated with the exposure variable, the false positive rates were substantially larger for reduced sample sizes ( Figure 2). False positive rates were at least 40% for a sample size of 300, at least 10% for a sample size of 1000, and at least 0.4% for a sample size of 3000 (Table 1). In all scenarios, the false positive rate when C was not associated with exposure E *OR is the odds ratio between exposure and outcome. Confounder Effect is the influence of the presence of the confounder variable on the outcome. The confounder halves the probability of the outcome (P(Y)x0.5), has no effect on the outcome (P(Y)x1), or doubles the probability of the outcome (P(Y)x2). † P(Confounder|Exposure) is the association between the possible confounder and exposure variable. The probability of the confounder among exposed subjects is 0.25, 0.50, 0.67, or 0.75. The probability of confounder among unexposed subjects is 0.75, 0.50, 0.33, and 0.25, respectively. ‡ Row displays False Positive Rate as there is no association between possible confounder and outcome, Y, and therefore no confounding. * OR is the odds ratio between exposure and outcome. Confounder Effect is the influence of the presence of the confounder variable on the outcome. The confounder halves the probability of the outcome (P(Y)x0.5), has no effect on the outcome (P(Y)x1), or doubles the probability of the outcome (P(Y)x2). † P(Confounder|Exposure) is the association between the possible confounder and exposure variable. The probability of the confounder among exposed subjects is 0.25, 0.50, 0.67, or 0.75. The probability of confounder among unexposed subjects is 0.75, 0.50, 0.33, and 0.25, respectively. ‡ Row displays False Positive Rate as there is no association between possible confounder and outcome, Y, and therefore no confounding.  was similar to or smaller than the false positive rate for C having no association with the outcome. For large sample sizes, when C was not associated with exposure E, the false positive rates approached zero (Table 1, 2; Figure 2).
The sensitivity and false positive rate of the change in estimate method for detecting confounding derived from equation 6 were very similar to probabilities from simulated data. The largest difference observed had a magnitude of 0.05; however most differences were less than +/-0.015 (Table 1-4). As derivations based on equation 6 are asymptotic in nature, increased sample sizes provided more similar sensitivity and false positive rates between the computed value from the lognormal distribution and the observed values from simulated data.
Similar results were observed for a 5% probability of E with reduced sensitivity and increased false positive rates, particularly for a sample size of 300 (Table 3, 4).

Discussion
Across the examined sample sizes and strengths of associations between confounders, exposures, and outcomes, the change in estimate confounding detection method, applied with a 10% cutoff, had high sensitivity. The method had low false positive rates when there was no association between the exposure and third extraneous variable. False positives were more common when the exposure and third variable were associated but there was no association between the third variable and outcome (Figure 2). The inflated false positive rate may be due to the strong association between the exposure and the third variable resulting in an apparent weak association between the third variable and the outcome. As a result, the third variable contributes similar information about the outcome as the exposure and the regression coefficient for the exposure variable may be affected. When the exposure and third variable are independent they contribute different information about the outcome and the crude and adjusted regression coefficients are similar.
The results presented from this study are for a logistic regression with a single exposure and possible confounder variable. Despite this, the mathematical derivation of the lognormal distribution can be similarly applied to other generalized linear models and we would expect similar results. We have not examined the implications of these results in the context of time to event analysis, Cox proportional hazard models, or semi-and nonparametric analyses. In this study we considered a dichotomous exposure and confounder. Results do not rely on the distribution of either variable and similar results are expected for continuous, nominal, and ordinal predictor variables. We examined the implications of a 10% rule for confounder detection. Similar discussions can be made for a 15% or 20% cutoff which provide lower sensitivity and higher specificity than those observed here. Of note, in these scenarios we would expect the same patterns of increased false positive rates when the outcome is not associated with the third variable regardless of the cutoff percentage. While we have not examined all scenarios, we believe that the presence of additional covariates, confounders, and interactions in regression models would not affect the results of the confounder detection methods examined in this study.
For linear regression, it is straightforward to show that the comparable statistic for the relative difference in parameter estimates can be expressed as the ratio of two correlated normally distributed random variables. Determining the sensitivity and false positive rates in this setting are left for future research. Further research is also required to understand the implications of false positive and false negative confounding results with survival analysis, semi-, and nonparametric regression methods.
While investigators often assume that a change in effect size in a single realization of data indicates confounding [3,11], crude and adjusted effect sizes may differ for other reasons. Noncollapsibility is when effect measures change upon stratification of the covariate [1][2][3]. It can occur when the third variable is affected by the exposure, and, therefore, does not meet the definition of a confounder. The result can  Figure 2 illustrates the false positive rates (1-specificity) of two contrasting scenarios, each with an odds ratio between exposure (E) and outcome (Y) of 2.0. In the first scenario there is no association between Y and third variable (C) but there is an association between E and C. Exposed subjects have twice the probability of trait C as unexposed subjects. (Line CY is missing in Figure 1.) In the second scenario there is no association between E and C but trait C doubles the probability of outcome Y. (Line EC is missing in Figure  1.) All probabilities are derived from the lognormal distribution using empirical estimates from simulated data for parameter values. be large changes between crude and adjusted models that are not due to confounding [3].
The effect measure (i.e., risk ratios, odds ratios, logodds, etc.) compared will affect the distribution of the change in effect size statistic and the results of the comparison [7]. For example, simulated data with 1000 observations, a dichotomous predictor, an odds ratio of 2.0, a probability of confounder of 0.75 among exposed subjects, and a doubling of the probability of disease when the confounder is present had an unadjusted odds ratio, log odds and risk ratio of 1.641, 0.495, and 1.076, respectively. The adjusted model had an odds ratio, log odds and risk ratio of 1.994, 0.696 and 1.101, respectively corresponding to a 22.2%, 40.5% or 2.3% change. As a result, the conclusion of whether confounding exists would depend on the effect measure selected.
It is possible for a variable to satisfy all three criteria of being a confounder (see Figure 1) while not meeting the definition of a confounder. For example, a dichotomous variable may have opposing directions of association with the outcome and exposure variable, thereby canceling the effect. In this circumstance, the variable is associated with both the exposure and the outcome and does not lie in the causal pathway; however it may have no effect on the association between the exposure and outcome [7]. The third variable would not be classified as a confounder by definition or by a change in estimate method, though it does meet all three standards.
It is important to identify and control for confounder variables to better understand the underlying relationships between predictors and outcomes. Unmeasured confounding alone can cause effect sizes of magnitudes commonly observed in epidemiologic studies [12]; however the expansion of models to include additional variables is not without cost. In this study, we found that when the exposure and possible confounder were associated with one another but the possible confounder was not associated with the outcome, false positives where the change in estimate had magnitude of at least 10% were often observed. This rate was magnified for smaller sample sizes. A practical implication of this is when investigators performing randomized clinical trials compare the experimental and control treatment arms at baseline. The groups may differ on some underlying characteristic and it is standard practice to control for variables associated with treatment group [1]. The result is the misclassification of confounding, the selection of models with more predictor terms than needed, with fewer available degrees of freedom, and with inflated confidence intervals for the exposure of interest. Such misclassification is more problematic with small sample sizes where less precision and fewer degrees of freedom are available. For small or moderate sample sizes, in a single realization of data, investigators will have an increased probability of false positives and may include variables in regression models that are, in truth, unnecessary and may alter the association observed; conclusions made, and reduce the precision of estimated effects when compared to population parameters [2,6,7].

Conclusions
Comparing the relative change in estimated odds ratios between crude and adjusted logistic regression models provides a simple, readily available and easily applicable confounder detection method for statistical and epidemiological applications. Such methods have high sensitivity and low false positive rates when there are true associations between the outcome and exposure and the outcome and possible confounder, regardless of sample size or the association between the exposure and confounder. Higher false positive rates are expected when the outcome is not associated with the third variable, particularly for small or moderate sample sizes.
Researchers must be aware of possible false positives when applying change in estimate methods. When evaluating confounding in studies with fewer than 1000 subjects, we recommend evaluating the association between the outcome and possible confounder variable as well as the change in estimated odds ratios. While the examination of the magnitude of effect size change should not be used as the only source of information when selecting possible confounders and building models for analysis, when used in conjuncture with other information, change in estimate methods may provide useful information to identify confounders, build parsimonious models, and better understand associations between outcomes and exposures.