Exploring Safety Culture and Patient Outcomes in Academic Health Science Centers: A Pilot Study of Methodology

Background: Safety culture within healthcare organizations has been implicated, based on other high risk industries, as a contributor to patient outcomes. While the healthcare industry is exploring the possibility of that relationship, little empirical evidence exists to date. The purpose of this pilot study was to examine feasibility and methodology of exploring the relationship between safety culture and patient outcomes in United States (U.S.) academic health science centers (AHSCs). Methods: A retrospective, exploratory design was used. Safety culture was measured by the Hospital Survey on Patient Safety Culture (HSOPS) and patient outcomes were mortality-related patient safety indicators. All variables were collected from secondary data sources. Pearson product moment correlations were used to determine if linear relationships between the variables existed. Results: Analysis of the data suggested a relationship exists between safety culture and mortality-related patient outcomes in AHSCs, but the significance of the analyses cannot be confirmed due to the small sample size. Conclusion: Methodology issues such as sample size, access to safety culture data, mortality as an outcome, and collapsed data scores for the instrument were revealed, while limited access to adequate safety culture data sets rendered the actual analysis unreliable. Policy implications, particularly with regard to access of government owned and collected data should be further explored.


Introduction
Since the release of the 1999 Institute of Medicine's [1] dramatic report that over 98,000 patients die each year in hospitals as a result of medical errors, the healthcare industry has been forced to consider faults in healthcare systems and improve the quality and safety of care delivered. Based on findings from other high-risk industries, the safety culture within healthcare organizations has been implicated as a contributor to patient outcomes. While the healthcare industry is exploring that possibility, little empirical evidence exists to substantiate that relationship. Given the sheer number of people who experience hospitalization, this issue becomes one of protecting the public's health.
Organizations with effective safety cultures have a commitment to safety as a top priority [2]; and these actions and attitudes are evident throughout the organization. Industries and organizations that operate in hazardous and high risk conditions and yet, are consistently successful in maintaining excellent safety performance (such as nuclear power and aviation), ascribe much of their success to their safety culture [3]. Furthermore, many experts in patient safety research believe that there is an association between an organization's culture and its outcomes, and that strengthening a healthcare organization's safety culture will in turn improve patient outcomes [4]. Studies of this kind are expanding and the link between organizational safety culture and outcomes, including patient outcomes, is gaining credibility in the healthcare industry [5]. However, there remains a crucial need to demonstrate the empirical links between organizational safety culture and patient outcomes in healthcare.
The IOM's report, To Err is Human: Building a Safer Health System was the first of its kind to estimate the number of actual lives lost due to medical errors, revealing that medical errors rank as the eighth leading cause of death, killing more Americans than motor vehicle accidents, breast cancer, or AIDS [1]. To Err is Human removed doubt that preventable medical injuries were a significant problem and the report also developed the concept that errors in healthcare are the result of poorly created systems not bad people [6]. The IOM reported that medical errors cost an estimated $37.6 billion annually, including about $17 billion associated with errors that were deemed preventable (1999).
In 2005, six years after To Err is Human, two authors of the original report, Leape and Berwick [6], posed the question is healthcare any safer now? The authors noted that although small local improvements had been made, progress toward reducing medical errors at a national level was slow. The authors attributed this delay to the current culture of healthcare and the reluctance of the industry to create a culture of safety [6].
From 2006-2008, almost one million patient safety incidents occurred among Medicare patients alone with an associated cost of $8.9 billion. Even more alarming, one in ten patients who experienced a patient safety incident during that time died as a result of the event. Unfortunately, the number of patients affected by these events was virtually unchanged from previous years [7] creating an urgent need for more research in this area.
The most widely recognized and commonly accepted definition of safety culture was published by the Health and Safety Commission of Great Britain [8]: It is the product of individual and group values, attitude, competencies and patterns of behavior that determine the commitment to, and the style and proficiency of, an organization's health and safety programs. Organizations with a positive safety culture are characterized by communications founded on mutual trust, by shared perceptions of the importance of safety, and by confidence in the efficacy of preventive measures [8].
Safety culture research became prevalent in the 1980s after a string of major disasters occurred within high-risk industries such as nuclear power, aviation, railways, and off-shore oil drilling. As these disasters were investigated, a poor safety culture was reported as a factor that resolutely affected the tragic outcomes [9]. Efforts to extend this proposition as a factor in the healthcare industry has led to studies seeking to examine the link between safety culture and patient outcomes [5].
In December, 2008, The Research Priority Setting Working Group of the World Health Organization (WHO) published Global priorities for research in patient safety that identified patient safety research needs in "the broad areas where there are substantial knowledge gaps and where it is expected that further knowledge would significantly contribute to improving patient safety and reducing harm" [10]. Twenty leading research topics were recognized along with preliminary research questions associated with each topic. Poor safety culture was named as one of the principal topics and the suggested preliminary research question was "What is the relationship between culture and patient safety outcomes in different settings?" [10].
Numerous studies have been conducted surrounding safety culture, safety culture dimensions, and patient outcomes, yet most studies have only examined independent relationships between dimensions of safety culture and patient outcomes. Only two published studies were found to have specifically examined the relationship of safety culture and patient outcomes.
Singer et al. [11] were the first to narrow the gap with their research on the relationship of safety climate and safety performance in hospitals. The results of this exploratory study supported the researcher's hypothesis that predicted higher levels of safety climate would be associated with higher safety performance (a lower incidence of PSIs). These results were the first published empirical evidence that supports a positive relationship between organizational safety culture and patient outcomes within a healthcare organization [11].
The most recent study specifically exploring safety culture and adverse clinical events examined relationships between all 15 HSOPS variables and rates of in-hospital complications and patient outcomes. Mardon et al. [12], collected retrospective safety culture data from the 2007 HSOPS Comparative Database. A series of multiple regression models revealed that hospitals with higher patient safety culture scores had fewer adverse events, when controlled for hospital bed size, teaching status, and hospital ownership. All of the 15 HSOPS variables were correlated with the PSI composite; the strongest correlation revealed was between handoffs and transitions and the PSI composite. The authors concluded that further research is needed to determine generalizability and to explore associations between specific dimensions of safety culture and adverse patient safety events [12].
As mentioned earlier, one deficit in patient safety research is that different hospital settings have not been studied; hospitals of all sizes have been grouped together and studied as a single population. Among healthcare organizations in the U.S., Academic Health Science Centers (AHSCs) play a crucial leadership role in the movement to provide quality and safe healthcare. Because AHSCs prepare future healthcare providers and serve as a learning lab for healthcare research, these organizations have the opportunity to "be the source of new knowledge on healthcare safety and the transmitter of new skills in safe patient care for the healthcare providers of the future," [13]. Furthermore, leaders in AHSCs recognize the most fundamental and important challenge to improving safety is developing the culture within AHSCs into a culture of safety [13]. Given the potential impact of organizational safety culture on patient outcomes and the opportunity to educate healthcare providers in an improved culture of safety, AHSCs should be studied more closely.
While there is a great deal of evidence to be found in other industry literature, so little evidence for healthcare organizations around such a meaningful and potentially lifesaving topic is alarming. And, previous studies failed to include mortality indicators as patient outcomes. The need for further empirical analyses of the relationship of safety culture and patient outcomes is important, as this knowledge may help prioritize initiatives and resource allocation in an industry seeking efficiency as well as effectiveness. The original predictive study was designed to explore this relationship in AHSCs. The design was based on two similar studies [11,12], but in both cases, the researchers had unlimited access to safety culture data because they developed the safety culture survey tool used and housed the data. Ultimately, insurmountable data collection issues resulted in a pilot study to examine feasibility and methodological issues.

Materials and Methods
This pilot study design was a cross-sectional, correlational design. The independent variable was safety culture as measured by the Hospital Survey on Patient Safety Culture (HSOPS). The dependent variables were patient outcomes in AHSCs, specifically death among surgical patients with serious treatable complications and death in low mortality DRGs. These outcome measures were obtained within the year the respective organization's HSOPS surveys were administered, 2009 or 2010. Covariates were Magnet hospital designation, hospital bed size, and hospital ownership and control. These variables were treated as covariates in the statistical analyses for their potential relationship with safety culture and/or patient outcomes.
The population studied was academic health science centers in the U.S. Inclusion criteria for the sample were (1) AHSCs were members of the University Health System Consortium (UHC), (2) they participated in the HSOPS in 2009 or 2010, and (3) they submitted their data to AHRQ for inclusion in the national database.
Safety culture was measured at the AHSC (organizational) level using the AHRQ 2009 and 2010 HSOPS database. The HSOPS was developed by AHRQ to provide a easily accessible, no cost, valid safety culture tool for hospitals to administer to their staff to assess patient safety culture [14] and to determine areas of priority to target for improvement [15]. Hospitals that participate in the HSOPS can voluntarily submit their data to the database for benchmarking purposes and data are made available to researchers through the AHRQ. The HSOPS measures 12 specific dimensions of patient safety culture at multiple levels, with each dimension measured by 3 or 4 survey questions and has extensive reliability and validity documentation [16][17][18]. The HSOPS dimension scores were combined to create safety culture composite scores for each AHSC. The two patient outcomes were death in low mortality DRGs and death among surgical patients with serious treatable complications. Chosen among a myriad of other outcomes of health care delivery due to their direct measure of mortality, improvement in these indicators would result in lives saved. Death rates are routinely accepted and used by healthcare organizations as a measurement of quality and patient safety [19], and thus chosen as patient outcome variables in this study.
These outcome measures are Patient Safety Indicators (PSIs) developed by the AHRQ. Death in low mortality DRGs is defined as in-hospital deaths per 1,000 patients in DRGs with less than 0.5% mortality rate [20]. The second chosen PSI, death among surgical patients with serious treatable complications, identifies patients whose in-patient death follows the development of a complication. This indicator is defined as deaths per 1,000 patients having developed specified complications of care during hospitalization [20]. Institutional Review Board (IRB) exemption was requested and granted for this study prior to data collection. Because of the complexity of obtaining data use agreements from AHSCs, the logistics of data collection, and the limited number of AHSCs, it was determined that a robust sample size might not be possible. The pilot study was continued to examine the logistical and methodological issues of the design prior to a full scale study.
In order to extract identifiable safety culture data from the AHRQ database, a data use agreement addendum had to be signed from each hospital from which the researcher requested data. Westat Corporation, a research corporation that provides research services to AHRQ and houses the HSOPS database, disclosed that there were a total of 33 UHC AHSCs with HSOPS data in the 2009 database alone, but their identity had to remain anonymous. A data use agreement addendum and informational letter regarding this study were e-mailed directly to all Chief Privacy Officers (CPOs) (or persons in similar positions) at all UHC full member hospitals (AHSCs exclusive of their affiliated hospitals). The data use agreement addendum sent to all 121 CPOs requested permission for the researcher to use identified HSOPS data. Reminder e-mails were sent after two weeks and again one week prior to the deadline. Following the one month deadline, 11 responses were received: one organization did not use the HSOPS; six organizations participated in the HSOPS but did not submit their data to the database; one organization did participate in the HSOPS and submitted their data to the database, but declined to participate in the study; and three signed data use agreements were received.
In an effort to increase the sample size for this study, identical invitations to participate in the study were posted to the UHC Chief Quality Officer (CQO) list serve by a UHC contact. All 108 CQOs at UHC member hospitals participating in this electronic list serve at the time received the e-mail. A total of nine e-mail responses were received by the researcher: one organization participated only in 2011, and six signed data use agreement addendums were returned. The remaining two responses were from CQOs that were uncertain if their HSOPS data were submitted to the national database, neither of which provided signed data use agreement addendums.
After all feasible avenues of obtaining signed data use agreements were exhausted, only nine were collected. A scanned copy of all nine signed data use agreement addendums received were sent to a research associate at Westat Corporation. Of the nine AHSCs, only eight were found to have submitted HSOPS data in either 2009 or 2010 to the national database.
A CD of the eight AHSCs and their identified HSOPS data in SPSS was sent to the researcher. All data were identified by AHSC name and included only organizational level survey data. Percent positive scores were reported and consisted of the number of positive responses to that item within a hospital divided by the total number of responses to that item within a hospital.
Patient outcome data for the eight AHSCs with HSOPS data were made available through the UHC database. Outcome PSIs for each of the eight AHSCs were abstracted from the respective year their safety culture data were collected. Data were then merged to match HSOPS data and converted to SPSS (Statistical Package for the Social Sciences) for analysis.
The final database was screened for missing values and outliers. One AHSC was missing all outcome data because it served only pediatric patients. Mortality related PSI data are not collected for patients younger than 18 years of age; consequently this AHSC was dropped from the sample, reducing it to seven. Values of death in low mortality DRGs were also missing from two AHSCs within the sample. However, all other data sets in these sample units were complete, so they were included in the analysis.
Descriptive statistics were applied to explore characteristics and variations among the sample. The seven AHSCs were located among three of the eight American Hospital Association's (AHA) defined regions: one in the South Atlantic region, two in the Mountain region, and the remaining four in the East North Central region. Overall, there was very little variability among the AHSCs in relation to the confounding variables. Magnet designation and hospital ownership and control were redundant measures since the four AHSCs with Magnet designation were also the only AHSCs in the sample under government ownership and control. This redundancy would have disallowed the researcher from distinguishing which of the covariates actually affected patient outcomes. Therefore, they were eliminated due to the lack of variability.
Since the sample size of seven was not representative of AHSCs in the U.S., the assumptions of linear regression were not met. Therefore, linear regression was not applied; instead, the Pearson correlation was applied as a descriptive statistic to describe the linear relationship between the two patient outcomes and the safety culture dimensions without making inferences to the parent population. Pearson correlations were calculated for all of the dimensions of safety culture with each of the two patient outcomes. Table 1 displays the correlations between the safety culture dimensions and the two outcome measures. A low positive correlation was found between the mean safety culture composite score and death in low mortality DRGs (0.388). This positive correlation reflects that the two variables covary as the safety culture composite score increases (improves), the rates of deaths in low mortality DRGs increase as well. This relationship was unexpected and the authors caution against any inference of these data given the sample size and pilot study design. The strongest correlations existed between death and low mortality DRGs and supervisor/manager expectations (0.739), and handoffs and transitions (0.824).

Discussion
The purposes of a pilot study are to assess the feasibility of the study, the adequacy of the instrumentation, and the problems in data collection and proposed methods [21]. Therefore, the following discussion will address these issues with regard to sampling and data collection, instrumentation and analysis.

Sampling and data collection
The sample size was also limited due, in some cases, to the lack of willingness of AHSCs to share their identified HSOPS data and their lack of awareness regarding their participation and submission of data. The researcher received several e-mailed responses from Chief Privacy Officers and Chief Quality/Patient Safety Officers that refused to share their identified HSOPS data for research purposes, and some were unaware whether their organization previously participated in the HSOPS and/or submitted data to the AHRQ database. These responses were surprising, but may be indicative of why research using measures of safety culture is still in its infancy. This lack of awareness may also point to the under examination of data sets that could provide fodder for patient safety improvement. Data are being reported as required, but organizations may be missing opportunities to put this data to use.

Instrumentation
The HSOPS was developed for healthcare organizations to assess the status of their existing patient safety culture and to determine areas of priority to target for improvement [15]. The survey assesses opinions of hospital staff about patient safety issues, medical error, and event reporting using a 5-point Likert response scale in terms of agreement; and, consists of 42 items to measure 12 dimensions of patient safety culture. Previous psychometric analyses of the HSOPS determined the tool to be moderately reliable and valid at the organization level, for which it was designed [17,18].
There are several notable strengths with the use of HSOPS. First and foremost, the AHRQ's HSOPS Comparative Database is currently the only known national repository for safety culture data, which can be used for research purposes. Therefore, in order to use secondary safety culture data as this study was designed, the HSOPS survey was logistically the only option. There are numerous safety culture research studies that have used the HSOPS, many of which are available on the AHRQ website (http://www.ahrq.gov/). Additionally, the HSOPS can be used to measure both the unit and organizational level, and through the AHRQ Comparative Database, data are available for benchmarking purposes among hospitals [18]. The survey is downloadable from the internet, free of charge, and the AHRQ website offers multiple resources for use in survey administration, data interpretation, and safety culture improvement initiatives. With regard to methodology and instrumentation, this tool has been tested extensively and appears to be a valid and reliable tool for measuring patient safety culture for an organization.

Analysis
Statistical analyses were conducted within SPSS using HSOPS percent positive data provided by AHRQ and patient outcome data provided by UHC. The use of percent positive scores was recommended by AHRQ since data presented as percentages may be more easily recognized by healthcare providers and administrators, and therefore useful for meaningful change. Furthermore, percentages are commonly used to set as goals for quality and safety improvement projects, thus a percentage would be familiar to healthcare staff. However, the use of percent positive composite scores for data analyses means a large portion of the data are not included in statistical analyses. The advantage of using percent positive scores over raw data for practice translation appears to be logical, yet the use of raw data over previously collapsed data lends greater depth of analysis and exploration. With regard to this patient safety survey, collapsing to percent positive ignores all neutral and negative responses.
Although the sample size was small, the analysis was continued for exploratory purposes of the pilot study. The relationship of safety culture and patient outcomes was first explored using Pearson product moment correlations to determine if linear relationships between the variables existed. The proposed analyses were appropriate to answer the research question, but the small sample size limited the ability to determine if correlation directions and magnitude were meaningful. Obviously, inability to access patient safety survey data severely limited the study.

Implications and Recommendations for Future Research
This pilot study was a foundational step in exploring the relationship between safety culture and patient outcomes in U.S. The necessary inclusion criteria limited the accessible sample to a maximum size of 33 AHSCs. A number of efforts were made to increase the number of participating AHSCs, but access to HSOPS data was extremely limited. Currently, the only way to obtain identifiable HSOPS data from the AHRQ database is to obtain signed data use agreement addendums directly from hospitals, but the names of those hospitals cannot be disclosed. This process forces the researcher to blindly request a signed data use agreement addendum from all hospitals within a given population-a logistically ineffective method. Data use agreements that expand the availability of data for research should be implemented, particularly when the research data is collected and housed by government funded agencies. AHSCs. The researchers found that some patient safety officers were not aware if their organizations participated in a safety culture survey. At the very least, organizations need to examine their data closely and discuss these considerations with their staff to best utilize the data. An even better option would be considering policy to improve data access to researchers. As we strive to build evidence to make decisions that are better for our patients, patient safety culture cannot be excluded.
While admitting the difficulty in obtaining an adequate organizational sample size for this type of study, the authors recommend AHSCs conduct their own case studies to examine specific HSOPS safety culture data and patient outcome data until larger data sets are more accessible for research purposes. An annual case study of this kind could provide useful trending information regarding the AHSC's progress towards improving patient safety over time, including an evaluation of quality and safety improvement activities. Given the flexibility case studies provide for single organizations and the ability to gather qualitative data that is currently absent from the literature, organizations would have a valuable data source for a more comprehensive understanding of patient safety culture [22].
Additionally, researchers and healthcare leaders should be mindful of the strengths and weaknesses of using percent positive safety culture data in statistical analyses rather than including the entire data set. Although the use of percent positive data may seem logical to demonstrate its practical use, this method ignores a significant portion of the data which may be needed for a more rigorous statistical analysis. The researcher should make an informed decision between data that is statistically significant versus data that is practically significant.
Also, this pilot study warrants the exploration of outcome measures that occur more frequently and therefore, may yield data more amenable to statistical analyses. Often researchers gravitate toward outcomes that seem to have the greatest impact, such as mortality. However, patient outcome measures which practitioners can more easily impact and those which can be more easily controlled, such as nosocomial infection rates, should be included in future studies. While the results of this pilot study cannot be used as reliable evidence, the data did suggest that perhaps mortality may have a unique relationship with safety culture.
According to a Health Grades report, the United States loses more American lives to patient safety incidents every six months than it did in the entire Vietnam War [23]. After more than a decade, the public's health is still compromised because significant numbers of lives are lost through issues of patient safety. While it is possible that individual organizations have improved, overall patient safety improvement and its connection to lives saved, is not demonstrated in current literature. Organizational research in patient safety culture and its connection to patient outcomes will not be possible until accurate and adequate data is accessible. More exploratory studies, using the described methods discovered in this pilot study, are needed to better understand the relationships that exist and the impact of changes in patient safety culture on patient outcomes. Research of this kind is imperative to provide reliable, empirical evidence for healthcare providers and leaders to improve patient safety.