Clinician Acceptance of Computerized Alerts for Public Health Surveillance

Electronic health records (EHR) are being rapidly adopted in hospital and ambulatory care settings and there is some evidence that they positively influence the quality of clinical decisions, avert medication errors, and improve effective delivery of preventive care and patient outcomes [1–5]. With exponential growth in medical knowledge and constantly shifting guidelines and recommendations, physicians are increasingly using EHRs and computer decision support systems and clinical decision support (CDS) alerts as they work in a complex and shifting information landscape [6]. Evaluations to assess whether CDS alerts improve clinical care are limited but are generally positive and demonstrate that CDS alerts can improve the quality of care [7–11]. However, multiple barriers to integrating EHRs and CDS alerts into clinical practice exist. These include cost, complexity, [5,12– 14] and a lack of understanding about how they are used and perceived by clinicians [15,16].


Introduction
Electronic health records (EHR) are being rapidly adopted in hospital and ambulatory care settings and there is some evidence that they positively influence the quality of clinical decisions, avert medication errors, and improve effective delivery of preventive care and patient outcomes [1][2][3][4][5]. With exponential growth in medical knowledge and constantly shifting guidelines and recommendations, physicians are increasingly using EHRs and computer decision support systems and clinical decision support (CDS) alerts as they work in a complex and shifting information landscape [6]. Evaluations to assess whether CDS alerts improve clinical care are limited but are generally positive and demonstrate that CDS alerts can improve the quality of care [7][8][9][10][11]. However, multiple barriers to integrating EHRs and CDS alerts into clinical practice exist. These include cost, complexity, [5,[12][13][14] and a lack of understanding about how they are used and perceived by clinicians [15,16].
More recently, CDS alerts were also used to communicate public health messages about emerging health issues [17,18]; however, reports of their use for public health surveillance are lacking, and clinician acceptance has never been evaluated in this context. As part of a comprehensive evaluation of a pilot respiratory virus surveillance system implemented in New York City immediately after onset of the 2009 pandemic influenza A (H1N1) (pH1N1) outbreak, primary care providers (PCPs) and nursing staff at participating clinics were surveyed to evaluate acceptability of CDS alerts designed to encourage diagnostic testing for public health surveillance.

Setting
The pilot respiratory virus surveillance system was implemented on May 26, 2009, at all nine Institute for Family Health (IFH) community health centers in Manhattan and the Bronx. The primary objective was to determine the epidemiology of respiratory virus strains circulating in the community by linking clinical, epidemiologic and diagnostic data. The system capitalized on a long-standing association between the New York City Department of Health and Mental Hygiene (DOHMH) and IFH. IFH has used Epic Ambulatory EHRs (Epic) since 2002 and routinely incorporates CDS alerts into clinical care [19]. CDS alerts can be designed to be directed to PCPs, nursing, or other staff. IFH clinics serve adult and pediatric patients and had approximately 2000 patient visits/week during the first 5 weeks of operation of the pilot surveillance system, May 26-June 30, 2009. IFH and DOHMH had previously evaluated and validated use of routine EHR data for syndromic surveillance for influenza-like illness (ILI) [20] and had piloted a program to incorporate CDS alerts to notify clinicians of local disease outbreaks [17].

CDS alert
Although diagnostic testing is sometimes used in clinical evaluation for ILI, it was not commonly used at IFH. During the 2009 pH1N1 outbreak, DOHMH recommended empiric treatment with antiviral medications rather than waiting for confirmatory diagnostic testing for mild ILI cases not requiring hospitalization [21]. However, to support surveillance, CDS alerts were uniformly integrated into the EHR at the nine IFH sites to encourage clinicians to perform a nasopharyngeal swab for ambulatory patients presenting with ILI at the point of care; clinical staff were instructed that diagnostic testing of patients with ILI would support public health surveillance for respiratory viruses. An e-mail message was sent to all clinicians at the 9 IFH clinics to inform them of the pilot surveillance system. Nursing staff were trained on how and why to perform nasopharyngeal swabs through an Internetbased instructional module and e-mail instructions.
IFH senior clinical leadership developed the CDS alert triggers which were programmed into the EHR by IFH staff and used at all clinic sites. Although multiple triggers were used for the CDS alert, the screen that appeared was always identical, independent of the inciting trigger and stated, "H1N1 Alert-Diagnosis consistent with influenza. If cough, patient needs isolation and/or face mask. Click here for CDC [Centers for Disease Control and Prevention] recommendations and Click here for NP [nasopharyngeal] swab instructions." Criteria for triggering a CDS alert during this surveillance period were broader than the ILI case definition and included: (1) a patient presenting with a measured temperature of ≥99.9ºF; or (2) a patient presenting with a constitutional complaint that included fever, chills, perspiration, or body aches and a respiratory complaint that included cough, cold symptoms, sinus problems, or sore throat; or (3) a diagnosis of unspecified viral infection, bronchitis, influenza, otitis media, or upper-respiratory tract infection (i.e., International Classification of Diseases, 9 th ed. [ICD-9] codes: 079.99, 466.0, 487.1, 382.00, or 465.9) [22]. An alert could be triggered multiple times during an encounter and ordering an nasopharyngeal swab did not prevent subsequent alerts being triggered during other parts of the clinical encounter. This CDS alert was designed specifically for this public health surveillance system and was evaluated as part of a comprehensive surveillance system evaluation. A discussion of this evaluation is beyond the scope of this paper.

ILI case definition
DOHMH's Primary Care Information Project (PCIP) generated weekly reports of ILI syndrome data, which were sent to IFH. ILI data were abstracted by using SAS ® version 9.2 (SAS Institute, Cary, North Carolina), [23] and either of the following case definitions: (1) a patient presenting with fever (i.e., reported subjective complaint or measured temperature >99.9ºF) and cough, or (2) a patient presenting with fever and a respiratory diagnosis (i.e., ICD-9 codes: 079.99, 466.0, 487.1, 382.00, or 465.9).

Diagnostic testing
Two diagnostic tests were performed. First, nasopharyngeal swabs obtained at IFH clinics were sent to a commercial laboratory where an enzyme immunoassay for influenza A and B (BiNaxNow ® ; Inverness Medical Innovations, Inc., Princeton, New Jersey) was performed. Diagnostic test results were transmitted electronically to IFH's EHR system and available to clinicians. From the commercial laboratory, aliquots were sent for testing by multiplex polymerase chain reaction (PCR) assay at a university laboratory [24]. De-identified results from the university laboratory were only sent to DOHMH for public health surveillance, because the multiplex assay was not commercially licensed or Food and Drug Administration-approved and results were not intended to guide clinical management of patients. The Institutional Review Board at IFH determined that routine informed consent procedures were sufficient because BiNaxNow was commercially available for routine clinical practice, minimal risk existed for patients, only discard aliquots were used for the multiplex assay, results were deidentified, and testing was provided at no cost to patients or insurers.

Clinician surveys
Two Internet-based surveys were developed for IFH clinicians using SurveyMonkey ™ (Palo Alto, California), one directed to PCPs, including family medicine attending physicians, family medicine residents, and nurse practitioners, and the other directed to nursing staff, including medical assistants. Key PCP and nursing staff survey questions are included in Appendices A and B, respectively. The PCP survey comprised 12 questions including five multiple choice, 4 using Likert 5-item scales, two yes/no, and one open-ended questions. The nursing staff survey comprised 10 questions including four multiplechoice, three using Likert 5-item scales, two yes/no, and one openended questions. In October 2009, 5 months after implementation of the surveillance system, an IFH staff member sent the surveys to all clinical staff working at the IFH clinics participating in the pilot surveillance system. Potential participants received three e-mail notices during a 23-day survey period. No financial or other incentives were provided. All survey responses were confidential.

Statistical analysis
Significance testing was conducted for comparison of PCPs and nursing staff responses using Fisher's exact test (Table 1). Significant differences were determined by 95% confidence intervals.

Results
Surveys were sent to 45 PCPs and 47 nursing staff with a response rate of 53% and 55%, respectively. Among PCPs surveyed, 62% (n=15) were family medicine attending physicians, 21% (n=5) nurse practitioners, and 17% (n=4) family medicine residents. Among nursing staff surveyed, 62% (n=16) were nurses and 38% (n=10) medical assistants. The majority of respondents were aware of the collaboration with DOHMH to "conduct surveillance for respiratory viruses in the community" (92% of PCPs and 89% of nursing staff). Fifty-eight percent of PCPs and 81% of nursing staff were aware that diagnostic testing was free to patients.
The majority of respondents (71% of PCPs and 89% of nursing staff) reported that using EHRs facilitated workflow and patient care, in addition, 13% of PCPs and 8% of nursing staff reported that although the EHR initially slowed workflow or hindered patient care that it currently facilitates workflow and/or patient care (Table 1; PCP and nursing staff survey questions are included in Appendices A and B, respectively). None of the nursing staff and only one PCP reported that EHRs slow workflow or hinder patient care; there was no statistical difference between PCPs and nursing staff (p=0.4565).

CDS alerts' impact on workflow and self-reported adherence
When asked specifically about CDS alerts, 65% of PCPs and 46% of nursing staff reported that they agreed or strongly agreed with the statement that CDS alerts "facilitate workflow and/or patient care," whereas 35% of nursing staff and none of the PCPs disagreed or strongly disagreed (p=0.002) ( Table 1). When prompted by a CDS alert, the majority of nursing staff (62%) but only 25% of PCPs reported that they performed or ordered an nasopharyngeal swab all or most of the time, and 35% of nursing staff and 75% of PCPs said they did this only some of the time, rarely, or never (p=0.007) ( Table 1).

Impact of perceived clinical utility on CDS alert adherence
None of the PCPs and 42% of nursing staff agreed or strongly agreed that, "The rapid flu test [BiNaxNow] is useful for guiding clinical decisions"; 79% of PCPs and 12% of nursing staff disagreed or strongly disagreed with this statement (p<0.001) ( Table 1).
Among PCPs, the most common reasons (80%) cited for not ordering an nasopharyngeal swab included a reason related to lack of clinical utility (e.g., disagreeing with the indication for diagnostic testing, disagreeing that nasopharyngeal swabs were the best choice of tests, or that the results would not change their treatment plan) ( Table  2). Although 24% of nursing staff reported that they always performed the nasopharyngeal swab, the most common reasons reported for not ordering or performing it were because they were instructed not to (32%) or it had not been ordered by a PCP (24%). Approximately 5% of nursing staff noted "not enough time" as the most common reason for not performing an nasopharyngeal swab.
A representative sample of clinician comments regarding the surveillance system is listed in Table 3. Comments from PCPs and nursing staff frequently described the CDS alerts as having a "low threshold," not specific for influenza, and repetitive because they could be triggered multiple times in a single clinical encounter.

Discussion
This paper describes the results of a survey of PCPs and nursing staff perceptions and self-reported adherence to CDS alerts prompting diagnostic testing for influenza as part of a pilot surveillance system for respiratory viruses temporarily mounted in New York City during  [25], and none have evaluated CDS alerts to collect public health surveillance data. Integration of CDS alerts into this surveillance system resulted in a mixed reception but provided insight into the different ways in which PCPs and nursing staff perceive and respond to CDS alerts.
The majority of PCPs and nursing staff at IFH were strongly supportive of EHRs and reported that EHRs facilitate workflow and patient care, but support waned when asked about CDS alerts specifically. PCPs and nursing staff differed in their perceptions of the utility of diagnostic testing (e.g., BiNaxNow) in guiding clinical decisions for patients presenting with ILI. None of the PCPs thought this diagnostic testing was clinically useful for decision making, contrasting with approximately half of nursing staff. Consistent with this finding, among PCPs, the most common reasons reported as to why a CDS alert was not followed were related to clinical utility, whereas the majority of nursing staff did not perform a diagnostic test because it was either not ordered by a PCP or they were instructed not to do so by a PCP. This finding might be because PCPs are more likely than nursing staff to review diagnostic test results and make empiric treatment decisions for patients with ILI. Therefore, PCPs are more likely to recognize whether a diagnostic test is clinically useful and use this knowledge to either adhere to or ignore the relevant CDS alert. This also demonstrates that, ideally, a diagnostic test integrated into clinical workflows for public health surveillance would also be clinically useful for the individual patient.
Reported lack of adherence to CDS alerts among PCPs due to clinical utility emphasizes the importance but also the limitations of education on the rationale behind a CDS alert before and throughout its implementation, especially when the CDS alert is related to public health surveillance objectives that might be unfamiliar to clinicians. Despite the fact that the majority of PCPs and nursing staff were aware of the collaboration with DOHMH to conduct surveillance for respiratory viruses, the objectives of diagnostic testing for surveillance might not have been clear. The CDS alert, which did not explicitly specify to conduct an nasopharyngeal swab for public health surveillance, might have been superseded by the low clinical utility of the diagnostic test [26,27]. Clinical staff also might have been confused by the conflicting public health messages at the spring peak of the 2009 pH1N1 outbreak, which encouraged empiric treatment of mild ILI without diagnostic testing [21]. Senior IFH staff reported that when syndromic data from DOHMH were shared with clinicians on a regular basis, they saw an increase in submission of nasopharyngeal swabs. This response reinforces that sustained communication between public health surveillance staff and clinicians is needed. Public health staff should communicate findings with clinicians in a timely manner to promote increased understanding of clinicians' role and of the implications of their participation for public health surveillance.
In principle, the majority of PCPs were strongly supportive of the statement "CDS alerts facilitate patient care/workflow," whereas nursing staff were not. However, in practice, PCPs were less likely to report adherence to CDS alerts in this surveillance system than nursing staff. This difference highlights a distinction in how PCPs and nursing staff perceive and respond to CDS alerts in this context. As stated previously, nursing staff were more likely to adhere to a CDS alert unless otherwise directed by PCPs, whereas PCPs were more likely to use clinical judgment to determine whether to adhere to alerts which is consistent with prior studies on pharmacy-related alerts [28,29]. The distinctions between nursing staff and PCP adherence demonstrate that CDS alerts for surveillance purposes might be more effective if they targeted nursing staff, especially if orders were in place for them to perform diagnostic testing for patients meeting the ILI case definition on initial evaluation without requiring PCP approval. Although clinicians were initially concerned regarding the impact of CDS alerts on workflow, none of the PCPs and only one nursing staff respondent reported that "not enough time" was the most common reason they did not adhere to a CDS alert. This indicates that impact on workflow was less important than perceived utility among clinicians in not adhering to CDS alerts.

Primary care providers
• "Sometimes the BPA* fires even when the patient obviously doesn't qualify." • "The BPA comes up for anyone who has a fever and a sore throat. Some people do not have muscle aches or other concerns for the flu and have a low pretest probability of a positive test. It would be a waste of resources to swab the unlikely people . . . ." • "The threshold for firing the influenza BPA is very low." • "We need to get the results back in a more timely fashion in order for them to be clinically useful, especially for treatment decisions." • "BPAs are repetitive." • "Tweak the indications so that the BPA fires appropriately. Make sure that doing the test causes the BPA to go away!" • "[A]uthorizing nursing [staff] to swab anyone with given criteria could facilitate larger enrollment if there is no limitation on the number of tests."

Nursing staff
• "Patient symptoms are not indicative to ordering test such as no fever or sore throat." • "Fever seems to prompt the BPA for flu testing. Patients will present with fever due to other reasons -sometimes with urinary problems -and an influenza swab is not needed. We can use our judgment, as nurses, to consult and communicate with doctors when BPAs are not needed. If it is a strict rule, then there will be some problems." • "More help -staff." BPA = best-practice alert (clinical decision support alert).
* Comments listed are representative of opinions offered by at least two respondents. One of the surveillance system objectives was to capture as many ILI cases as possible therefore CDS alerts were designed to be broad and relatively nonspecific. However clinician comments reveal this was frustrating to them and, consistent with prior studies, might discourage adherence, cause CDS alert fatigue, and result in their being ignored [15,17,[29][30][31]. The low threshold for triggering a CDS alert, that it could be triggered multiple times in a single clinical encounter, the low sensitivity of the initial diagnostic test, and insufficient understanding of the role of diagnostic testing in public health surveillance, all might have contributed to suboptimal adherence to CDS alerts. Of note, although 100% adherence is not required for purposes of public health surveillance, lack of specificity in how CDS alerts are triggered and targeted is known to lead to alert fatigue. Furthermore, nonspecific CDS alerts might result in unintended bias by skewing diagnostic testing to patients perceived to be more convenient (e.g., adults rather than children) and therefore limit generalizability of surveillance findings. Nonspecific CDS alerts triggered multiple times for different providers in the same clinical encounter also might lead to confusion regarding the roles and responsibilities of PCPs and nursing staff and became another barrier to adherence [31]. To maintain adherence, avoid alert fatigue and confusion regarding roles, we recommend that CDS alerts in similar public health surveillance systems be designed with clear goals regarding roles and responsibilities of clinicians, and which patients to be tested. For example, depending on the volume of patients at a clinic, a CDS alert might be designed to trigger for only nursing staff on every fifth patient, the first 20 patients per week presenting with ILI, or selectively test patient populations on the basis of age, comorbidities, or other criteria. Lastly, the findings of this survey and prior reports demonstrate that approaching the design and implementation of CDS alerts for public health surveillance with a user-centered design framework [11,12,[31][32][33][34][35][36] and the following questions in mind might be useful: 1. Who is the targeted clinician (i.e., PCPs, nursing staff or pharmacy staff))? Who is the targeted patient (i.e., adults, children, older persons, patients with a specific condition)?
2. When during a clinical encounter should a CDS alert be triggered to be most effective and least disruptive to workflow? Is a particular alert related to season, time of year, type of outbreak?
3. Where is this CDS alert geographically relevant (e.g., when does it relate to notification of an outbreak)?
4. What triggers a CDS alert? What is the CDS alert prompting (i.e., a preventive measure, screening, diagnostic test)? What is the desired response or outcome? What is the clinician's role and responsibility?
5. Why do clinicians respond to a particular CDS alert the way they do (i.e., ignore or adhere)?
6. How does the alert impact clinical care? How does it impact workflow? How will the information be used?
These findings are subject to certain limitations. First, this study is limited by its sample size. Second, a potential nonresponse bias exists; those who are strongly supportive or strongly critical of EHR, CDS alerts, or the surveillance system might have been more motivated to respond to the survey. Third, the low sensitivity of the diagnostic test might have affected acceptability of the surveillance system as a whole and CDS alerts specifically. A potential also exists for clinician confusion due to DOHMH recommendations during the 2009 pH1N1 outbreak for empiric treatment rather than diagnostic testing of mild ILI cases [21]. Moreover, because of the CDS alert design self-reported adherence patterns to CDS alerts could not be corroborated with actual use and it was not possible to assess whether it was the PCP or nursing staff that were adhering to the alert. Nor was clinicians' understanding of surveillance objectives and how this might have affected adherence to CDS alerts evaluated. Finally, IFH clinics are experienced using EHR and CDS alerts therefore these findings might not be generalizable to other clinics or settings in which EHRs have been introduced more recently.

Conclusion
This study's findings are relevant and useful to the design and implementation of CDS alerts to augment public health surveillance. While there are an increasing number of studies on the role of and clinician perception of CDS alerts for preventing medical errors or guiding clinical decision-making, there are no published reports related to clinician perceptions of CDS alerts in the context of public health surveillance. The major lessons learned relate to the differing views toward and adherence to CDS alerts among PCPs and nursing staff and that a user-centered framework should be considered in the design and implementation of CDS alerts in public health surveillance. Given the patterns of acceptability demonstrated in this study, future designers of surveillance systems that rely on diagnostic testing might consider not only using diagnostic tests with greater clinical utility but also targeting CDS alerts at the initial encounter between patient and nursing staff rather than PCPs, increasing training and education before implementation, especially regarding surveillance objectives that might be unfamiliar to clinicians, and periodically evaluating adherence to and acceptability of CDS alerts.