Received Date: June 02, 2016; Accepted Date: June 28, 2016; Published Date: June 30, 2016
Citation: Fernanda CG Polubriaginof , Paulo G Pastore, Drohan B, Griffin P, Kevin S Hughes (2016) Comparing Patient-Reported Medical Problems with the Electronic Health Record Problem List. Gen Med (Los Angel) 4:258. doi:10.4172/2327-5146.1000258
Copyright: © 2016 Fernanda Polubriaginof CG, et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Visit for more related articles at General Medicine: Open Access
Electronic health record (EHR) problem lists are extremely important, and yet they are often incomplete and out of date. We compared the EHR problem list to a self-reported problem list obtained via a tablet-administered questionnaire to identify potential synergy. We conducted a retrospective review comparing the EHR problem list to the patients’ self-reported problem list during the year of 2011. To confirm the accuracy of patient self-reports, we also analyzed medication lists, and laboratory results for two selected conditions, hypercholesterolemia and diabetes mellitus. Overall, 1472 patients at the Massachusetts General Hospital (MGH) used the tablet questionnaire. Of these, 843 (57.27%) had no problem reported in the EHR and some problem reported on the tablet (“Tablet Only”); 42 patients (2.85%) had no problem reported on the tablet or in the EHR (“None”); 17 patients (1.15%) had some problem reported in the EHR and no problem reported on the tablet (“LMR Only”); and 570 patients (38.72%) had some problem reported both on the tablet and in the LMR (“Both”). Overall, we studied 59 conditions, of which twelve had enough patients to run Chi Square analysis. Of the 12 conditions analyzed, 10 were significant, and 9 out of 10 conditions favored the tablet-administered questionnaire. Medication lists and laboratory results were reviewed to confirm the presence of the selected conditions. We reviewed the EHR problem list one year after the study was initiated to update the conditions under study. The additional data corroborated 107 additional self-reported conditions in 97 patients. In summary, a self-administered tablet questionnaire is an acceptable method for collecting the medical history. When combined with the EHR problem list, self-reported medical history is optimal for obtaining the most accurate problem list possible.
Electronic health record; Healthcare; Patients
Many had hoped that the implementation of the electronic health record (EHR) would improve quality of care and enhance safety by making data for secondary purposes more accessible to healthcare providers . However, the preponderance of free text, coupled with the high likelihood of redundant, contradictory, incomplete, or missing data, has been a barrier to achieving this goal. In particular, the EHR problem list suffers from being both incomplete and unstructured . The issue is that creating and maintaining a comprehensive and accurate problem list is critical and yet difficult to execute.
Patients can be engaged to assist in this activity by completing selfadministered questionnaires. While paper-based questionnaires have been the norm, these tools require transcription of that information into the EHR, costing time or preventing data entry. An alternative is to use patient-entered questionnaires, delivered via the Web or on a PC tablet, to electronically collect structured data, thereby eliminating the need for data entry by the professional staff.
Armed with the knowledge that problem lists are often incomplete,  we studied the degree of missing data in the problem list in the EHR (Longitudinal Medical Record, or LMR) at MGH, and examined the accuracy of a self-reported problem list from a tablet questionnaire (hereafter referred to as the Tablet). Our goal was to compare the accuracy and completeness of patient-entered versus the EHR problem list.
With Massachusetts General Hospital - Partners, Boston MA Institutional Review Board approval and in compliance with all appropriate regulations, we conducted a retrospective chart and database review. Patients seen at the Avon Breast Evaluation Center and a pilot sample of those seen at the Pre-Anesthesia Testing Area (PATA) completed a Tablet questionnaire via the Hughes RiskApps, LLC software [4,5]. Patients were asked to fill out questions regarding their medical history, family history and risk factors. A retrospective analysis was carried out on patients seen between January 1, 2011 and December 31, 2011, comparing the problem list available in the EHR (and other corroborating data from the EHR) to the patients’ selfreported problem list generated using the Tablet questionnaire. The data was not anonymized and de-identified prior to analysis due to need to join multiple patient records.
Methods regarding accuracy of patient-entered data compared to EHR problem list
To produce consistent comparisons between the EHR and the Tablet problem lists, a specific and generic SNOMED-CT (Systematized Nomenclature of Medicine - Clinical Terms) code was manually identified for all problems whether free text or structured data. The code described the specific problem listed in the EHR (usually free text) or the Tablet questionnaire. The generic code was derived in accordance with the SNOMED-CT hierarchy to identify a problem type as opposed to a specific problem, since the patient’s reports are generally more generic than the physician’s problem list. For example, if the patient selected “Disorder of liver” in the Tablet and the EHR had the diagnosis of “Cirrhosis of liver,” the generic SNOMED-CT code would help determine whether they were likely to be describing the same disorder, but with a different level of complexity (Table 1).
|Specific SNOMED CT code||Specific text||Generic SNOMED CT code||Generic text|
|166603001||Liver function tests abnormal||235856003||Disorder of liver|
|3738000||Viral hepatitis||235856003||Disorder of liver|
|50711007||Viral hepatitis C||235856003||Disorder of liver|
|40468003||Viral hepatitis, type A||235856003||Disorder of liver|
|442191002||Steatohepatitis||235856003||Disorder of liver|
|154350000||Other hepatitis||235856003||Disorder of liver|
|197321007||Steatosis of liver||235856003||Disorder of liver|
|66071002||Type B viral hepatitis||235856003||Disorder of liver|
|19943007||Cirrhosis of liver||235856003||Disorder of liver|
Table 1: Using specific and generic SNOMED CT codes.
The analysis was limited to problems included in the Tablet to minimize reporting bias. Table 2 contains the list of problems included in the tablet questionnaire.
|Cardiac revascularization with bypass anastomosis|
|Chronic obstructive lung disease|
|Cirrhosis of liver|
|Conduction disorder of the heart|
|Congenital heart disease|
|Factor V Leiden mutation|
|Gastroesophageal reflux disease|
|Hypertensive disorder, systemic arterial|
|Implantation of heart valve|
|Intraductal carcinoma in situ of breast|
|Malignant neoplasm of liver|
|Malignant neoplasm of uterus|
|Malignant tumor of breast|
|Malignant tumor of cervix|
|Malignant tumor of kidney|
|Malignant tumor of large intestine|
|Malignant tumor of lung|
|Malignant tumor of ovary|
|Malignant tumor of pancreas|
|Malignant tumor of prostate|
|Malignant tumor of stomach|
|Malignant tumor of thyroid gland|
|Neoplasm of brain|
|Non-Hodgkin lymphoma - category|
|Obstructive sleep apnea syndrome|
|Placement of stent in coronary artery|
|Right heart failure|
|Steatosis of liver|
|Thrombosis of blood vessel|
|Transient ischemic attack|
|Type B viral hepatitis|
|Viral hepatitis C|
|Von Willebrand disorder|
Table 2: All studied problems.
Methods regarding completeness
The EHR problem list was revisited in December 2012 to determine if more data had been entered for the same cohort of patients during the intervening months that might improve the accuracy of the EHR problem list and/or corroborate the patient-entered data.
Accuracy compared to laboratory results and compared to medications list
To confirm the accuracy of patient-reported and EHR-recorded problem lists, relevant medications and laboratory results were examined for hypercholesterolemia and diabetes mellitus. For the purposes of this research, the assumption was made that the degree of corroboration determined for the selected conditions could be extrapolated to other conditions.
To make judgments about the accuracy of the sample to see if it reflected the characteristics of the patient population from which it was drawn, we ran a Chi Square test, using the contingency tables found at Vassar Stats website, http://www.vassarstats.net/tab2x2.html accessed on August 1, 2013.
This analysis determined whether the observed values for the tablet questionnaire and the EHR data cells deviated significantly from the corresponding expected values for those cells. A large discrepancy between the observed and expected values would generate a large X2 statistic, suggesting a significant difference between observed and expected values. Along with this statistic, an alpha level of significance was determined a priori and set at 0.05 and the phi coefficient was computed. Values of p<0.05 were considered significant.
The study population, with a final sample size of N=1472, is defined as those patients who answered the Tablet questionnaire. Of this sample, 828 (56.25%) were female and 644 (43.75%) male.
Results regarding accuracy of patient-entered data compared to EHR problem list
Of the total sample, 843 patients (57.27%) had no problem reported in the EHR with some problem reported on the Tablet (“Tablet Only”); 42 patients (2.85%) had no problem reported on the Tablet or in the EHR (“None”); 17 patients (1.15%) had some problem reported in the EHR and no problem reported on the Tablet (“EHR Only”); and 570 patients (38.72%) had some problem reported both on the Tablet and in the EHR (“Both”).
We then studied in more detail patients that had some problem reported in the EHR (“EHR Only” and “Both” - 587 patients). Among them, 70 patients (12.28%) had a perfect problem list match, which means that all problems present in the EHR problem list were present on the Tablet and vice-versa; 34 patients (5.96%) had no match (33 patients had two completely different problem lists and one patient had all problems missed by the EHR); and 466 patients (81.75%) had some problems that matched and some that were missing from one of the sources. In the last group, for the problems that didn’t match, 159 patients (34.12%) had some problems that were missed by the EHR, and different problems that were missed by the Tablet (both lost); 252 patients (54.08%) had problems that were missed only by the EHR (pure EHR loss); and 55 patients (11.80%) had problems that were missed only by the Tablet (pure tablet loss) (Figure 1).
In this group of patients we have also compared both problems lists by disease (Figure 2).
Overall, we studied fifty-nine conditions (Tables 2 and 3), of which twelve had enough patients to run Chi Square analysis. Of the 12 analyzed conditions, 10 were significant, with a p value of<0.05 (Table 4). Of those with significant differences, 9 out of 10 conditions favored the Tablet questionnaire. The only condition with significant difference between the groups that favored the EHR was hypothyroidism. For this condition, the Tablet found 68 patients and the EHR found 74.
|SNOMED CT code||Text||Tablet Only||EHR Only||Tablet And EHR (Both)||Total Patients With Disease|
|90205004||Cardiac revascularization with bypass anastomosis||3||0||20||23|
|13645005||Chronic obstructive lung disease||11||9||14||34|
|19943007||Cirrhosis of liver||2||0||3||5|
|44808001||Conduction disorder of the heart||40||1||41||82|
|13213009||Congenital heart disease||3||0||6||9|
|307091009||Factor V Leiden mutation||3||0||7||10|
|235595009||Gastroesophageal reflux disease||81||64||50||195|
|38341003||Hypertensive disorder, systemic arterial||49||23||246||318|
|112815000||Implantation of heart valve||3||0||4||7|
|109889007||Intraductal carcinoma in situ of breast||1||0||5||6|
|93870000||Malignant neoplasm of liver||1||1||4||6|
|371973000||Malignant neoplasm of uterus||2||2||4||8|
|254837009||Malignant tumor of breast||13||8||65||86|
|363354003||Malignant tumor of cervix||2||2||2||6|
|363518003||Malignant tumor of kidney||9||0||7||16|
|363510005||Malignant tumor of large intestine||6||2||14||22|
|363358000||Malignant tumor of lung||7||0||7||14|
|363443007||Malignant tumor of ovary||2||0||1||3|
|363418001||Malignant tumor of pancreas||2||1||5||8|
|399068003||Malignant tumor of prostate||11||2||13||26|
|363349007||Malignant tumor of stomach||2||0||0||2|
|363478007||Malignant tumor of thyroid gland||3||4||10||17|
|126952004||Neoplasm of brain||2||7||6||15|
|128929007||Non-Hodgkin lymphoma - category||2||0||5||7|
|78275009||Obstructive sleep apnea syndrome||1||0||3||4|
|36969009||Placement of stent in coronary artery||3||0||32||35|
|128404006||Right heart failure||6||0||12||18|
|197321007||Steatosis of liver||19||3||12||34|
|439129009||Thrombosis of blood vessel||8||5||12||25|
|266257000||Transient ischemic attack||4||0||7||11|
|66071002||Type B viral hepatitis||6||0||1||7|
|50711007||Viral hepatitis C||6||0||8||14|
|128105004||Von Willebrand disorder||0||0||1||1|
Table 3: All studied conditions by category (EHR only, Tablet only or Both).
|Disorder||Tablet Only||EHR Only||Tablet And EHR (Both)||Total Patients With Disease||Neither||p-value|
|Gastroesophageal reflux disease||81||64||50||195||392||0.001|
|Hypertensive disorder, systemic arterial||49||23||246||318||269||0.0001|
|Malignant tumor of breast||13||8||65||86||501||0.0001|
|Thrombosis of blood vessel||8||5||12||25||562||0.004|
|Chronic obstructive lung disease||11||9||14||34||553||not ss|
Table 4: Results of a Chi-Square analysis for the 12 conditions where statistic numbers allowed us.
Results regarding completeness
After one year, we reviewed the EHR problem lists for the same cohort of patients to evaluate interval data for the conditions under study. Of the added 272 problems, 107 corroborated that which had not been previously corroborated in 97 patients (Figure 3 and Table 5).
|SNOMED CT Code||SNOMED CT Text||Tablet Only 1 Year Follow-Up||EHR Only 1 Year Follow-Up||Tablet And EHR (Both) 1 Year Follow-Up||Number Changed*|
|38341003||Hypertensive disorder, systemic arterial||20||22||275||29|
|254837009||Malignant tumor of breast||4||8||74||9|
|363518003||Malignant tumor of kidney||4||0||12||5|
|399068003||Malignant tumor of prostate||6||2||18||5|
|235595009||Gastroesophageal reflux disease||77||55||54||4|
|197321007||Steatosis of liver||17||3||14||2|
|363358000||Malignant tumor of lung||5||0||9||2|
|363478007||Malignant tumor of thyroid gland||1||4||12||2|
|13645005||Chronic obstructive lung disease||10||9||15||1|
|90205004||Cardiac revascularization with bypass anastomosis||2||0||21||1|
|363443007||Malignant tumor of ovary||1||0||2||1|
|371973000||Malignant neoplasm of uterus||1||1||5||1|
Table 5: Conditions that changed after 1 year in all cases the number of “Tablet Only” decreases and the number of “Both” increases.
Accuracy compared to laboratory results and compared to medications list
As previously mentioned, medication lists and laboratory results were reviewed to confirm the presence of hypercholesterolemia and diabetes mellitus.
Results for selected conditions
Hypercholesterolemia: We reviewed patients with hypercholesterolemia reported as a condition by the “Tablet Only” or by “EHR Only” or “Both.” Of these, 48 had hypercholesterolemia in the “EHR Only,” 78 had hypercholesterolemia reported in the “Tablet Only,” and 166 had hypercholesterolemia reported in “Both.” We established any cholesterol result >200 mg/dl as abnormal before reviewing the cholesterol laboratory tests . We concomitantly reviewed patient medication lists searching for cholesterol-lowering agents (Table 6). Of the 244 patients who self-reported hypercholesterolemia, either by “Tablet only” or “Both,” 213 were corroborated by laboratory values or lipid-lowering medications (Table 7).
Table 6: Cholesterol-lowering agents.
|Description||Tablet Only||EHR Only||Both||Total|
|Have lab result in EHR||63||41||139||243|
|Lab results abnormal (>200)||19||21||51||91|
|Number of patients with some lab abnormal, or using meds, or both*same patient can be in more than one group.||61||37||152||250|
Table 7: Hypercholesterolemia summary.
Of the 244 patients with hypercholesterolemia reported in the Tablet, 213 corroborated (87.29%) with the lab review, and of the 210 with hypercholesterolemia reported in the EHR, 189 corroborated (90%) with the lab review, suggesting that self-reports collected by the Tablet are as accurate as the EHR data. 26.71% of patients had hypercholesterolemia reported only on the tablet (“Tablet only”), and 16.44% of patients had hypercholesterolemia reported in the EHR but not in the Tablet (“EHR Only”).
Diabetes mellitus: A similar validation process was conducted with diabetes mellitus patients. Of this patient subset, 4 reported diabetes mellitus (DM) on the Tablet but the problem did not show in the EHR (“Tablet Only”) problem list; 17 had DM in the “EHR Only”; and 79 had DM reported in “Both” (Table 8). Blood glucose levels >126 mg/dl and glycohemoglobin (HbA1c) > 6.4 were considered abnormal when examining laboratory tests . Of the patients taking glucose-lowering medications (Table 9): Two out of 4 patients (50%) had reported DM on the “Tablet Only,” 9 out of 17 (52.94%) had a diagnosis of DM present only in the EHR (“EHR Only”), and 71 out of 79 patients (89.87%) had DM reported in “Both” (Table 8). Of the 83 patients with DM reported in the Tablet, 73 corroborated (87.95%) with our lab review, and of the 96 with DM reported by EHR, 80 corroborated (83.33%) with the lab review, suggesting that self-reports collected by the Tablet are as accurate as the EHR. 4% of patients had DM reported only on the tablet (“Tablet Only”) and 17% of patients had DM reported in the EHR but not in the Tablet (“EHR Only”).
|Description||Tablet Only||EHR Only||Both||Total|
|Have some lab result||4||16||72||92|
|Lab results abnormal||2||8||47||57|
|Number of patients with some lab abnormal, or using meds, or both* same patient can be in more than one group||2||9||71||82|
Table 8: Diabetes mellitus summary.
Table 9: Glucose-lowering agent medications.
It is assumed that this level of corroboration of results can be extrapolated to other conditions.
We have shown that the EHR problem lists in this study were incomplete, both at the start of the study and after one year of added emphasis under the Federal Meaningful Use program . We found that out of 1472 patients, 57% of EHR records (843 patients) had no reported problems initially despite the presence of patient-reported problems via the Tablet. Our analysis also included a review of the EHR problem list for the same cohort of patients after one year to determine if the accuracy of the reported problems had improved. Even one year later, 49% of EHR records (716 patients) had no diseases reported in the problem list. While it might be true that a portion of patients seen at the hospital do not have a disease diagnosis, that is not true for the majority of the cases. Despite hospital support for accurate problem lists, emphasized by the problem lists being included in the Meaningful Use objective, change was minimal and the EHR problem list remained incomplete.
Statistical analysis was conducted by performing Chi-squared test to determine if there was a statically significant difference between the problems reported on the EHR compared to the Tablet. Results of the analysis, in which 10 out of 12 reported disorders were statistically significant (p <0.05), indicate the existence of a real difference between the lists. Among those, only one condition was reported more often only in the EHR than by patients on the Tablet. This finding would indicate that the Tablet problem list was considerably more comprehensive than the EHR problem list. The absence of problems in the problem list can be explained in part by MGH’s role as a tertiary referral center. Patients are often followed elsewhere for basic care while seeking specialized care at MGH. Thus, many patients have only been seen by specialists, who may not have a vested interest in adding unrelated problems to the problem list.
We have shown that self-reported data appear to be accurate. Physician-reported diagnoses in the EHR problem list, laboratory results, and patient medication lists for hypercholesterolemia and diabetes mellitus were reviewed to validate the Tablet problem list. We have shown and it has been demonstrated elsewhere that this is a valid approach to identifying patient problems .
Of the 244 patients self-reported with hypercholesterolemia, 213 were corroborated by EHR, laboratory values or medication lists, supporting the accuracy of this self-reported problem. Absence of self-reporting was shown to miss 26.71% of patients with hypercholesterolemia, and absence of the EHR was shown to miss 16.44% of this patient population.
In 2002, Natajaran et al.  analyzed self-reports of hypercholesterolemia. They concluded that such information should be used carefully, since despite high specificity, the sensitivity was low. In our study, 87.3% of patients that reported having diagnosis of hypercholesterolemia had abnormal laboratory cholesterol levels or were taking cholesterol-lowering agents demonstrating that self-reports of hypercholesterolemia were accurate.
Of the 83 patients self-reported with diabetes mellitus, 73 corroborated with other data, supporting the accuracy of this selfreported problem list. Absence of self-reporting misses 4% of patients with diabetes mellitus and absence of EHR misses 17% of such patients.
A study conducted by Goldman et al.  revealed that self-reports of diabetes are an accurate estimate of the prevalence of diabetes in the studied population. Similar results were found by Goto et al.  in 2013, who found 70.4% sensitivity and 97.3% specificity in identifying patients with diabetes using self-reports. In our study, 87.9% of patients with self-reported diabetes mellitus had either abnormal laboratory results (blood glucose levels and HbA1c) or were taking glucoselowering agents, which corroborates with diabetes mellitus diagnosis.
Using both the patient-reported data and the EHR problem list is the best way of obtaining the most accurate and complete patient problem list. A study conducted in 2008 by Zakim and colleagues reached the same conclusion when comparing physician interview with computer-acquired medical histories . A logical approach to this problem would be to have the patient complete a self-administered questionnaire which is then reviewed by the physician for validation. Once validated, uploading the structured data into the EHR problem list is appropriate and should be relatively simple with good EHR design.
This study finds that the EHR problem list is often incomplete,  that the use of a self-administered Tablet questionnaire is an acceptable method for collecting family and personal history [6,10-13]. Using the EHR problem list and self-administered Tablet questionnaire together is the optimal approach to obtaining the most accurate patient problem list possible.
Conflict of Interest Disclosures
Dr. Hughes and Dr. Drohan are co-founders of Hughes RiskApps, LLC. Dr. Hughes is a speaker at Myriad Genetics.
This research was supported by the Division of Surgical Oncology, Massachusetts General Hospital.
Role of the Sponsor
The funding agreements ensure the authors’ independence in designing the study, interpreting the data, writing, and publishing the report.