| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Computer Applications |
1 From the Departments of Medical Informatics (G.H., C.F.) and Radiology (J.H.M.A., P.O.A.), Columbia University, 622 W 168th St, VC-5, New York, NY 10032; and Department of Computer Science, Queens College, City University of New York (C.F.). Received June 29, 2001; revision requested July 27; revision received September 27; accepted November 12. Supported by National Library of Medicine grants R01-LM06910, R01-LM06274, and R29-LM05627. Address correspondence to G.H. (e-mail: hripcsak@columbia.edu).
 |
ABSTRACT |
|---|
 TOP ABSTRACT INTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONREFERENCES |
|---|
MATERIALS AND METHODS: A natural language processor coded 10 years of narrative chest radiographic reports from an urban academic medical center. Coding for 150 reports was compared with manual coding. Frequencies and co-occurrences of 24 clinical conditions (diseases, abnormalities, and clinical states) were estimated. The ratio of right to left lung mass, association of pleural effusion with other conditions, and frequency of bullet and stab wounds were compared with independent observations. The sensitivity and specificity of the system’s pneumothorax coding were compared with those of manual financial coding.
RESULTS: The system coded 889,921 reports on 251,186 patients. On the basis of manual coding of 150 reports, the processor’s sensitivity (0.81) and specificity (0.99) were comparable to those previously reported for natural language processing and for expert coders. The frequencies of the selected conditions ranged from 0.22 for pleural effusion to 0.0004 for tension pneumothorax. The database confirmed earlier observations that lung cancer occurs in a 3:2 right-to-left ratio. The association of pleural effusion with other conditions mirrored that in the literature. Bullet and stab wounds decreased during 10 years at a rate consistent with crime statistics. A review of pneumothorax cases showed that the database (sensitivity, 1.00; specificity, 0.996) was more accurate than financial discharge coding (sensitivity, 0.17; P = .002; specificity, 0.996; not significant).
CONCLUSION: Internal and external validation in this study confirmed the accuracy of natural language processing for translating chest radiographic narrative reports into a large database of information.
© RSNA, 2002
Index terms: Computers • Picture archiving and communication system (PACS) • Quality assurance • Thorax, radiography, 60.1215
|
INTRODUCTION |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONREFERENCES |
|---|
Natural language processing (5–11) offers a solution. It converts machine-readable narrative text into a structured form. For example, a natural language processor might code this excerpt from a radiographic report, "Improved patchy opacity in the left lower lobe, no effusions seen," as follows: Finding, opacity; descriptor, patchy; body location, left lower lobe of lung; change, better; finding, pleural effusion; certainty, no. This structured format allows the data to be used for clinical research—generating and testing hypotheses with large samples and screening patients for studies on a large scale—and for clinical care by means of automatically generated alerts and reminders.
At least two independent groups have demonstrated that natural language processing can be as accurate as expert human coders for coding radiographic reports, as well as more accurate than simple text-based methods, such as searching for relevant phrases in the reports (9,12,13). Demonstration of the use of natural language processing to code complex reports, such as admission notes and discharge summaries, is promising but preliminary (10,11,14–19). Investigators in one study demonstrated the effective use of natural language processing to improve clinical care by improving respiratory isolation for tuberculosis (20). The potential of natural language processing to facilitate clinical research has been recognized (21,22) and demonstrated in a stroke database of 471 patient records (23). The purpose of our study was to evaluate translation with natural language processing of our 10-year database of chest radiographic reports and to compare our findings with those in the literature.
|
MATERIALS AND METHODS |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONREFERENCES |
|---|
An existing noncommercial natural language processor (MEDLEE) developed at Columbia University, New York, NY, and Queens College, City University of New York (8), was used to code all the information in the radiographic reports. It converted the narrative text to a semantic structure populated with a controlled vocabulary. The vocabulary was built especially for natural language processing, by using a large set of reports as its basis (thus ensuring essentially full coverage of the intended target reports), and contains links to standardized vocabularies such as International Classification of Diseases, Ninth Revision, Clinical Modification, or ICD-9-CM, diagnostic coding. The present study assessed 24 clinical conditions, which were chosen either because the processor’s accuracy had already been confirmed for them (10 conditions) (9,12,20) or because their frequency or co-occurrence was of interest. Both common (eg, atelectasis) and uncommon (eg, tuberculosis) conditions were included. The radiographic reports consisted of a clinical indication, a description of the findings, and an overall impression. Clinical indications represented information entered by the ordering physician, not by the radiologist. Suboptimal examinations were defined as those having inadequate patient positioning, poor aeration, inadequate collimation, or radiographic underexposure or overexposure. Examinations described as normal and as showing no active disease were lumped together for this analysis, but the natural language processor coded them separately.
Conditions were derived either from specific mention in a report ("possible pneumonia") or from other findings ("right lower lobe consolidation"). A computer-based query translated the detailed coding produced by the processor into a dichotomous answer (present or absent) plus an optional side of the body (right, left, or bilateral) for each condition. For example, the findings "pneumonia," "patchy opacity," "consolidation," "infiltrate," and so forth, indicated the possibility of acute bacterial pneumonia as long as the pneumonia certainty was not "no" or "very low certainty" and the pneumonia status was not "resolved." In some cases, the same finding was part of the differential diagnosis for more than one condition (eg, "upper lobe infiltrate" for tuberculosis and acute bacterial pneumonia); in these cases, both conditions were indicated as possible. The 24 queries were written by one of the authors (G.H.). It took an average of 4 hours to write and test each query and an average of 
hour to apply it to the database. Writing a query required selecting terms from a vocabulary and optionally adding logic. Most of the 4 hours was actually spent manually reviewing a large sample of reports to verify and improve accuracy.
Because the accuracy of the processor had not been tested for many of the conditions, a coder was enlisted to verify the accuracy of the translation. The coder was a medical school graduate with medical informatics training (focused especially on coding issues). The definitions used for coding were based on advice from two experienced radiologists (J.H.M.A., P.O.A.) and an internist (G.H.). A reliability study was used to compare this coding with that of six radiologists and seven internists doing the coding themselves. The study verified that the coder was sufficiently reliable to create a reference standard for estimating metrics such as sensitivity and specificity (24). The coder reviewed the text of 150 reports chosen randomly from 1995 and classified each condition as present or absent for each report. Using that classification as the reference standard, we estimated the accuracy of the natural language processor and the computer queries (the "system") and compared these findings with previous estimates of accuracy of the system.
The conditions were tallied for the full 10-year set of reports and for an "entry reports" subset. An entry report was defined as the first report available for a patient, reflecting the patient’s first chest radiograph obtained at the medical center. The term signifies the patient’s entry into the institution. Whereas the full set of reports emphasized inpatients and especially intensive care unit stays, the entry reports emphasized outpatient, emergency department, and admission examinations. Co-occurrence of conditions was determined for the full set and for the entry reports. Those conditions for which a side of the body was relevant were classified as right, left, bilateral, or indeterminate (not stated). Ratios of right to left lesions were calculated.
Four hypotheses, each chosen before the data were analyzed, served as external validation for the system. The first hypothesis was based on the work of Goldman et al (25), who reported that lung cancer favors the right lung over the left by a 3:2 ratio. Although radiographic analysis does not prove a diagnosis of lung cancer, we selected as a surrogate marker reports with lung masses, excluding small nodules, ill-defined opacities, and clearly metastatic or inflammatory lesions. We estimated the right-left ratio for lung mass in the full set of reports, and, to factor out any effect of patients with multiple reports, for each patient we selected only the first report that suggested a lung mass.
The second hypothesis was that pleural effusion would be associated with other conditions in a frequency commensurate with reports in the literature (26–36). We measured the frequency of pleural effusions in patients with congestive heart failure, lung mass, and opacity consistent with acute bacterial pneumonia, and we tallied the side of the body on which effusion occurred in congestive heart failure (a disease for which the distribution of pleural effusions is of clinical interest [37]). For the latter analysis, one author (G.H.) manually reviewed 200 randomly chosen reports of congestive heart failure and pleural effusion (50 bilateral, 50 right, 50 left, and 50 indeterminate) to eliminate effusions due to other obvious causes.
The third hypothesis asserted that the frequency of bullet and stab wounds seen or assessed on radiographs should track the decade-long reduction in crime rate in U.S. cities (38). The 10-year trend of bullet and stab wounds in chest radiographs (based on mention in the description or impression) was estimated by using a generalized linear model (binomial distribution and logit link function) (39). The 10-year trends of violent crime, aggravated assault, and murder in New York City were estimated by using the 1989 to 1998 Uniform Crime Reporting statistics (38), by using a generalized linear model (Poisson distribution and log link function). The trends (time coefficients in the models) were compared.
The fourth hypothesis asserted that the system’s coding would be more accurate than the hospital’s financial coding for conditions that are normally diagnosed at radiography. We selected pneumothorax for this analysis and used data from February 1997. We compared patients with a diagnosis made by using the system based on radiographs to patients identified by using the hospital’s ICD-9-CM diagnostic coding. The latter coding was performed manually by coders who had access to the complete medical record, including all of the radiographic reports. One author (G.H.) reviewed all selected records and assigned patients to three categories: definite (definite radiographic evidence with clinical correlation), possible (uncertain pneumothorax noted on chest radiograph with no correlating clinical evidence), or no pneumothorax (no evidence to suggest pneumothorax).
Coding accuracy was quantified by calculating sensitivity, specificity, and area under the receiver operating characteristic (ROC) curve (Az) (40) for each condition and then by averaging across conditions. The exact method of Stern (41) was used to calculate binomial confidence intervals throughout. Findings were compared with those in the literature.
|
RESULTS |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONREFERENCES |
|---|
On the 150 manually coded reports, the system’s average sensitivity was 0.81 (95% CI: 0.71, 0.87), average specificity was 0.99 (95% CI: 0.97, 0.99), and average ROC curve area was 0.95 (95% CI: 0.92, 0.97). These findings corresponded well with earlier results for that system (sensitivity, 0.81–0.86; specificity, 0.98; Az, 0.95–0.96; differences not significant) (9,12) and were comparable to the results of previous reports of expert human coding (sensitivity, 0.85–0.87; specificity, 0.98; Az, 0.96; differences not significant) (9,12,13).
Tables 1–3 illustrate the content of the database. Table 1 shows the frequencies of conditions coded by the system. Conditions associated with intensive care and complications had high frequencies in the full set of reports: pleural effusion (22% [197,359 of 889,921 reports]), congestive heart failure (20% [177,821 of 889,921 reports]), pneumonia (18% [157,293 of 889,921 reports]), atelectasis (15% [136,089 of 889,921 reports]), and central venous catheter (11% [98,880 of 889,921 reports]). Reports without active disease (16% [137,661 of 889,921 reports]) were also fairly common. For the subset of entry reports, conditions associated with intensive care and complications were much less frequent (eg, "central venous catheter" appeared in only 2% [5,491 of 251,186] of entry reports), and reports without active disease (30% [75,747 of 251,186]) were more common. Chronic conditions (eg, chronic obstructive pulmonary disease, neoplastic process) were approximately equally common in both sets of reports. Ordering physicians are mandated to enter clinical indications on all radiographic orders; compliance was 84% (750,724 of 889,921 reports).
|
|
|
Clinical requests to rule out a condition were accompanied by that condition to varying degrees (Table 2). In entry reports, congestive heart failure occurred in 34% (1,030 of 3,068), pneumonia in 19% (4,538 of 24,446), and rib fracture in 12% (194 of 1,666) of reports of studies ordered to rule out the condition. Pneumothorax accompanied "rule out pneumothorax" in 19.3% (4,577 of 23,723) of all reports with the request and in 7.8% (212 of 2,718) of entry reports with the request (not shown in Table 2 because other conditions were more frequent). Pneumonia and congestive heart failure most often accompanied shortness of breath for entry reports. In general, clinical indications were most often associated with abnormal conditions in the full set of reports but with normal examinations in the subset of entry reports.
The body side of the lesion varied among the assessed conditions: Atelectasis and pleural effusion tended to be left sided, whereas bullet and stab wounds, neoplastic processes, pneumothorax, and tuberculosis tended to be right sided (Table 3). For conditions such as central venous catheter placement, location is ambiguous: a "right central" catheter could enter the body on the right side or it could enter the right side of the heart. For conditions such as redistribution of pulmonary blood flow in congestive heart failure, a body side is not usually specified, so most of the 95% of indeterminate cases were presumably bilateral.
The data supported the observation of Goldman et al (25) of a 3:2 right-left ratio for lung cancer. The ratio of right to left lung masses was 1.53 (95% CI: 1.46, 1.61), based on 4,456 right lung mass reports and 2,904 left lung mass reports (Table 3). The right-left ratio for the first occurrence of a lung mass was 1.49 (95% CI: 1.40, 1.58), based on 2,303 right lung mass and 1,547 left lung mass reports. Goldman et al (25) found a 1.45 ratio (95% CI: 1.31, 1.62) based on word counts in a database of thoracic radiology reports for patients with lung cancer and a 1.47 ratio (95% CI: 1.34, 1.61) based on a review of published lung cancer series.
The frequency of pleural effusion in congestive heart failure, lung cancer, and bacterial pneumonia is shown in Table 4. The system agreed well with the literature in all three conditions (26–30). In the present study, pleural effusions in association with all reports of congestive heart failure were 54% (46,702 of 85,739) bilateral, 14% (12,260 of 85,739) only on the right, 20% (16,978 of 85,739) only on the left, and 11% (9,799 of 85,739) indeterminate. After elimination of other causes and indeterminate cases, pleural effusions in patients with congestive heart failure were 78% bilateral (31 of 50 x 54%), 12% (18 of 50 x 14%) only on the right, and 10% (12 of 50 x 20%) only on the left. These results are well supported by clinical series (67% bilateral, 23% only on the right, 10% only on the left [31–33]) and postmortem studies (81% bilateral, 11% only on the right, 7% only on the left [26,34–36]), although older clinical series suggest that bilateral pleural effusions are less common than unilateral effusions (35% bilateral, 49% only on the right, 17% only on the left [34–36]).
|
The system appeared to be more accurate than financial discharge coding for pneumothorax. In February 1997, 2,897 patients were admitted, and 1,340 (46%) of them had at least one chest radiograph. Financial discharge coding reported 11 definite cases of pneumothorax, and one false-positive case ("s/p PTX" was mistakenly coded as pneumothorax instead of parathyroidectomy). The system reported 83 cases: the 11 definite cases marked by financial discharge coding, 55 other definite cases, 16 cases of possible pneumothorax, and one false-positive case (a radiologist’s suggestion of repeat examination to rule out pneumothorax was erroneously interpreted as possible pneumothorax). By considering only definite cases of pneumothorax, the system had a sensitivity of 1.00 (66 of 66) and a specificity of 0.9996 (2,814 of 2,815), and financial discharge coding had a sensitivity of 0.17 (11 of 66) and a specificity of 0.9996 (2,814 of 2,815). Even after cases of small or moderate pneumothorax after invasive chest procedures were eliminated, financial discharge coding had a sensitivity of only 0.35 (six of 17). Financial discharge coding even missed one of the two cases of spontaneous pneumothorax. Therefore, the system was significantly more sensitive than financial discharge coding (P = .002), although the specificity of the two methods did not differ.
|
DISCUSSION |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONREFERENCES |
|---|
The latter result probably stems from the constraints of financial coding: One can code only a limited number of diagnoses, and the goal is to maximize payment, not to achieve perfect clinical accuracy. For example, a small postoperative pneumothorax might not normally require coding. Nevertheless, the result demonstrates that automated coding of clinical reports is a better source of information than manual financial discharge coding for answering clinical questions, at least for some conditions. This result is important, given the widespread use of manual financial discharge coding to support clinical research (2), especially for screening.
The predominant use for this database at our institution has been clinical research. It has been used to screen patients for enrollment in studies. For example, we found 30,136 reports with pneumothorax for 8,790 patients; these reports and the associated medical records will be further screened for a study on spontaneous pneumothorax. In another study, we found that 14,597 radiographic reports on 8,398 patients demonstrated a lung mass. An assessment of the intrapulmonary distribution of those masses is in progress.
The database does have limitations. Neither natural language processing nor the reports themselves are perfect. The radiograph must capture the patient’s state, the radiologist who reads the image must recognize that state, and the radiologist’s intention must be transcribed accurately by the typist or recording system and then must be understood by the person who reads the report (despite, for example, variations in clinicians’ vocabularies). Even if the radiograph, the radiographic report, and the natural language processor were perfect, the drawing of clinical conclusions only on the basis of radiographic findings would still lead to uncertainty. Databases produced in this way may therefore not achieve the accuracy and completeness possible with a manually collected and verified clinical registry and cannot be expected to replace rigorous clinical trials. Nevertheless, the size and clinical detail of the database support queries that would not be feasible without it: There are too many reports for manual review, and the financial discharge coding has not proved reliable (2,3). Furthermore, the database is not limited to the current study’s 24 conditions. The authors chose those conditions as representative of common or important clinical entities. In fact, the natural language processor generated over 4,000 different conditions and findings for the 889,921 reports. The combination of screening through automated coding and selected intensive manual review is a powerful tool that exploits the automated system’s ability to handle high volume and a person’s ability to make complex judgments.
Although this study did not address the ability to use the natural language processor at multiple institutions, another study did address it (12). The processor was in fact transferable to another institution, although the clinical queries that interpreted the processor output did benefit from some modification.
Alternatives to natural language processing have been assessed. In some cases, a simple text-based search engine can detect relevant reports in a large database (25,42). Nevertheless, it has been demonstrated that natural language processing can achieve higher accuracy than such search engines (9,12,13). Most of the difference was attributable to reduced specificity, since the search engine selected many reports in which a condition was actually being denied (eg, "no evidence to suggest pneumonia or pneumothorax").
Chest radiographic reports are fairly simple, but the system has also been tested on more complex narrative reports such as CT and MR imaging of the head (23). All reports generated by the radiology department at the medical center are currently coded by means of natural language processing. The full potential of natural language processing will not be realized until it can be extended to very complex narrative reports such as admission notes and discharge summaries (4). Although preliminary work (10,11,14–19) shows promise, further research is needed and accuracy must be confirmed with studies similar to those that have been done for radiographic reports.
In its current form, natural language processing can put millions of clinical reports at the fingertips of researchers and clinicians. The reports can now be coded about as accurately as if a coder had gone through each one by hand. The queries that are needed to convert the radiographic findings to clinically accurate answers do require knowledge of the coding and of medicine, as well as several hours of effort. Once queries are written for one study, however, they can often be reused for similar studies, with minimal effort.
In conclusion, we used natural language processing to create a database of 889,921 chest radiographic reports on 251,186 patients. The accuracy of the database’s coding was comparable to that of manual coding and superior to the accuracy of financial discharge coding for pneumothorax. The frequencies and co-occurrences of several clinical conditions identified with natural language processing matched those previously reported in the literature. Natural language processing can produce huge databases of coded clinical information and offers the potential to become an important tool for clinical research.
|
FOOTNOTES |
|---|
Abbreviation: ROC = receiver operating characteristic
C.F. is named in a patent held by Columbia University for the natural language processor described in this report.
Author contributions: Guarantor of integrity of entire study, G.H.; study concepts and design, all authors; literature research, all authors; clinical studies, G.H.; data acquisition, G.H., C.F.; data analysis/interpretation, all authors; statistical analysis, G.H.; manuscript preparation, G.H.; manuscript definition of intellectual content, all authors; manuscript editing, G.H.; manuscript revision/review and final version approval, all authors.
|
REFERENCES |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODSRESULTSDISCUSSIONREFERENCES |
|---|
This article has been cited by other articles:
|
|
 |
|
  F. P. Morrison, L. Li, A. M. Lai, and G. Hripcsak Repurposing the Clinical Record: Can an Existing Natural Language Processing System De-identify Clinical Notes? J. Am. Med. Inform. Assoc., January 1, 2009; 16(1): 37 - 39. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 D. L. Weiss and C. P. Langlotz Structured Reporting: Patient Care Enhancement or Productivity Nightmare? Radiology, December 1, 2008; 249(3): 739 - 747. [Full Text] [PDF]
|
|
|
|
 |
|
 P. A. Dang, M. K. Kalra, M. A. Blake, T. J. Schultz, E. F. Halpern, and K. J. Dreyer Extraction of Recommendation Features in Radiology with Natural Language Processing: Exploratory Study Am. J. Roentgenol., August 1, 2008; 191(2): 313 - 320. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
E. S. Chen, G. Hripcsak, H. Xu, M. Markatou, and C. Friedman Automated Acquisition of Disease Drug Knowledge from Biomedical and Clinical Documents: An Initial Study J. Am. Med. Inform. Assoc., January 1, 2008; 15(1): 87 - 98. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. H. Brown, T. Speroff, E. M. Fielstein, B. A. Bauer, D. L. Wahner-Roedler, R. Greevy, and P. L. Elkin eQuality: Electronic Quality Assessment From Narrative Clinical Reports Mayo Clin. Proc., November 1, 2006; 81(11): 1472 - 1481. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 C. J. McDonald Corporate Strategies for Computerization Ann Intern Med, September 5, 2006; 145(5): 395 - 396. [Full Text] [PDF]
|
|
|
|
|
|
Y. Huang, H. J. Lowe, D. Klein, and R. J. Cucina Improved Identification of Noun Phrases in Clinical Radiology Reports Using a High-Performance Statistical Natural Language Parser Augmented with the UMLS Specialist Lexicon J. Am. Med. Inform. Assoc., May 1, 2005; 12(3): 275 - 285. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
B. J. Thomas, H. Ouellette, E. F. Halpern, and D. I. Rosenthal Automated Computer-Assisted Categorization of Radiology Reports Am. J. Roentgenol., February 1, 2005; 184(2): 687 - 690. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
K. J. Dreyer, M. K. Kalra, M. M. Maher, A. M. Hurier, B. A. Asfaw, T. Schultz, E. F. Halpern, and J. H. Thrall Application of Recently Developed Computer Algorithm for Automatic Classification of Unstructured Radiology Reports: Validation Study Radiology, February 1, 2005; 234(2): 323 - 329. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
C.-S. Yam, N. Rofsky, J. Kruskal, and A. Sitek Development of a Radiology Report Monitoring System for Case Tracking Am. J. Roentgenol., January 1, 2005; 184(1): 343 - 346. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
Y. Huang, H. J. Lowe, and W. R. Hersh A Pilot Study of Contextual UMLS Indexing to Improve the Precision of Concept-based Representation in XML-structured Clinical Radiology Reports J. Am. Med. Inform. Assoc., November 1, 2003; 10(6): 580 - 587. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
C. P. Langlotz Automatic Structuring of Radiology Reports: Harbinger of a Second Information Revolution in Radiology Radiology, July 1, 2002; 224(1): 5 - 7. [Full Text]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| RADIOLOGY | RADIOGRAPHICS | RSNA JOURNALS ONLINE |
