| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH |
|
|
|
|||||||
|
|||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Special Report |
1 From the Division of Computing and Information Services, Department of Radiology, Massachusetts General Hospital and Harvard Medical School, 100 Charles River Plaza, Suite 471, Cambridge St, Boston, MA 02114. Received January 9, 2004; revision requested March 12; revision received April 6; accepted May 19. Address correspondence to K.J.D. (e-mail: kdreyer@partners.org).
 |
ABSTRACT |
|---|
 TOP ABSTRACT INTRODUCTIONMATERIALS AND METHODS RESULTSDISCUSSIONAPPENDIXREFERENCES |
|---|
MATERIALS AND METHODS: The study was approved by the Human Research Committee of the institutional review board. Consecutive de-identified radiology reports (n = 1059) comprising results of barium studies (n = 99), computed tomography (n = 107), mammography (n = 90), magnetic resonance imaging (n = 108), nuclear medicine (n = 99), positron emission tomography (n = 106), radiography (n = 212), ultrasonography (n = 131), and vascular procedures (n = 107) were independently analyzed by two radiologists and then with LEXIMER to categorize the reports into FT and FT0 (containing or not containing clinically important findings) categories and RT and RT0 (containing or not containing recommendations for subsequent action) categories. Accuracy, sensitivity, specificity, and positive and negative predictive values of LEXIMER for placing reports into FT and FT0 and RT and RT0 categories were assessed by using appropriate statistical tests.
RESULTS: There was strong interobserver concordance between the two radiologists for placing radiology reports into FT and RT categories (
= 0.9, P < .01). For the LEXIMER program, accuracy, sensitivity, specificity, and positive and negative predictive values, respectively, were 97.5% (95% confidence interval [CI]: 96.6%, 98.5%), 98.9% (95% CI: 97.9%, 99.6%), 94.9% (95% CI: 93.1%, 96.0%), 97.5% (95% CI: 96.6%, 98.0%), and 97.7% (95% CI: 95.8%, 98.8%) for placing radiology reports into FT and FT0 categories and 99.6% (95% CI: 99.2%, 99.9%), 98.2% (95% CI: 95.0%, 99.6%), 99.9% (95% CI: 99.4%, 99.99%), 99.4% (95% CI: 96.3%, 99.9%), and 99.7% (95% CI: 98.9%, 99.9%) for placing reports into RT and RT0 categories.
CONCLUSION: LEXIMER is an accurate automated engine for evaluating the percentage positivity of clinically important findings and rates of recommendation for subsequent action in unstructured radiology reports.
© RSNA, 2004
|
INTRODUCTION |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODS RESULTSDISCUSSIONAPPENDIXREFERENCES |
|---|
Thus, so that we could analyze diagnostic yield and frequency of recommendations for subsequent action in unstructured radiology reports, we have developed an automatic computer algorithm based on information theory that is called Lexicon (a machine-readable dictionary) Mediated Entropy Reduction (LEXIMER). The purpose of this study was to validate the accuracy of LEXIMER, an information theory–based computer algorithm for independent analysis and classification of unstructured radiology reports based on the presence of clinically important findings (FT, where T represents "true") and the presence of recommendations for subsequent action (RT).
|
MATERIALS AND METHODS |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODS RESULTSDISCUSSIONAPPENDIXREFERENCES |
|---|
Absent information included that pertaining to patient identifiers, date of examination report, and referring and reporting physicians. To ensure that the study cohort included one radiology report per patient, 50 reports of examinations performed in the same patients were excluded from the study cohort. Ten administrative reports without report text were also excluded. Thus, the study cohort comprised 1059 radiology reports that included reports of barium studies (n = 99), CT (n = 107), mammography (n = 90), MR imaging (n = 108), nuclear medicine (n = 99), PET (n = 106), radiography (n = 212), US (n = 131), and vascular procedures (n = 107). The data were collected in the Department of Radiology Informatics at Massachusetts General Hospital. Reports were included in the study irrespective of patient age and sex, presenting symptoms, current treatment, and results of previous tests. The study cohort represented reports from 42 staff radiologists from the full spectrum of the major radiology subspecialties at our institution; this spectrum includes pediatric, abdominal, thoracic, vascular and interventional, and musculoskeletal radiology divisions, as well as neuroradiology and nuclear medicine divisions.
Report Database Categorization by Radiologists
Two radiologists (M.M.M., with 10 years of experience, and M.K.K., with 5 years of experience) independently categorized the de-identified radiology reports on the basis of the clinical importance of findings (ie, whether the findings indicated the presence of disease and/or had the potential to alter patient care and/or outcome) and recommendations (ie, whether the report contained advice for any sort of subsequent action). Both radiologists were blinded to the results of analysis of radiology reports with LEXIMER and to any clinical information. After independent assessment, and to minimize the possibility of human error, the consensus opinion of the two radiologists was regarded as the reference standard for categorization of the radiology reports.
The unstructured radiology reports were categorized independently on the basis of the presence of clinically important findings (FT) and the presence of recommendations for subsequent action (RT). Specifically, reports were classified into FT (reports that contain clinically important findings) and FT0 (reports that do not contain clinically important findings) categories and into RT (reports that contain recommendations for subsequent action) and RT0 (reports that do not contain recommendations for subsequent action) categories. Most reports at our institution, although they are unstructured overall, are divided into the following three sections: study technique or protocol, findings, and impression. If an impression section was present in the report, findings in the report were not considered to be clinically important unless they were reintroduced or summarized in the impression. With this practice, we hoped to avoid inclusion of incidental clinical findings that were unrelated to the study indications; such incidental findings are often mentioned in the findings section of reports.
Positive findings were defined as new observations that could be of clinical importance and included the following findings: all interventional procedures in which the procedure was initiated; hiatus hernia of any size; gastroesophageal reflux; esophageal dysmotility (specific or nonspecific); gallstone(s); presbyesophagus; diverticulosis with or without diverticulitis; any change in size or description (such as contrast enhancement at imaging) of any previous finding; general results (eg, bone density, ventilation-perfusion quantitation results, bone age), unless they were stated as normal; radiation therapy change(s) if mentioned in the impression; atelectasis; emphysema; findings classified into Breast Imaging Reporting and Data System (BI-RADS) categories 3, 4, or 5; findings classified into BI-RADS category 0 (this category can often involve a finding); cerebral volume loss if not noted as being consistent with the patient’s age; old infarcts; degenerative changes (when mentioned in the report’s impression section); and paranasal sinus disease (when deemed in the report’s impression section to exist).
When found alone, findings were categorized as negative if they represented one or more than one of the following: a stable finding noted in prior imaging reports, a finding that had not changed in any way from that seen at a previous examination (exception: a finding that was newly defined as something that it was not defined as initially [eg, "no change in X and therefore this represents Y"]), expected postsurgical findings, descriptions of expected surgical outcomes (eg, "surgical clips in the gallbladder consistent with prior cholecystectomy"), microangiopathic or white matter ischemic changes, findings classified into BI-RADS category 1 with no other mentioned findings, and findings classified into BI-RADS category 2 with no other mentioned findings.
In addition, radiologists also categorized each report into those with recommendations and those without recommendations. A report containing a suggestion, a recommendation, or advice for any action(s) was considered an RT report and included recommendations that one or more of the following be performed: other examinations or procedures, clinical correlation, comparison of present results with those of other examinations, follow-up procedures, and review of the results of another examination or its report.
Report Analysis with LEXIMER
Please see the Appendix for details regarding LEXIMER. For analysis of de-identified radiology reports, a Health Level 7, or HL-7, link was created to transfer reports directly from our hospital’s radiology information system to the LEXIMER engine for processing. HL-7 is a specification protocol for electronic data (encompassing clinical and administrative information) interchange system between health care institutions and between different computer systems within these institutions. It defines the format and the content of the messages that applications pass to one another in different circumstances. A computer database was created to serve as the repository for the preliminary data, distilled information, and statistical results. A software programmer (A.M.H.), who was blinded to the results of radiologists’ interpretation of the reports, uploaded all 1059 radiology reports to the LEXIMER search engine for automatic categorization of reports into FT and RT categories (with a binary end score).
No human judgment was involved in the categorization of reports with the LEXIMER program. In addition, neither the radiologists nor the LEXIMER program had access to clinical reports or previous radiology reports. The time taken by the program for analysis and classification of all reports into FT and RT categories was also recorded. The time studies were conducted with a standard personal computer with a 750-MHz central processing unit before any code-optimization techniques were applied.
Statistical Analysis
The sample size of the radiology reports assessed in the present study was determined by the width of the anticipated 95% confidence interval (CI). The interpretations of the two radiologists and LEXIMER in terms of FT and RT category placement were tabulated in EXCEL program worksheets (Microsoft, Redmond, Wash) for data analysis (T.S.). Concordance between the two radiologists was determined with the test of interobserver agreement (E.F.H.). Subsequently, any difference of opinion between the radiologists was resolved by mutual consensus. Total numbers of true-positive, true-negative, false-positive, and false-negative cases in FT and RT categories were determined for LEXIMER by using the consensus of the radiologists’ opinions as the standard of reference.
Statistical analysis was performed with the Excel software program to determine accuracy, sensitivity, specificity, and positive and negative predictive values (M.K.K., E.F.H.). The accuracy (A) of LEXIMER was calculated by using the following formula: A = 100 · [(TP + TN)/SS], where TP represents the number of true-positive categorizations; TN, the number of true-negative categorizations; and SS, the sample size.
In addition, accuracy, sensitivity, specificity, and positive and negative predictive values of LEXIMER for each individual imaging modality were determined. The frequency (F) of positive FT and RT categorizations in radiology reports was calculated by using the following equation: F = 100 · (PC/SS), where PC represents the number of positive categorizations (into either the FT or the RT category) and SS represents the sample size.
|
RESULTS |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODS RESULTSDISCUSSIONAPPENDIXREFERENCES |
|---|
= 0.9, P < .01). The overall frequency or yield of positive FT (clinically important findings) categorizations in the 1059 radiology reports assessed in the present study was 66.8% (707 of 1059 reports; 95% CI: 63.9%, 69.6%). The overall frequency of RT categorizations was 16.3% (173 of 1059 reports; 95% CI: 14.7%, 18.7%). A breakdown of the frequency rates of positive FT and RT categorizations of radiology reports for each imaging modality assessed in our study is provided in the Figure.
|
LEXIMER’s classification of the 1059 radiology reports into the RT category resulted in 170 (16.1%) true-positive categorizations, 885 (83.6%) true-negative categorizations, one (0.1%) false-positive categorization, and three (0.3%) false-negative categorizations. The accuracy, sensitivity, specificity, and positive and negative predictive values of LEXIMER for classification of radiology reports into the RT category (with respect to the radiologists’ consensus opinion) were 99.6% (95% CI: 99.2%, 99.9%), 98.2% (95% CI: 95.0%, 99.6%), 99.9% (95% CI: 99.4%, 99.99%), 99.4% (95% CI: 96.3%, 99.9%), and 99.7% (95% CI: 98.9%, 99.9%), respectively.
The accuracy, sensitivity, specificity, and positive and negative predictive values of LEXIMER for classifying radiology reports into FT and RT categories, according to each individual imaging modality, are summarized in Tables 1 and 2.
|
|
|
DISCUSSION |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODS RESULTSDISCUSSIONAPPENDIXREFERENCES |
|---|
Radiology reports can exist in one of two forms—structured and unstructured. Results of recent studies have demonstrated the potential benefits of structured medical reports for research, teaching, and organization of patient medical records (4–6). In a recent editorial (3), Langlotz opined that structured radiology report databases can aid in quantifying the yield of radiology procedures for clinically important findings and recommendations, in addition to providing real-time decision support to the radiologist.
Despite the advantages of a structured reporting format, most radiology reports are unstructured and exist in free text form, which makes them difficult to summarize and analyze with an automated program or process. Hripcsak et al (7) recently described the use of an automatic noncommercial natural language processor (developed at Columbia University and Queens College of the City University of New York) for automatic coding of unstructured reports. The authors reported that, as compared with manual coding, the natural language processor had 81% sensitivity and 99% specificity in the coding of frequencies and co-occurrences of 24 clinical conditions (including diseases, abnormalities, and clinical states) in 150 chest radiography reports. Subsequently, the authors used the processor to code 889 921 unstructured chest radiography reports for clinical conditions.
Inspired by these impressive results (7) and the opinions expressed by Langlotz (3), we developed an automatic search engine, LEXIMER, to analyze unstructured radiology reports for clinically important findings in order to determine the diagnostic yield of radiology procedures and the rates of recommendation of further radiologic or clinical examinations. To the best of our knowledge, LEXIMER represents the first automatic search engine developed to assess for FT and RT in unstructured radiology reports with the objective of determining the yield and recommendation practices related to all major imaging modalities, including radiography, barium studies, mammography, nuclear medicine, PET, US, CT, MR imaging, and vascular studies. A U.S. patent request for LEXIMER has been submitted and is currently pending approval.
Our study results show that LEXIMER is a sensitive, specific, and accurate program for classifying free-text radiology reports into FT and RT categories, regardless of imaging modality. The sensitivity and specificity of LEXIMER for assessing unstructured reports, as determined in this study, which included 1059 reports, are better than the sensitivity and specificity of the natural language processing program evaluated in an earlier study, which included 150 reports (7). However, the accuracy of LEXIMER for categorization may also be affected by a particular mixture of reports from different imaging modalities analyzed in this study. It is important to note that the BI-RADS classifications stated in mammography reports may have also contributed to the greater accuracy of the LEXIMER program.
In addition to finding that LEXIMER is sensitive, specific, and accurate, when we divided the time taken by radiologists by the time taken by LEXIMER (ie, 21.2 hours divided by 24 seconds) we found that report interpretation with LEXIMER is about 3180 times faster than manual interpretation of reports. We believe that server-based operation of optimized code will result in a further tenfold increase in performance. Speed of interpretation is critical because assessment of radiology practice requires interpretation of large databases of radiology reports.
Interestingly, contrary to the usual perception, the results of our study showed high diagnostic yield and low rates of recommendations in radiology reports for most imaging modalities except mammography (which is predominantly a screening modality) and CT scanning (for which recommendations are higher than for other imaging modalities). However, we believe that any inference about diagnostic yield and recommendation rates in radiology practice or for modalities must be supported by the analysis of comprehensive databases. These inferences are the subject of future works.
There were some considerations involved in using the LEXIMER program. Most false-negative categorizations by LEXIMER resulted from a failure of radiologists to include all positive findings or recommendations in the impression section of the reports. Most false-positive categorizations were noted to have occurred owing to lengthy impression sections in which "stability" or "no change" was not explicitly mentioned for each finding in the impression. Both radiologists reported the results of these examinations as true-negative by matching impressions with the text of the reports. However, LEXIMER can be adjusted according to patterns of unstructured radiology reports to increase its sensitivity and specificity for an individual imaging modality, specialty, or report type.
In classifying radiology reports into FT and RT categories, LEXIMER can be trained to identify and classify information such as ordering physician, reporting radiologist, imaging procedure, patient age and sex distribution, diagnosis, indications mentioned in the report, and report word count. It can be used to assess the diagnostic yield of radiology reports (according to whether they report positive or negative results) and suggested recommendations; these assessments can aid in the calculation of examination effectiveness or a "radiology significance index" for various combinations of radiologic procedures, indications, and demographics.
The program can also be used to determine deviations from benchmark standards for all permutations of high-radiology-significance-index procedures, including deviations from internal cohorts, deviations from nationally observed standards, and temporal deviations. LEXIMER can help in evaluating the yield of examinations for performance analysis (quality assurance and quality control) to stratify radiology departments on the basis of yield and recommendations and benchmark individual radiologists against radiologist cohorts or nationally observed standards. In addition, LEXIMER can be used to offer real-time automatic notification of missing recommendations when they are indicated by examination findings and automatic display of protocol advice for particular reports. Real-time monitoring of reports with LEXIMER can assist in reducing errors in radiology reports through notification of report inconsistencies at the time of interpretation and notification of recommendation guidelines not followed in the report (eg, failure to recommend follow-up for a pulmonary nodule).
We believe that direct assessment of radiology reports with LEXIMER will provide a better estimate for measuring radiology service utilization compared with metrics such as examination volume and growth rates. In addition, utilization of high-cost examinations can be evaluated to stratify individual physicians or groups ordering practices, benchmark individual physician practices against those of physician cohorts or nationally observed standards, monitor practice change within groups, and assess effects on education and retraining processes (8).
There were limitations in our study. Because we validated LEXIMER by using radiology reports from only one institution, the results of this study may not be replicable in the evaluation of radiology report databases from other institutions. However, we believe that the large cohort of unstructured radiology reports used in the current analysis is representative of the pattern or style of radiology reports in most institutions in the United States; this suggests that our results regarding the accuracy of LEXIMER should also be reproducible at other institutions. In addition, the radiology reports assessed in our study cohort included reports generated by 42 radiologists from all radiology subspecialties; this fact emphasizes the wide inherent variation in the report database at our institution.
Although we determined the sample size of the reports from each imaging modality on the basis of the width of the anticipated CI, considering the potential applications of the program, a criticism of our study could be that we did not validate LEXIMER against a multi-institution report database. Furthermore, although classification of findings as positive or negative was decided in consultation with experienced radiologists from each subspecialty, we did not consult referring physicians or correlate findings with the clinical records of individual patients. In addition, we did not grade positive findings for importance with respect to the clinical indications for the imaging study. For example, diverticulosis, if mentioned in the impression, was considered to be a positive finding, but this finding may not be critical in a patient with a terminal illness. Although the greater frequency of positive findings noted in our study cohort may have been related to the particular set of findings defined as clinically important, LEXIMER can also be trained to identify a more stringent set of findings. Another criticism of our study could be that both LEXIMER and the radiologists categorized radiology reports without knowledge of the patient’s clinical history; this practice can make determination of the true importance of findings difficult.
In summary, the LEXIMER program performs accurate, rapid, and automatic analysis of unstructured radiology reports for clinically important findings and recommendations. Analysis of unstructured radiology report databases with LEXIMER can help in determining patterns of radiology utilization and radiology significance indexes and influence future insurance reimbursement decisions for radiology procedures.
|
APPENDIX |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODS RESULTSDISCUSSIONAPPENDIXREFERENCES |
|---|
With the inception of digital information, these concepts have been rediscovered and redeployed for purposes beyond their original intent that include the analysis of digital text for computational linguistics. Computational linguistics, also called natural language processing or natural language understanding, is an automated method for processing text and deducing its syntactic and semantic structures to understand the nature of language, extract specific information from a text, or produce automated summaries. The syntactic structure refers to the arrangement of a word in a sentence and the way that arrangement affects the meaning of the word, whereas semantic level identifies the meaning of words from the context of the sentences.
Previous investigators have reported the use of natural language processing in automatic indexing of radiology reports and information extraction (13–16). Hersh et al (13) assessed a system that matched text to concepts in the Unified Medical Language System, or UMLS, metathesaurus for automatic indexing of radiology reports to develop clinical image repositories that can be used for patient care and medical education. The authors performed selective, automated indexing of findings and diagnoses in electronic radiology reports by using a natural language processing program, the Semantic and Probabilistic Heuristic Information Retrieval Environment, or SAPHIRE, system. However, the precision of detecting concepts was only 30% with this automated information retrieval system.
Happe et al (14) described a technique of automatic concept extraction from spoken medical records that had a very low rate (less than 3%) of indexing errors for coronary angiography reports. The use of a natural language processing system, the Radiology Text Report Analyzer and Classifier, or RadTRAC, for automated monitoring of chest radiograph reports to identify reports that described new or expanding neoplasms so that patient follow-up could be monitored has also been described (15). In a set of 470 reports evaluated with the RadTRAC system and with retrospective expert review of logbooks for classification of the reports, RadTRAC had a sensitivity of 90% and a specificity of 82%.
Friedman et al (16) have also reported the application of natural language processing for identification of clinical information in unstructured radiology reports and mapping that information into a structured representation containing clinical terms. Their processor provided three processing phases, all of which were driven by different knowledge sources. The first phase performed parsing by identifying the structure of the text through the use of a grammar that defined semantic patterns and a target form. The second phase, regularization, standardized the terms in the initial target structure through compositional mapping of multiple word phrases, and the third phase, encoding, mapped the terms to a controlled vocabulary. With training of the query component, automated encoding of impression sections of 230 radiology reports with the processor for occurrences of four diseases yielded recall and precision accuracies of 85% and 87%, respectively.
Likewise, the LEXIMER program was developed to apply the principles of information theory to analysis of an unstructured radiology report database by reducing entropy (or noise from report text) while preserving the outcome (signal) or positivity for measurement of FT and RT. Commonly, in an unstructured radiology report, there is limited diagnostic value to much of the verbiage, which represents a high entropy state. Indeed, only certain words used within each report carry the meaning and intent (signal), which represent the actual information.
During the development of the LEXIMER program, it was determined that each phrase must be assessed for its value and subsequently reduced to its root meaning (a process also known as stemming) with the help of a stemming algorithm, which we designed to have full control of the vocabulary used for stemming. Once this concept was envisioned, a simple method for phrase-level extraction was performed by using text parsing (to break text into smaller parts with punctuation-based phrase isolation through use of an internally developed parser) and syntactic algorithms (created to group phrases). After entropy or noise reduction, a second class of phrases that did not symbolize positive or negative findings was assessed to determine the recommendation for subsequent action in the report (RT).
At initial empirical inspection, it appeared that the location of a phrase weighed heavily on its importance (ie, FT"signal"). Statements contained within impressions, conclusions, or structured lists tended to summarize lesser statements and had a greater likelihood of high information content. Given that fact, at the time of phrase identification and extraction, phrase location was determined and priority was given to statements that appeared in summarizing locations of the report by processing these phrases for signal extraction in favor of those found in nonsummarizing locations. A computer algorithm was created with the C (computer) programming language to simulate the decisions performed by content experts. Knowledge and decision logic were represented through the use of Boolean trees (decision trees).
Decision-tree methods are useful when the data-mining task is the prediction of outcomes and the goal is to generate rules that can easily be translated into modern programming language logic. Decision trees are built through a process known as binary recursive partitioning. This is an iterative process of classifying the data according to the presence or absence of predictive variables (nodal decision terms). During formation of the LEXIMER decision trees, nodal decision terms were heuristically chosen through a manually intuitive selection of information terms that were reasoned to represent high signal within the domain set of radiology reports. Further programming logic was added to input a report from a text file, parse it into phrases, weigh the phrase by its location and value by using the hierarchic decision trees (containing 2154 nodes), determine its root classification, and label it as positive or recommending (signal classification). A controlled looping of the algorithm manages the overall document analysis and continues until all text has been parsed and processed.
The LEXIMER engine was initially trained through heuristics for 200 consecutive CT and MR imaging reports (not part of the validation set of reports in this study) and with known findings and recommendation classifications. Approximately 50 decision tree–optimizing iterations were performed while classification accuracy was monitored. The training set was then expanded to include all modalities through an additional training database of 180 reports from all imaging modalities, and 20 additional optimizing iterations were performed.
|
FOOTNOTES |
|---|
Authors stated no financial relationship to disclose.
Author contributions: Guarantors of integrity of entire study, K.J.D., J.H.T.; study concepts, K.J.D., M.K.K., J.H.T.; study design, K.J.D., M.K.K.; literature research, M.K.K., K.J.D.; clinical studies, K.J.D., M.K.K., M.M.M., B.A.A., T.S.; experimental studies, K.J.D., A.M.H., T.S.; data acquisition, M.K.K., M.M.M., A.M.H., B.A.A., T.S., K.J.D.; data analysis/interpretation, M.K.K., E.F.H.; statistical analysis, M.K.K., E.F.H.; manuscript preparation, K.J.D., M.K.K., M.M.M., J.H.T., E.F.H.; manuscript definition of intellectual content, K.J.D., M.K.K., J.H.T.; manuscript editing, K.J.D., M.K.K., J.H.T., M.M.M.; manuscript revision/review, K.J.D., M.K.K., M.M.M.; manuscript final version approval, all authors
|
REFERENCES |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODS RESULTSDISCUSSIONAPPENDIXREFERENCES |
|---|
This article has been cited by other articles:
|
|
 |
|
  P. A. Dang, M. K. Kalra, T. J. Schultz, S. A. Graham, and K. J. Dreyer Informatics in Radiology: Render: An Online Searchable Radiology Study Repository RadioGraphics, June 29, 2009; (2009) 295085036. [Abstract] [Full Text]
|
|
|
|
 |
|
 C. E. Kahn Jr. and D. L. Rubin Automated Semantic Indexing of Figure Captions to Improve Radiology Image Retrieval J. Am. Med. Inform. Assoc., May 1, 2009; 16(3): 380 - 386. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 D. L. Weiss and C. P. Langlotz Structured Reporting: Patient Care Enhancement or Productivity Nightmare? Radiology, December 1, 2008; 249(3): 739 - 747. [Full Text] [PDF]
|
|
|
|
 |
|
 P. A. Dang, M. K. Kalra, M. A. Blake, T. J. Schultz, E. F. Halpern, and K. J. Dreyer Extraction of Recommendation Features in Radiology with Natural Language Processing: Exploratory Study Am. J. Roentgenol., August 1, 2008; 191(2): 313 - 320. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
L. W. D'Avolio and A. A.T. Bui The Clinical Outcomes Assessment Toolkit: A Framework to Support Automated Clinical Records-based Outcomes Assessment and Performance Measurement Research J. Am. Med. Inform. Assoc., May 1, 2008; 15(3): 333 - 340. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
L. W. D'Avolio, M. S. Litwin, S. O. Rogers Jr., and A. A.T. Bui Facilitating Clinical Outcomes Assessment through the Automated Identification of Quality Measures for Prostate Cancer Surgery J. Am. Med. Inform. Assoc., May 1, 2008; 15(3): 341 - 348. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
G. P. Krestin, J. C. Miller, S. J. Golding, G. G. Frija, G. M. Glazer, H. G. Ringertz, and J. H. Thrall Reinventing Radiology in a Digital and Molecular Age: Summary of Proceedings of the Sixth Biannual Symposium of the International Society for Strategic Studies in Radiology (IS3R), August 25 27, 2005 Radiology, September 1, 2007; 244(3): 633 - 638. [Full Text] [PDF]
|
|
|
|
|
|
J. H. Thrall Building Research Programs in Diagnostic Radiology * Part III. Clinical and Translational Research Radiology, April 1, 2007; 243(1): 5 - 9. [Full Text] [PDF]
|
|
|
|
|
|
S. I. Lee, A. Saokar, K. J. Dreyer, J. B. Weilburg, J. H. Thrall, and P. F. Hahn Does Radiologist Recommendation for Follow-up with the Same Imaging Modality Contribute Substantially to High-Cost Imaging Volume? Radiology, March 1, 2007; 242(3): 857 - 864. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
B. de Bruijn, A. Cranney, S. O'Donnell, J. D. Martin, and A. J. Forster Identifying Wrist Fracture Patients with High Accuracy by Automatic Categorization of X-ray Reports J. Am. Med. Inform. Assoc., November 1, 2006; 13(6): 696 - 698. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. H. Thrall Reinventing Radiology in the Digital Age: Part II. New Directions and New Stakeholder Value Radiology, October 1, 2005; 237(1): 15 - 18. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH |
| RADIOLOGY | RADIOGRAPHICS | RSNA JOURNALS ONLINE |
