HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Figures Only Full Text (PDF) AVI movie All Versions of this Article: 2401040018v1 240/1/230    most recent Submit a response Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Das, M. Articles by Silverman, S. G. Search for Related Content PubMed PubMed Citation Articles by Das, M. Articles by Silverman, S. G.

Augmented Reality Visualization for CT-guided Interventions: System Description, Feasibility, and Initial Evaluation in an Abdominal Phantom¹

¹ From the Dept of Radiology (M.D., U.J.S., E.v.S., S.G.S.) and Surgical Planning Laboratory (R.K.), Brigham and Women's Hosp, Harvard Medical School, Boston, Mass; Inst of Diagnostic and Interventional Radiology, Muelheim Radiology Inst, Univ of Witten Herdecke, Muelheim, Germany (M.D.); Dept of Diagnostic Radiology, RWTH Aachen, Aachen, Germany (M.D.); Siemens Corporate Research, Princeton, NJ (F.S., A.K., S.K.V.); Dept of Radiology, Medical Univ of South Carolina, 169 Ashley Ave, Charleston, SC 29425 (U.J.S.); CT Division, Siemens Medical Solutions, Forchheim, Germany (S.S.); and Dept of Radiology, Dana-Farber Cancer Inst, Harvard Medical School, Boston, Mass (E.v.S.). From the 2002 RSNA Annual Meeting. Received January 5, 2004; revision requested March 4; revision received April 30, 2005; accepted June 30; final version accepted September 6. Supported by an unrestricted research grant provided by Siemens Corporate Research, Princeton, NJ. M.D. is supported by the Medical Image Processing Laboratory, Muelheim/Ruhr, Germany. U.J.S. is a medical consultant to Siemens Medical Solutions. Address correspondence to U.J.S. (e-mail: schoepf{at}musc.edu).

	ABSTRACT

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION References

 
The purpose of this study was to evaluate the feasibility and performance of an augmented reality (AR) visualization prototype for virtual computed tomography (CT)-guided interventional procedures in a multimodality abdominal phantom. With the aid of AR guidance, three radiologists performed 30 attempts at targeting simulated liver lesions of different sizes (range, 5–15 mm) with a biopsy needle. The position of the needle tip relative to the lesion was verified by using ultrasonography and CT. With AR guidance, lesions were successfully targeted with the first needle pass in all cases. On the basis of these results, AR visualization for CT-guided intervention appears feasible and allows intuitive and accurate lesion targeting in a phantom.

Supplemental material: radiology.rsnajnls.org/cgi/content/full/2401040018/DC1

© RSNA, 2006

	INTRODUCTION

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION References

 
Computed tomography (CT)-guided interventions have become an integral component of modern patient care (1). Certain factors, however, make the targeting of lesions more difficult, prolong procedure times, and increase the radiation burden to the patient and operator (1–4). Such factors include spatial constraints within the narrow CT gantry opening, locations that are difficult to target, and an increasing number of small lesions. Image-guided navigation for minimally invasive surgery and stereotactic interventions has grown in importance in recent years (5,6).

One area of development is the use of augmented reality (AR) visualization to guide medical procedures (7–9). The AR system generates a composite view by augmenting the real interventional field viewed by the user with additional information from a virtual scene generated by the computer. In such a setting, AR is used to map cross-sectional medical images into patient space, thereby enabling the physician to perceive images in situ. For example, cross-sectional images that contain a tumor are projected into patient space and appear in the location of the actual tumor. By simultaneously tracking an invasive instrument, such as a biopsy needle, the AR system can project the instrument into the same space so that the operator can perform needle placement with AR guidance.

Since its initial description, AR for medical applications has been refined continuously (9–11). The basic underlying principles and technical feasibility of AR navigation have been described in conjunction with neurosurgical settings in the context of an interventional magnetic resonance (MR) imaging suite and with interventional ultrasonography (US) (12–15). The purpose of our study was to evaluate the feasibility and performance of an AR visualization prototype for virtual CT-guided interventional procedures in a multimodality abdominal phantom.

	MATERIALS AND METHODS

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION References

 
Authors who were not employees or consultants of Siemens Medical Solutions (Forchheim, Germany) and Siemens Corporate Research (Princeton, NJ) had control of the inclusion of any data and information that might have presented a conflict of interest for those authors who were employees or consultants of Siemens Medical Solutions and Siemens Corporate Research.

System Description
The system used in our study (RAMP; Siemens Corporate Research) is an investigational prototype that is not yet approved by the U.S. Food and Drug Administration. Its centerpiece is a video-see-through head-mounted display (Proview XL 35; Kaiser Electro-Optics, Carlsbad, Calif) that provides the user with augmented vision (Fig 1). Two miniature color video cameras (a scene camera [GP-KS 1000; Panasonic, Osaka, Japan] and a tracking camera [XC-77RR; Sony, Tokyo, Japan]) serve as the user's artificial eyes and capture live images of the scene. The total weight of the prototype helmet is 3 lbs (1.4 kg) (Fig 1).

View larger version (125K): [in this window] [in a new window] [Download PPT slide]   Figure 1: Video-see-through head-mounted display. Two visualization screens (striped arrow) allow unrestricted motion of the operator's head and allow exploration of the augmented scene from a variety of viewpoints. The camera triplet has a stereo pair of scene cameras (open arrows) to capture the scene and a dedicated tracking camera (solid arrow) with dedicated light-emitting diodes to generate a constant infrared flash.

View larger version (18K): [in this window] [in a new window] [Download PPT slide]   Figure 2: System block diagram of the components of the AR guidance setup. Tracking camera and scene camera are connected to a computer (PC), which projects its information to the head-mounted display (HMD) control and forms the visual input displayed in the stereo head-mounted display. XGA = 1024 x 768 graphics resolution.

View larger version (97K): [in this window] [in a new window] [Download PPT slide]   Figure 3: Workspace design shows optical markers (open arrows) attached to the frame (solid arrow) around the phantom. Optical markers reflect infrared light, which is tracked by the tracking camera on top of the head-mounted display shown in Figure 1.

View larger version (138K): [in this window] [in a new window] [Download PPT slide]   Figure 4: Optical markers (open arrows) are shown attached to biopsy needle (solid arrow).

Calibration
Baseline system calibration prior to performing a procedure is based on three-dimensional (3D) and two-dimensional point correspondence (16). The 3D coordinates of the markers are measured by using a commercially available stereo system (ARTrack1/DTrack; A.R.T., Herrsching, Germany). The two-dimensional positions are based on actual camera images. For calibration of the camera triplet, the position parameters of the two scene cameras are deduced from the position parameters measured with the tracking camera. The registration accuracy of the augmentation that is measured in the object space is 1 mm (16). The system is operated by a single computer and achieves real-time performance with a latency of 0.1 second, thereby generating stable augmentation. The AR video system uses a full standard video rate of 30 frames per second. Video and graphics are synchronized, which eliminates lag time between visualization of real and virtual objects. Thus, visualization of virtual objects occurs simultaneously with visualization of real objects. The time from system set up and calibration to the initiation of needle tracking is only 5–10 minutes.

Feasibility and Phantom Evaluation
We used a multimodality (CT, US, and MR imaging) interventional 3D abdominal phantom (CIRS, Norfolk, Va) to evaluate the performance of the AR system (Fig 3). The phantom contains multiple simulated liver lesions that range from 5–15 mm in size (average, 10 mm). The phantom was scanned by using a four-section CT scanner (Sensation 4; Siemens Medical Solutions) with a standard abdominal CT protocol (4 x 1-mm collimation, 120 kV, 150 mAs [effective], and 1.25-mm section thickness). The experimental setup in the scanner gantry is shown in Figure 5. Sections containing simulated lesions were identified and registered with respect to the phantom. Three-dimensional rendering was performed by the computer for the first 21 of 30 target lesions. To this end, the computer-generated 3D objects were inserted into the real-time video sequence in alignment with the actual target (Fig 6; Movie [radiology.rsnajnls.org/cgi/content/full/2401040018/DC1]).

View larger version (134K): [in this window] [in a new window] [Download PPT slide]   Figure 5: Experimental setup is shown with interventional phantom placed in the stereotactic frame (arrow) in the opening of the CT scanner gantry.

View larger version (117K): [in this window] [in a new window] [Download PPT slide]   Figure 6: Augmented scene, as viewed by the operator, demonstrates a thin blue-colored cylinder that represents the virtual needle. The tip of the virtual needle is identified by means of a small blue sphere at the end of the virtual needle. The thin green-colored cylinder represents the extrapolated needle trajectory extending from the needle tip. The virtual extrapolation of the needle trajectory is used as an aid to align the projected needle path with the segmented lesion center, which is rendered as a short yellow cylinder. By using the extrapolated trajectory, operators can locate an appropriate point of entry on the surface of the phantom and a suitable angle of insertion. The white grid superimposed on the outer perimeter of the interventional phantom provides additional depth cues. (See also Movie; radiology.rsnajnls.org/cgi/content/full/2401040018/DC1.)

Three radiologists with varying levels of experience in CT-guided interventional procedures (1, 6, and 15 years) each performed 10 needle placements (total needle placements, 30) to target the lesions in the phantom. Each of the radiologists tried to target the same lesions in the phantom. The time from needle insertion to successful lesion localization was recorded for each needle placement.

In the augmented scene viewed by the operator, a thin blue-colored cylinder was used to represent the virtual needle (Fig 6; Movie [radiology.rsnajnls.org/cgi/content/full/2401040018/DC1]), and a thin green-colored cylinder (7 cm in length) was used to represent the extrapolated needle trajectory extending from the needle tip (Fig 6; Movie [radiology.rsnajnls.org/cgi/content/full/2401040018/DC1]). The virtual extrapolation of the needle trajectory was used as an aid to align the projected needle path with the center of the lesion. By using the extrapolated trajectory, operators located an appropriate point of entry on the surface of the phantom and a suitable angle of insertion. The tip of the virtual needle was identified as a small sphere at the end of the colored cylinder. The depth and the position of the target lesion were extracted from the CT coordinates, and the time from needle insertion to successful lesion localization was recorded for each needle placement.

Once the lesion was targeted, as indicated by the appearance of the virtual needle tip in the segmented target, the needle was left inside the phantom, and the position of the needle tip was confirmed with US (Elegra; Siemens Medical Solutions) and repeat CT (Fig 7). The position of the actual needle tip within the phantom was extracted from the CT coordinates and was compared with the location of the virtual needle tip in the AR coordinate system. The distance from the actual or virtual needle tip to the actual or virtual lesion center was calculated in millimeters. On the basis of these values, three types of errors were assessed to gauge the performance of the AR system.

View larger version (75K): [in this window] [in a new window] [Download PPT slide]   Figure 7: Target verification with US on transverse view shows the needle (open arrow) with its tip in the cranial portion of the target lesion (solid arrow).

Real placement error was expressed as the distance (in millimeters) between the real target center and the real needle tip, as computed by using verification scans. Real placement error was calculated as a – b = c, where a is the real target center position, b is the real needle tip position, and c is the real placement error in millimeters. This type of error was used to measure the accuracy of actual needle placement relative to the center of the actual target lesion.

System error was expressed as the distance (in millimeters) between the real needle tip and the virtual needle tip and was calculated as b – x = d, where d is the accuracy of the AR system in millimeters. This type of error was used to measure the accuracy with which the AR system tracked the needle position.

Data Evaluation
The average target depth and the average time that was needed by the radiologists to target the lesion were calculated. Averages were also calculated by using CT coordinates for virtual placement error, real placement error, and system error.

	RESULTS

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION References

 
The average depth of the target lesion relative to the entry point of the biopsy needle was 68 mm (range, 42–88 mm). Use of the navigational device resulted in successful localization of the target lesion in all 30 interventions, as confirmed with US and CT. In each case, the needle tip had contact with at least a portion of the target lesion, with no deviation of the needle tip from the intended target after verification of the needle position with US and CT. Segmented and unsegmented lesions were successfully targeted. Mean targeting time was 8 seconds (range, 5–14 seconds) for successful localization of target lesions. There was no substantial difference in targeting time with respect to the experience of the operator.

In Figure 8, the three different placement errors are graphically displayed. The distance between the virtual needle tip and the virtual lesion center (virtual placement error) was 2.4 mm on average and ranged from 1.3 to 3.1 mm. The distance between the real needle and the real lesion (real placement error) was also extracted and showed an error of between 2.7 and 4.2 mm, with an average of 3.5 mm. The system error was measured as the distance between the virtual needle and the real needle and ranged from 1.1 to 3.0 mm, with an average of 2.1 mm.

View larger version (17K): [in this window] [in a new window] [Download PPT slide]   Figure 8: Mean error of needle placement relative to the center of the respective target lesion.

	DISCUSSION

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION References

Recent developments in medical image processing and enhanced computer performance have paved the way for the introduction of image-guided navigation into minimally invasive and stereotactic procedures (5–7). The technical basis and feasibility of AR have been described in conjunction with US (15) and MR imaging (12). The AR system described here merges virtual objects in real time with the real world and firmly anchors them in a video scene. AR visualization provides intuitive access to 3D anatomy and enables the operator to perceive cross-sectional medical images in situ. Guided by an augmented video view that displays the lesion, surrounding anatomy, and invasive instrument, the user can navigate the intervention tool in patient space. The AR system additionally provides an extrapolation of the needle trajectory as a targeting aid.

Image-guided navigation techniques have already been established for a variety of clinical applications (eg, neurosurgery and ear, nose, and throat surgery). However, to date, little use for this technique has been found in cross-sectional image-guided intervention, likely because the current CT-guided targeting techniques are satisfactory for the majority of clinical scenarios. Still, there are some aspects that may make AR navigation a helpful addition to the interventional armamentarium in the future. For example, concerns have been raised over increased radiation exposure to both patients and radiologists (3,4,17). The ability to navigate on the basis of previously acquired CT data allows the procedure to be performed without radiation exposure to the interventionalist and those outside the CT scanner gantry, which eliminates the spatial constraint of a narrow gantry opening. Additionally, navigation on the basis of CT data acquired during distinct phases of contrast material enhancement may help avoid vulnerable vascular structures and aid in the targeting of lesions that are only visible during certain perfusion phases.

On the basis of our phantom data, we determined that the ability to navigate the interventional tool along a virtual path within a volume of the CT image data greatly facilitates off-plane targeting access. Guiding information is provided as real-time graphic augmentation of a real-time video view with CT data containing the target. In this way, the most suitable access angle can be chosen. At the same time, the actual needle tip is indicated throughout the procedure, thereby enabling the selection of a needle path that avoids vessels or other vulnerable structures. Use of such techniques may be helpful whenever oblique access to the transverse plane is necessary to avoid vital structures. Thus, AR systems may provide better hand-eye coordination by means of 3D perception of navigational information (ie, lesion, needle, and critical structures) in the context of the patient than previous targeting devices that are based on two-dimensional concepts.

Performing minimally invasive procedures that are guided in real time by merged 3D renderings of actual and virtual images is an attractive concept. On the basis of results from our initial phantom experiments, the AR navigation system tested in this study proved intuitive enough to enable accurate lesion targeting, even for less-experienced users. A number of limitations, however, need to be addressed before the results of our initial phantom experiments can be successfully translated to patients.

Obviously, use of AR requires additional room time for system set up and calibration. However, even with this prototype system, these procedures take no longer than 10 minutes. Future iterations in system development and greater user familiarity with the device will result in even shorter setup times. Moreover, because there is a greater chance that the lesion will be targeted accurately with the first needle pass, the overall length of the procedure and the risk of complications from multiple needle passes may be reduced.

This first prototype does not account for patient movement, such as respiratory excursion with displacement of the lungs and upper abdominal organs. To allow for successful localization of moving targets, a continuous update of the position of the target lesion relative to the interventional field will be necessary to track the displacement of the anatomy and maintain accurate coregistration of virtual objects and image data. In the case of respiratory motion, strategies such as respiratory gating could compensate for excursions of abdominal organs (eg, CT-guided biopsies are sometimes performed in the liver dome or in other sites where US is less effective). Such procedures are technically more complex and difficult but were not tested with the simple phantom used here. Thus, it is foreseeable that the application of such tools will initially be reserved for cases with complicated but relatively fixed anatomy (eg, for deep retroperitoneal lymphadenopathy and adrenal, renal, or extraperitoneal pelvic lesions) rather than those for which organs could be in motion during needle placement.

While this system proved fairly accurate for targeting lesions in an interventional phantom and had only a few millimeters of error, this error rate may still be problematic when targeting complicated lesions (eg, a deep lymph node adjacent to the abdominal aorta).

In summary, our preliminary phantom experience with the prototype system shows that AR guidance can be successfully used to target small lesions during CT-guided interventions. Although the addition of AR navigation tools to the arsenal of image-guided instruments may broaden the scope of interventional radiology, extensive clinical testing will be needed.

	FOOTNOTES

 

Abbreviations: AR = augmented reality • 3D = three-dimensional

See Materials and Methods for pertinent disclosures.

Author contributions: Guarantors of integrity of entire study, M.D., U.J.S.; study concepts/study design or data acquisition or data analysis/interpretation, all authors; manuscript drafting or manuscript revision for important intellectual content, all authors; manuscript final version approval, all authors; literature research, all authors; experimental studies, all authors; statistical analysis, all authors; and manuscript editing, all authors

	References

TOP ABSTRACT INTRODUCTION MATERIALS AND METHODS RESULTS DISCUSSION References

 

1. Mueller PR, vanSonnenberg E. Interventional radiology in the chest and abdomen. N Engl J Med 1990;322:1364–1374.[Medline]
2. Silverman SG, Tuncali K, Adams DF, Nawfel RD, Zou KH, Judy PF. CT fluoroscopy-guided abdominal interventions: techniques, results, and radiation exposure. Radiology 1999;212:673–681.[Abstract/Free Full Text]
3. Nawfel R, Judy PF, Silverman SG, Hooton S, Tuncali K, Adams DF. Patient and personnel exposure during CT fluoroscopy-guided interventional procedures. Radiology 2000;216:180–184.[Abstract/Free Full Text]
4. Wagner LK. CT fluoroscopy: another advancement with additional challenges in radiation management. Radiology 2000;216:9–10.[Free Full Text]
5. Vannier MW, Marsh JL. Three-dimensional imaging, surgical planning and image-guided therapy. Radiol Clin North Am 1996;34:545–563.[Medline]
6. Vannier MW, Haller JW. Navigation in diagnosis and therapy. Eur J Radiol 1999;31:132–140.[CrossRef][Medline]
7. Cleary K, Freedman M, Clifford M, Lindisch D, Onda S, Jiang L. Image-guided robotic delivery system for precise placement of therapeutic agents. J Control Release 2001;74:363–368.[CrossRef][Medline]
8. Birkfellner W, Figl M, Huber K, et al. A head-mounted operating binocular for augmented reality visualization in medicine: design and initial evaluation. IEEE Trans Med Imaging 2002;21:991–997.[CrossRef][Medline]
9. Rosenthal M, State A, Lee J, et al. Augmented reality guidance for needle biopsies: an initial randomized, controlled trial in phantoms. Med Image Anal 2002;6:313–320.[CrossRef][Medline]
10. Grimson WE, Kikinis R, Jolesz FA, Black PM. Image-guided surgery. Sci Am 1999;280:62–69.[Medline]
11. Edwards PJ, Hawkes DJ, Hill DL, et al. Augmentation of reality using an operating microscope for otolaryngology and neurosurgical guidance. J Image Guid Surg 1995;1:172–178.[CrossRef][Medline]
12. Sauer F, Khamene A, Rubino GJ, et al. A head-mounted display system for augmented reality image guidance: towards clinical evaluation for iMRI-guided neurosurgery. In: MICCAI 2001, Lecture Notes in Computer Science 2208. New York, NY: Springer, 2001; 707–716.
13. Sauer F, Wenzel F, Bani-Hashemi A, et al. Augmented reality visualization of ultrasound images: system description, calibration and features. IEEE and ACM, ISAR 2001; 30–39.
14. Sauer F, Khamene A, Vogt S, et al. An augmented reality system for ultrasound guided needle biopsies. In: Westwood JD et al, eds. Medicine Meets Virtual Reality MMVR 2002. Amsterdam, the Netherlands: IOS Press, 2002; 455–460.
15. Stetten GD, Chib VS. Overlaying ultrasonographic images on direct vision. J Ultrasound Med 2001;20(3):235–240.[Abstract/Free Full Text]
16. Vogt S, Khamene A, Sauer F, Niemann H. Single camera tracking of marker clusters: multi-parameter cluster optimization and experimental verification. In: Proceedings of IEEE & ACM International Symposium on Mixed and Augmented Reality, ISMAR. Los Alamitos, Calif: Institute of Electrical and Electronics Engineers, 2002; 127–136.
17. Rogers LF. Serious business: radiation safety and radiation protection. AJR Am J Roentgenol 2001;177:1–10.[Free Full Text]

This Article Abstract Figures Only Full Text (PDF) AVI movie All Versions of this Article: 2401040018v1 240/1/230    most recent Submit a response Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Das, M. Articles by Silverman, S. G. Search for Related Content PubMed PubMed Citation Articles by Das, M. Articles by Silverman, S. G.