|
|
||||||||
Musculoskeletal Imaging |
1 From the Departments of Radiology (P.W.J.V., B.P.M.t.B., A.R.v.E., J.L.B.) and Medical Statistics (W.P.), Leiden University Medical Center, Albinusdreef 2, 2333 ZA Leiden, the Netherlands; Department of Radiology, Westeinde Hospital, The Hague, the Netherlands (T.P.W.d.R.); and Department of Radiology, Leyenburg Hospital, The Hague, the Netherlands (W.M.C.M.). Received April 26, 2001; revision requested June 9; revision received September 6; accepted October 16. Address correspondence to P.W.J.V. (e-mail: p.w.j.vincken@lumc.nl).
| ABSTRACT |
|---|
|
|
|---|
MATERIALS AND METHODS: In a prospective multicenter study, MR imaging was performed at 0.5 T in 430 consecutive patients. The sensitivity and specificity of MR imaging in the patients who underwent arthroscopy and the corrected sensitivity and specificity of MR in all the study patients were calculated. For this correction, patients with negative MR and arthroscopic results were considered representative of the patients with negative MR results who were conservatively treated, and the number of the former was doubled. The standard errors of the corrected values were adjusted with the
method.
RESULTS: At MR imaging, arthroscopy was indicated in 221 patients, 200 of whom underwent arthroscopy. Two hundred nine patients with negative MR imaging results were randomized for arthroscopic (105 patients) or for conservative treatment (104 patients). Of the 105 patients randomized for arthroscopy, 93 actually underwent arthroscopy. Arthroscopic treatment was necessary in 13 of 93 patients with a negative diagnosis at MR imaging. Arthroscopic treatment was necessary in 179 of 200 patients with a positive diagnosis at MR (sensitivity, 93.2%; specificity, 79.2%). Sensitivity and specificity corrected for randomization were 87.3% and 88.4%. Sensitivity and specificity corrected for randomization, respectively, were 84.1% and 94.2% for the diagnosis of medial meniscal tears and 69.5% and 94.5% for the diagnosis of lateral meniscal tears at MR.
CONCLUSION: MR imaging is an effective tool in the selection of patients for arthroscopy from among a general population.
© RSNA, 2002
Index terms: Knee, injuries, 452.4852, 452.4857 Knee, ligaments, menisci, and cartilage, 452.4852, 452.4857 Knee, MR, 452.121411, 452.121412, 452.121415
| INTRODUCTION |
|---|
|
|
|---|
The clinical relevance of MR imaging, however, is determined in one way by its value in the selection of patients for or exclusion of patients from treatment with therapeutic arthroscopy. This overall assessment of the entire joint, also called composite diagnosis (4), is more relevant than the accurate diagnosis of all specific lesions of the various anatomic structures.
Determination of the clinical relevance of MR imaging can be affected by selection bias. Selection criteria for arthroscopy, results of which are used as the reference standard, play a role in most studies and potentially have a major influence on the interpretation of MR imaging results.
The aim of this prospective multicenter study was to determine the effectiveness of MR imaging of the knee for appropriately identifying patients who require arthroscopic therapy from among those in whom there is a high clinical suspicion of internal derangements of the knee.
| MATERIALS AND METHODS |
|---|
|
|
|---|
Patients were included in the study at the departments of orthopedics or surgery of three unaffiliated hospitals (one university and two general hospitals) involved in this cooperative study. In all three hospitals, our protocol received approval of the institutional review board. The study was funded by the Dutch Insurance Council. Informed consent was obtained from 613 patients.
All 613 patients underwent a standardized physical examination and anteroposterior and lateral radiography of the knee. Radiographs were not used in the clinical assessment. On the basis of the standardized physical examination results, the orthopedic surgeon categorized each patient as having clinical findings highly suggestive of internal knee derangement requiring arthroscopic treatment (category 1) or as having no need of arthroscopic treatment (category 2). For this assessment, we used the criteria of the Dutch Orthopedic Society. These criteria are as follows: the presence of marked joint effusion or at least a "bulge sign" (ie, a visible bulge next to the patella caused by displacement of fluid) at physical examination; an extension deficit of at least 10°; a flexion deficit of at least 20°; instability at the varus and valgus stress test, Lachman test, anterior and posterior drawer test, and/or pivot test; at least one positive meniscal provocation test (McMurray, Apley, and squat tests); and atrophy of at least 2 cm relative to the contralateral leg measured 15 cm above the medial joint line. If at least one of these criteria was met (ie, the patient had clinical findings suggestive of knee derangement requiring arthroscopic treatment), the patient was included in the study.
Ultimately, 430 patients had a positive clinical diagnosis and were included. The patients had a mean age of 30.6 years; 299 (69.5%) of the patients were male.
Study Design
MR imaging was performed in all patients within 2 weeks after inclusion in the study. The subsequent course of each patients treatment was determined by the diagnosis at MR imaging (Fig 1). Patients with a positive MR imaging result (ie, those in whom arthroscopic treatment was indicated) proceeded to undergo arthroscopy. Patients with a negative MR result (ie, those in whom arthroscopic treatment was not indicated) were randomly assigned to treatment with one of two strategies. To this end we used random permuted tables. Half of these patients underwent arthroscopy; the other half were treated conservatively. Only those patients in whom arthroscopy was performed within 100 days of MR imaging were accepted for this study.
|
The standardized MR imaging protocol consisted of three sequences: a sagittal and a coronal dual spin-echo (SE) sequence and a sagittal T1-weighted three-dimensional gradient-echo sequence with frequency-selective fat suppression. The following parameters were used for both SE sequences: 140160-mm field of view and 20 and 80 msec echo times. The sagittal dual SE sequence had a repetition time of 2,350 msec, a matrix of 256 x 179, and a section thickness of 4 mm with a 0.4-mm intersection gap. The coronal dual SE sequence had a repetition time of 2,100 msec, a matrix of 256 x 205, and a section thickness of 5 mm with a 0.5-mm intersection gap. The parameters for the sagittal T1-weighted three-dimensional gradient-echo sequence with frequency-selective fat suppression were as follows: repetition time msec/echo time msec, 70/13; flip angle, 45°; field of view, 160 mm; matrix, 256 x 205; and section thickness, 4 mm with 2-mm overlap.
The total time it took to perform the MR imaging examination (including the initial survey sequence) was 26 minutes.
Interpretation of MR Images
One of six appointed radiologists (including T.P.W.d.R., W.M.C.M., and J.L.B.) with at least 4 years of experience with musculoskeletal MR imaging prospectively and individually evaluated the MR images in each patient. The radiologist was informed about the patients history and the findings at physical examination because we tried to mimic the normal clinical situation as much as possible. After the radiologist had recorded the findings on a standardized case record form, the radiologist characterized the composite diagnosis at MR (Table 1) according to the following four categories: category 1, normal MR study; category 2a, some findings of injury but arthroscopy not indicated; category 2b, equivocal findings at MR imaging, after which the final diagnosis and therefore the decision to perform arthroscopy was based on the assessment of an experienced panel; and category 3, major findings of injury, arthroscopy indicated. For patients in category 2b, the members of the panel (ie, the radiologist who evaluated the MR images and an orthopedic surgeon) decided, on the basis of the patients history and findings at physical examination and MR, whether the patient could potentially benefit from arthroscopic therapy (final positive diagnosis) or not (final negative diagnosis). Patients in the former group were selected for arthroscopy, whereas patients in the latter group were randomly selected for either arthroscopy or conservative treatment.
|
The anterior cruciate ligament (ACL) was considered normal when it appeared as a band of fibers of low to intermediate signal intensity on both sagittal and coronal dual SE images. The ACL was considered to be partially torn when there was abnormal signal intensity within the ligament or when otherwise intact fibers appeared wavy on sagittal or coronal dual SE images. The ACL was considered to be completely torn if there was disruption of all fibers or if it was not discernible at all on MR images (810). For statistical analysis, we considered normal and partially torn ligaments as one group and complete tears as another group.
Commonly accepted criteria were used to establish a diagnosis of other abnormalities such as ligamental tears (11) and bone bruises (12,13).
Arthroscopy
All arthroscopic examinations were videotaped and were performed in the three participating hospitals by an experienced orthopedic surgeon or by a resident supervised by an orthopedic surgeon. A total of 17 surgeons participated in the study. Just like the radiologist, the surgeon was informed of the patients history and of the findings at physical examination. The surgeon, however, was informed only of the diagnostic category at MR imaging, not the detailed MR diagnosis. The arthroscope, which had a 30° viewing angle, was introduced into the knee through an anterolateral or transpatellar portal. All structures were probed as well as visualized. After the diagnostic part of the examination, the arthroscopist recorded the arthroscopic diagnosis and therapeutic intentions, if any. To this end, a case record form was used that was identical to that used at the interpretation of the MR images. Subsequently, one of the authors (P.W.J.V. or B.P.M.t.B.), who was present at the arthroscopic examination, revealed the detailed diagnosis at MR imaging to the arthroscopist. In case of a discrepancy, the arthroscopist took a second look at the area during the same arthroscopic session. Next, depending on the diagnostic findings, the arthroscopist terminated the procedure or continued with the therapeutic part of the procedure.
Data Analysis
The composite diagnosis at MR imaging (Table 1) and the MR imaging diagnosis of injuries to individual structures were compared with the outcome of arthroscopy. Arthroscopic findings were considered positive when a therapeutic intervention was performed. Arthroscopic findings were considered negative if the procedure was terminated without arthroscopic treatment. For each individual structure, the diagnosis at MR imaging was compared with the diagnosis at arthroscopy.
Because patients with a negative MR imaging result were randomly selected for one of two treatments, and thus only half of the patients with a negative MR result underwent arthroscopy, we introduced a verification bias by artificially increasing the prevalence of MR imaging findings of injury in the patients whose MR results could be correlated with arthroscopic results. We calculated the sensitivity and specificity of MR imaging in the patients who underwent arthroscopy, and we calculated a corrected sensitivity and specificity for all patients who were included in the study to eliminate this verification bias. For this correction, we presumed the two randomized groups of patients to be equal, which would be true if the randomization was successful. Thus, the findings in the patients with a negative MR result who underwent arthroscopy are representative of the findings expected in patients with a negative MR result who underwent conservative treatment. These findings were doubled to enable the calculation of corrected sensitivity and specificity values for all patients included in the study. Negative and positive predictive values are not influenced by the randomization process.
Because of the introduction of additional uncertainty by doubling the number of patients with negative MR imaging results and arthroscopic correlation, we had to adjust the standard errors of the corrected sensitivity, specificity, and negative and positive predictive values by means of a standard statistical method (
method) (14); this process resulted in adjusted CIs.
| RESULTS |
|---|
|
|
|---|
Thirteen of these 221 patients refused arthroscopy; among them was one of the two patients who had equivocal signs of injury at MR imaging. Eight patients were excluded because the interval between MR imaging and arthroscopy was longer than 100 days. Therefore, data from 200 patients with a positive MR imaging result who underwent arthroscopy according to our protocol were included in our final analysis.
MR imaging results were negative in 209 (48.6%) of 430 patients. Of these 209 patients with a negative MR result, 105 were randomly selected for immediate arthroscopy. Of these 105 patients, 93 actually underwent arthroscopy according to protocol. Eleven patients refused arthroscopy, and one patient underwent arthroscopy more than 100 days after MR imaging.
Thus, ultimately, data from 293 patients were analyzed. The mean interval between MR imaging and arthroscopy for these 293 patients was 29.5 days (median, 28 days; range, 387 days).
In these 293 patients, the sensitivity of MR imaging for detecting composite knee injury was 93.2% (179 of 192), the specificity was 79.2% (80 of 101), and the accuracy was 88.4% (259 of 293). The sensitivity of MR imaging for detecting medial meniscal tears was 90.4% (122 of 135), the specificity was 92.4% (146 of 158), and the accuracy was 91.5% (268 of 293). The sensitivity of MR imaging for detecting lateral meniscal tears was 74.7% (65 of 87), the specificity was 92.7% (191 of 206), and the accuracy was 87.4% (256 of 293). The sensitivity of MR imaging for detecting ACL ruptures was 75.0% (27 of 36), the specificity was 93.8% (241 of 257), and the accuracy was 91.5% (268 of 293).
The sensitivity and specificity of the composite diagnosis at MR and of the diagnosis of injuries to individual structures, corrected for verification bias, are presented in Table 2.
|
Negative Composite Diagnosis at MR Imaging
Of the 93 arthroscopic procedures performed despite a negative MR imaging result, 80 remained purely diagnostic. In 13 patients, arthroscopy revealed a pathologic condition that was subsequently treated, indicating that the diagnosis at MR imaging was false-negative in these patients. Eight of these false-negative diagnoses were true mistakes at MR imaging; arthroscopy would not have been performed in these patients in clinical practice because of this false-negative diagnosis. The findings at arthroscopy that were not observed at MR imaging were medial meniscal tear (n = 3) and lateral meniscal tear (n = 5). Therapeutic arthroscopy was not indicated in the remaining five patients according to our protocol, but the arthroscopist decided to treat these patients anyway because of the following specific reasons: a medial meniscal tear smaller than 5 mm (missed at MR imaging, but by our definition not an indication for arthroscopy) in one patient, a lateral meniscal tear smaller than 5 mm (also diagnosed at MR) in one patient, displaced fibers of partial ACL tear in two patients (both partial tears were appreciated at MR imaging), and a thickened plica in the medial compartment in one patient.
Individual Structures
Of the 134 medial meniscal tears diagnosed at MR imaging, 122 were confirmed at arthroscopy. In six of the 12 patients with a false-positive diagnosis of medial meniscal tear at MR imaging, other pathologic findings led to therapeutic arthroscopy. In two patients, a tear smaller than 5 mm (not an indication for arthroscopy) that was seen on MR images was not confirmed at arthroscopy. In the remaining four patients, a false-positive diagnosis of meniscal tear at MR imaging would have led to unnecessary arthroscopy (Fig 2). On the other hand, arthroscopy revealed 13 tears in 159 medial menisci that were considered to be normal at MR. Of these additional 13 tears, only six were treated arthroscopically. In two of the six treated meniscal tears, arthroscopy was also indicated on the basis of other pathologic findings at MR imaging. Thus, only four false-positive diagnoses at MR imaging and four false-negative diagnoses at MR would have had clinical consequences. Corrected for verification bias, the sensitivity of MR imaging for detecting medial meniscal tears was 84.1% (122 of 145; these numbers are corrected for randomization); the specificity was 94.2% (227 of 241; these numbers are corrected for randomization).
|
|
|
|
In 30 (83%) of all 36 patients with arthroscopically proved complete ACL rupture, arthroscopy was indicated because of meniscal tear diagnosed at MR imaging (tear in medial meniscus, 56% [20 of 36]; tear in lateral meniscus, 14% [five of 36]; tear in both menisci, 14% [five of 36]). Four of the six arthroscopically proved isolated ACL ruptures were appreciated at MR imaging; one of these six ACLs was considered to be partially torn at MR.
The sensitivity, specificity, and negative and positive predictive values of MR imaging for the diagnosis of meniscal tear and complete ACL rupture are listed in Table 2.
| DISCUSSION |
|---|
|
|
|---|
The sensitivity of the composite diagnosis at MR imaging is somewhat higher than that of the diagnosis in individual structures, while the specificity of the composite diagnosis is somewhat lower than that of the diagnosis in individual structures. The higher sensitivity is explained by the fact that in the concept of composite diagnosis at MR imaging, signs of injury in more than one individual structure can lead to a positive MR result, as well as by the fact that injuries are often not isolated. On the other hand, if patients are selected for therapeutic arthroscopy on the basis of the composite diagnosis at MR imaging and subsequent arthroscopy confirms the presence of injury but the arthroscopist does not deem it necessary to treat the injury, in hindsight the selection for arthroscopy was not correct: specificity will be low relative to that for individual structures.
Thus, the concept of a clinically relevant composite diagnosis at MR imaging can help correct for the low sensitivity of MR imaging for individual structures. In our study, the corrected sensitivity for lateral meniscal tear was 69.5%. This low sensitivity of MR imaging for the diagnosis of lateral meniscal tear, especially when associated with ACL injury, is well known (3). However, because lateral meniscal tears were rarely isolated in our study, this low sensitivity was of little consequence; in 15 of the 22 lateral tears missed at MR imaging, arthroscopy was indicated anyway based on the presence of other injury.
The sensitivity of MR imaging for the detection of ACL ruptures, especially when corrected for randomization, was rather low in our study. This is probably secondary to the criteria used by the radiologist and orthopedic surgeon to distinguish between normal and partially and completely ruptured ACLs. Partial ruptures are especially difficult for the radiologist and the orthopedic surgeon to define in common terms. The arthroscopic definition of partial rupture is large and amorphous and ranges from the presence of some disrupted fibers to a subtotally ruptured ACL. Results could have been better if we had adopted a scoring system proposed by Rubin et al (1), which distinguishes between high- and low-grade injuries and thereby discards the diagnosis of partial rupture. The fact that there was only one false-negative case and one false-positive case in the distinction between a normal and a completely torn ACL supports the approach used by Rubin et al.
The discordance between the MR imaging findings and the arthroscopic findings of partial rupture is of little clinical importance because only a minority of complete ACL ruptures are isolated. Therefore, 30 of the 36 patients with complete ACL rupture in our study were assigned to undergo arthroscopy because of accompanying meniscal tears.
In our study, we did not consider an isolated ACL tear to be an indication for arthroscopy. In the Netherlands, an isolated rupture of a cruciate ligament is not considered to be an indication for arthroscopy in a general population (as opposed to ACL tear in professional athletes). Fewer than 1% of our patients were athletes who performed on a high competitive level and trained on a daily basis.
If an isolated complete ACL rupture had been considered to be an indication for arthroscopy in our study, the effect on our results would have been minimal. Only six complete ACL ruptures proved to be isolated at arthroscopy. Four of these were diagnosed at MR imaging as complete tears.
Our selection of patients on the basis of their history and the findings at physical examination increases the prevalence of pathologic conditions revealed at MR imaging relative to a situation without such a selection. The positive predictive value will be higher and the negative predictive value will be lower relative to those in an unselected population, although the effect on sensitivity and specificity is minor.
Including larger fractions of older and male patients in a study will also increase the prevalence of pathologic conditions and will thus affect MR imaging results. These population characteristics are reflected in the percentages of negative MR examination results in the various studies. Study design also has a major effect on MR imaging results. When MR imaging is used in the selection of patients for arthroscopy, as is the case in many retrospective studies, sensitivity and specificity will be substantially influenced because not all patients with a negative MR imaging result will undergo arthroscopy. This leads to a sensitivity that is overrated and a specificity that is underrated. The true sensitivity and true specificity of MR imaging can only be calculated when all patients with a negative MR imaging result undergo arthroscopy as well.
Bui-Mansfield et al (4) found a sensitivity of 94% and a specificity of 93% for composite diagnosis at MR imaging in a group of 50 patients selected on the basis of criteria related to surgical indications for monitoring appropriateness. In this group, results of 15 MR imaging examinations (30%) were considered negative. Correction for verification bias was not necessary because all patients underwent arthroscopy. The population consisted of predominantly male (90%) military personnel between 18 and 50 years of age with a higher prevalence of pathologic conditions relative to that in the population in our study. Therefore, the percentage of negative MR imaging results was rather low compared with that in our study. The sensitivity and specificity were better than in our study. However, the CIs for the calculated sensitivity and specificity values in the study by Bui-Mansfield et al were rather large because of the relatively small number of patients studied. Our bias-corrected sensitivity and specificity levels (Table 2) are well within their confidence intervals.
Ruwe et al (15) found that, in a group of 103 patients with clinical findings that necessitated diagnostic arthroscopy, 62 (60%) had negative MR imaging results. Forty-one percent of the patients, who were between 11 and 72 years of age, were female. This could explain why the percentage of negative MR imaging results was rather high compared with that in our study. Of the 62 patients with negative MR imaging results, only 10 underwent arthroscopy. A total of 44 patients underwent immediate arthroscopy. The sensitivity of 100% and specificity of 83% were thus very much influenced by verification biasthe sensitivity was overrated and the specificity was underrated; both are therefore not comparable with our results. Ruwe et al, however, used clinical outcome rather than arthroscopy as the standard against which MR imaging was compared, so there was no reason to correct for verification bias.
Rappeport et al (16) examined 47 patients between 19 and 54 years of age, 68% of whom were male, who were suspected of having intraarticular knee injuries. All 20 (43%) patients who had negative MR imaging results underwent arthroscopy; therefore there was no verification bias. This population was most comparable to our study population. Rappeport et al found a sensitivity of 86% and a rather low specificity of 65%. This low specificity in part explains the rather low percentage of negative MR imaging results (43%) in the patients studied compared with those in our study and in the study of Ruwe et al.
Other studies (1721) are difficult to compare with ours because they either did not include clearly described selection criteria, considered only individual structures without regard to therapeutic consequences, or did not define precisely which diagnoses at MR imaging indicated a need for arthroscopy.
The use of a 0.5-T system instead of a 1.5-T system deserves some attention. Several researches suggest that field strength is not an important determinant of diagnostic accuracy (2228). Only Fischer et al (29) found a statistically significant difference between a 0.35-T and a 1.5-T system, and then only in imaging of the medial meniscus. This study, however, was biased by the use of a more extensive scanning protocol with the higher-field-strength unit. It is therefore unlikely that field strength differences are important.
We conclude that, in a general population such as that described in this study, a composite diagnosis obtained at MR imaging after adequate clinical selection is accurate, despite the lower sensitivity of MR imaging for the diagnosis of injuries in individual knee structures. Therefore, the combination of a clinical examination and MR imaging is useful in selecting patients for arthroscopy.
| FOOTNOTES |
|---|
Author contributions: Guarantor of integrity of entire study, J.L.B.; study concepts and design, A.R.v.E., J.L.B.; literature research, B.P.M.t.B., P.W.J.V.; clinical studies, T.P.W.d.R., W.M.C.M.; data acquisition, B.P.M.t.B., P.W.J.V.; data analysis/interpretation, W.P., B.P.M.t.B., P.W.J.V.; statistical analysis, W.P., B.P.M.t.B., P.W.J.V.; manuscript preparation, P.W.J.V.; manuscript definition of intellectual content, J.L.B., B.P.M.t.B., P.W.J.V.; manuscript editing, P.W.J.V.; manuscript revision/review, J.L.B., T.P.W.d.R., P.W.J.V.; manuscript final version approval, J.L.B.
| REFERENCES |
|---|
|
|
|---|
This article has been cited by other articles:
![]() |
R. Crawford, G. Walley, S. Bridgman, and N. Maffulli Magnetic resonance imaging versus arthroscopy in the diagnosis of knee pathology, concentrating on meniscal lesions and ACL tears: a systematic review Br. Med. Bull., December 1, 2007; 84(1): 5 - 23. [Abstract] [Full Text] [PDF] |
||||
![]() |
P. W. J. Vincken, A. P. M. ter Braak, A. R. van Erkel, E. G. Coerkamp, T. P. W. de Rooy, S. de Lange, W. M. C. Mallens, L. N. J. E. M. Coene, R. M. Bloem, P. A. van Luijt, et al. MR Imaging: Effectiveness and Costs at Triage of Patients with Nonacute Knee Symptoms Radiology, December 1, 2006; 242(1): 85 - 93. [Abstract] [Full Text] [PDF] |
||||
![]() |
P. R. Kornaat, J. L. Bloem, R. Y. T. Ceulemans, N. Riyazi, F. R. Rosendaal, R. G. Nelissen, W. O. Carter, M.-P. Hellio Le Graverand, and M. Kloppenburg Osteoarthritis of the Knee: Association between Clinical Features and MR Imaging Findings. Radiology, June 1, 2006; 239(3): 811 - 817. [Abstract] [Full Text] [PDF] |
||||
![]() |
S. S. Boks, D. Vroegindeweij, B. W. Koes, M. G. M. Hunink, and S. M. A. Bierma-Zeinstra Follow-up of Posttraumatic Ligamentous and Meniscal Knee Lesions Detected at MR Imaging: Systematic Review Radiology, March 1, 2006; 238(3): 863 - 871. [Abstract] [Full Text] [PDF] |
||||
![]() |
E. H. G. Oei, J. J. Nikken, A. Z. Ginai, G. P. Krestin, J. A. N. Verhaar, A. B. van Vugt, and M. G. M. Hunink Acute Knee Trauma: Value of a Short Dedicated Extremity MR Imaging Examination for Prediction of Subsequent Treatment Radiology, January 1, 2005; 234(1): 125 - 133. [Abstract] [Full Text] [PDF] |
||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| RADIOLOGY | RADIOGRAPHICS | RSNA JOURNALS ONLINE |