| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Pediatric Imaging |
1 From the Department of Diagnostic Imaging (R.F., A.D., G.T., P.S.B., D.E.M., A.M.) and the Divisions of Neonatology (A.M.M., J.H., J.H.K.), Pathology (C.S.), and Pediatric General Surgery (T.G.), Hospital for Sick Children and University of Toronto, Toronto, Ontario, Canada. Received October 23, 2003; revision requested January 13, 2004; final revision received July 9; accepted August 4. Address correspondence to R.F., Department of Medical Imaging, Montreal Children’s Hospital, 2300 Tupper St, Montreal, QC, Canada H3H 1P3 (e-mail: ricardo.faingold@muhc.mcgill.ca).
 |
ABSTRACT |
|---|
 TOP ABSTRACT INTRODUCTIONMATERIALS AND METHODS RESULTSDISCUSSIONREFERENCES |
|---|
MATERIALS AND METHODS: This study was approved by the research ethics board, and informed consent was obtained. Sixty-two neonates enrolled in the prospective study underwent US of the bowel wall. Neonates were divided into two groups. Group A included 30 control subjects with gestational ages (GAs) ranging from 24 to 41 weeks. Group B included 32 neonates with GAs ranging from 24 to 40 weeks who were clinically proved to have or suspected of having NEC. All neonates in group B underwent abdominal radiography. Normative values were calculated in group A. In group B, the sensitivities and specificities of color Doppler US and abdominal radiography for detection of bowel necrosis were computed by using the modified Bell staging criteria for NEC as the reference standard.
RESULTS: Two neonates were excluded from group B; thus, a total of 60 neonates were included in the study. In group A, bowel wall thickness ranged from 1.1 to 2.6 mm. Bowel wall perfusion was detected with color Doppler US in all 30 neonates. Color Doppler signals ranged from one to nine dots per square centimeter. Twenty-two of 30 neonates in group B received a diagnosis of NEC. Mild to moderate NEC was diagnosed in 12 neonates. Color Doppler US depicted an isolated segment of bowel-absent blood flow in two neonates; this finding was confirmed with laparotomy. In 10 neonates with severe NEC, color Doppler US depicted isolated or multiple segments of bowel with absent perfusion. Pneumoperitoneum was present in only four neonates. The remaining eight neonates at risk for NEC had no evidence of loops without perfusion at color Doppler US. The sensitivity of free air at abdominal radiography as a positive sign for severe NEC with necrotic bowel was 40% compared with the 100% sensitivity of absence of flow at color Doppler US (P = .03).
CONCLUSION: Color Doppler US is more accurate than abdominal radiography in depicting bowel necrosis in NEC.
© RSNA, 2005
|
INTRODUCTION |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODS RESULTSDISCUSSIONREFERENCES |
|---|
Abdominal radiography has been the standard method for detecting and monitoring NEC. In the neonatal intensive care unit, the presence of pneumatosis intestinalis atabdominal radiography is almost always indicative of NEC, although neonates with Hirschsprung disease and intestinal obstruction may also have intramural air. However, just as the early radiographic findings of NEC are nonspecific, so are the advanced findings (1). Pneumoperitoneum is the only sign that has been universally agreed on as an indication for surgery or other interventions (1); however, this is not present in all babies with bowel necrosis and perforation.
A number of investigators have described the role of ultrasonography (US) in NEC (1,5,8–15). These investigators have evaluated pneumatosis intestinalis, bowel wall thickening, and portal venous gas. In addition, they have performed Doppler US of the superior mesenteric artery (SMA) and celiac trunk.
The evaluation of bowel wall blood flow with color Doppler US has been used to assess bowel ischemia in adults in only a few series (16–18) and to assess bowel ischemia in older pediatric populations in one retrospective study (19). Color Doppler US was also used to correlate intestinal mural vessel thickness with Crohn disease activity (20). To our knowledge, the evaluation of bowel wall perfusion with color Doppler US and the correlation of absence of blood flow with necrotic bowel in NEC have not previously been investigated in neonates. Thus, the purpose of our study was to prospectively determine whether the absence of bowel wall perfusion at color Doppler US in neonates with NEC is indicative of bowel necrosis.
|
MATERIALS AND METHODS |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODS RESULTSDISCUSSIONREFERENCES |
|---|
Group A consisted of 30 control neonates: 17 boys and 13 girls with a mean gestational age of 36.8 weeks (range, 24–41 weeks) and a mean corrected age of 39.9 weeks (range, 31–47 weeks). They were referred for abdominal US because of nongastrointestinal abnormalities such as antenatal hydronephrosis, multicystic dysplastic kidney, dysmorphic features, or urinary tract infection. None of the neonates had been fed for 3 hours before the US examinations, and none had clinical evidence of intestinal or cardiovascular disease.
Group B consisted of 32 preterm and term neonates who were clinically proved to have or suspected of having NEC. There were 19 boys and 13 girls with a mean gestational age of 32.7 weeks (range, 24–40 weeks). There were 23 preterm and nine term neonates, and their mean corrected age was 35 weeks (range, 25–46 weeks). All 32 neonates had undergone abdominal radiography before US as part of their clinical assessment. In neonates with NEC, the disease stage was determined by physicians in the neonatal intensive care unit according to the modified Bell staging criteria (3) before color Doppler US was performed. The examiner was not aware of the clinical stage or the findings at abdominal radiography.
The modified Bell staging criteria (3) are a clinical classification and management tool for NEC in which a composite of clinical signs and symptoms (eg, abdominal distention, bloody stools, or hypotension), biochemical parameters (eg, thrombocytopenia or neutropenia), and radiographic signs (eg, pneumatosis or pneumoperitoneum) is used to grade the severity and aid in the reporting of NEC.
Patients were excluded from the study if they had abdominal wall defects, large amounts of bowel gas, hemodynamic instability, mean blood pressure lower than gestational age (in weeks), heart rate less than 100 beats per minute or more than 200 beats per minute, severe metabolic acidosis with a pH level of less than 7.2, platelet count of less than 50 000 x 109/L, and oxygen saturation less than 90%. Infants undergoing high-frequency oscillation were also excluded.
The study was approved by the research ethics board at the Hospital for Sick Children. Written informed consent was obtained from the parents of the neonates in the control group. The research ethics board approved the color Doppler US examinations as part of the clinical work-up of bowel viability in the patients in group B, and parents gave oral informed consent.
Abdominal US
Abdominal US was performed with a Sequoia scanner (Acuson, Mountain View, Calif). All neonates in group B were monitored by a bedside nurse throughout the examination, which took approximately 25 minutes. An examination was stopped if any changes in vital signs and oxygen saturation levels occurred that met the exclusion criteria given earlier. In group A, neonates were monitored clinically. They were referred from an outpatient clinic and were not acutely ill. Therefore, they were monitored differently in that we did not record vital signs during the examination.
The US examination was divided into two parts. Initially, abdominal US was performed with an 8–5-MHz vector-array transducer to assess the relationship between the SMA and the superior mesenteric vein (SMV), SMA flow, portal venous gas, free fluid and its character, and free air. The SMA-SMV relationship was considered to be normal when the SMV was located to the right of the SMA on transverse sections. Peritoneal fluid was considered complex when echogenic material or septa were noted. Then, the bowel loops were evaluated with a 15–8-MHz linear-array transducer. The abdomen was divided topographically into four quadrants. All quadrants were interrogated with gray-scale and color Doppler US in real time by one examiner (R.F., 6 years of US experience) to standardize technique, and sample static images were obtained in each quadrant where the most loops were concentrated. The results of the examinations were then reported, immediately after they were downloaded from the portable unit to the picture archiving and communication system, by two of the four pediatric radiologists involved in the study (R.F., A.D., D.E.M., and P.S.B. with US experience of 6, 23, 14, and 15 years, respectively). Results were reached by consensus.
Gray-scale US evaluation included assessment of bowel wall echotexture for normal stratified bowel appearance or loss of "gut signature," measurement of wall thickness with calipers, and determination of the presence of intramural air and peristalsis. Peristalsis was assessed according to the presence or absence of bowel contractions during 1 minute. Color Doppler US was used to evaluate intestinal mural blood flow with a standard protocol and parameters including the lowest possible pulse repetition frequency without aliasing, a low wall filter, and the highest Doppler gain settings without flash artifacts. Velocity was set at 0.029–0.11 m/sec. In group A, mural perfusion was evaluated by counting the number of color Doppler signals as dots and lines per square centimeter. Color Doppler flow was determined to be present when color Doppler signals were reproducible or confirmed with pulsed Doppler waveforms. In group B, bowel loops were interrogated for the presence or absence of mural blood flow and compared with adjacent loops. When bowel loops without mural blood flow were identified, the settings were changed to more sensitive settings in an attempt to detect the slowest possible velocities down to 0.029 m/sec, below which only flash artifacts were obtained.
Abdominal Radiography
Abdominal radiography was performed with a computed radiographic technique in all group B neonates. Two views were obtained: a frontal and a horizontal beam view. Reports were made in a standard fashion by seven staff pediatric radiologists in our department (7–35 years of experience), and the images and the reports were available only after the results of the color Doppler examinations were reported.
Statistical Analysis
In group A, descriptive statistics were calculated for measurements derived from color Doppler US scans. Linear mixed-effects models, with a random effect for subject and a fixed effect for quadrant, were used to analyze wall thickness and the number of dots per square centimeter while accounting for correlations between measurements from quadrants in the same subject. These models made it possible to estimate overall means and the standard errors of the mean (SEMs) and to test for systematic differences between quadrants in these variables.
In group B, the sensitivities and specificities of color Doppler US and abdominal radiography were computed by using the modified Bell staging criteria for NEC (3) as the reference standard; the Wilson method was used to compute confidence intervals (CIs) for these measures.
The sensitivities and specificities of color Doppler US and abdominal radiography were compared by using an exact form of the McNemar test for paired proportions. The Fisher exact test was used to assess differences in gestational age and sex between groups A and B. A statistically significant difference was indicated by a P value of less than .05. All analyses were performed with S-Plus 2000 for Windows (Mathsoft, Seattle, Wash).
|
RESULTS |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODS RESULTSDISCUSSIONREFERENCES |
|---|
In group A, SMA flow was present and the SMA-SMV relationship was normal. There was no evidence of intramural gas, portal venous gas, free air, or ascites. (A trace of free fluid was found in 14 neonates.) Bowel wall thickness ranged from 1.1 to 2.6 mm (mean, 1.72 mm; SEM, 0.05). Normal neonatal bowel appearance (gut signature) showed a particularly prominent hypoechoic halo, which was thought to represent the muscularis propria (Fig 1a). There was no significant difference in mean thickness according to quadrant (P = .120). Peristalsis was present in all neonates, with a minimum of 10 contractions per minute in all quadrants.
|
|
|
Group B initially included 32 neonates who were proved to have or suspected of having NEC. Two neonates with clinically proved NEC were excluded because the gray-scale and color Doppler US scans were not interpretable owing to artifacts caused by a large amount of bowel gas. Thus, the final group B consisted of 30 neonates. Color Doppler US findings were compared with those from abdominal radiography in the remaining 30 neonates (93.7% of the cohort). SMA flow was present in all of these neonates.
Of the 30 neonates in group B, 22 received a diagnosis of NEC on the basis of the modified Bell staging criteria (3), for which a combination of clinical and abdominal radiographic findings are used (Table 1). The other eight neonates were at risk for NEC and bowel ischemia but did not develop any overt clinical or radiologic features of NEC.
|
|
|
|
|
|
|
|
|
|
|
|
|
The other 10 patients had severe NEC (Bell stage III); pneumoperitoneum was present at abdominal radiography in four. Color Doppler US depicted isolated or multiple segments of bowel with absent perfusion in all 10 neonates. In six patients, findings at pathologic examination (four laparotomies, two autopsies) helped confirm the presence of bowel necrosis that corresponded to the areas of absent perfusion. Of the other four patients, three had perforation with pneumoperitoneum at abdominal radiography and underwent placement of peritoneal drains. The fourth patient’s clinical condition deteriorated (no pneumoperitoneum was seen at abdominal radiography), and the patient died. Autopsy was not performed. Six of the 10 neonates with absent perfusion died.
In eight neonates, the severity of disease as assessed with the Bell staging system was changed due to the color Doppler US findings of necrotic bowel: In two patients the disease stage was reclassified from Bell stage II to IIIB, and in six patients the stage was reclassified from Bell stage IIIA to IIIB.
The other eight neonates in group B who were at risk for NEC had a nonspecific clinical picture. Four with complex congenital heart disease had increased lactate levels, and three with sepsis had coagulopathy. The eighth patient had pulmonary hypoplasia and malrotation and was the only neonate in our series with an abnormal SMA-SMV orientation. None of these eight neonates had features indicative of NEC at abdominal radiography, gray-scale US, or color Doppler US. There was no evidence of pneumatosis intestinalis. There was, however, mild bowel wall edema and free intraperitoneal fluid in these patients, including three with complex fluid (two with complex congenital heart disease and one with sepsis). Color Doppler US depicted bowel perfusion in all eight neonates. In four patients, bowel perfusion was normal, with perfusion in three patients being at the upper limits of normal: Two had sepsis without peritonitis, and one had peritonitis. In the four patients with complex congenital heart disease, there was a low flow state, with color Doppler signals depicted only when the color Doppler US parameters were set for detection of very slow flow (0.029–0.057 m/sec). Absence of necrotic bowel was confirmed at laparotomy in two patients and at autopsy in three who died of nongastrointestinal causes. The other three patients underwent at least 6 months of clinical follow-up with no evidence of gastrointestinal abnormality.
To compute measures of diagnostic performance, we defined the "disease" group (with necrotic bowel) to be the patients with Bell stage III and the "disease-free" group (no necrotic bowel) to be the patients without NEC and those with Bell stages I and II NEC. A positive finding at color Doppler US was considered to be any bowel loop showing absence of flow (Table 2); a negative finding at color Doppler US was all scanned loops showing some evidence of flow (Table 2). Sensitivity was 100% (10 of 10 patients; 95% CI: 72%, 100%), and specificity was 90% (18 of 20 patients; 95% CI: 70%, 97%). The positive predictive value was 83% (10 of 12 patients; 95% CI: 55%, 95%), and the negative predictive value was 100% (18 of 18 patients; 95% CI: 82%, 100%). The sensitivity of free air at abdominal radiography (indicative of bowel necrosis with perforation) as a positive sign for Bell stage III was 40% (four of 10 patients) (Table 1). This was significantly lower (P = .03) than the 100% sensitivity of absence of flow at color Doppler US.
|
|
DISCUSSION |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODS RESULTSDISCUSSIONREFERENCES |
|---|
The standardized color Doppler US protocol introduced in this study demonstrated that bowel wall perfusion can be detected accurately and reproducibly in the healthy infant with color Doppler US. The gray-scale US findings helped to establish normal data for bowel wall echotexture, thickness, and peristaltic activity.
In the second phase of this study, we attempted to evaluate bowel wall perfusion in group B. We ensured that each patient was stable enough to undergo the examination by monitoring the patient and adhering to the parameters of vital signs described earlier.
A variety of patterns of increased bowel wall perfusion were seen in the patients with Bell stages I and II NEC. Increased bowel wall perfusion was depicted as a marked increase in color Doppler signals in the bowel wall and/or mesentery and was interpreted as reflecting marked hyperemia due to vasodilatation. In some of these involved loops, the color Doppler signals formed a complete ring of color in the bowel wall. In others, the flow in the supplying subserosal mesenteric vessel and bowel wall had a Y-shaped appearance. In some loops, the color Doppler signals in the thickened, parallel valvulae conniventes formed a zebra pattern. These characteristic hyperemic patterns of flow were not seen in our control group or in the eight patients who were at risk for NEC and who had bowel wall thickening for reasons other than NEC. We believe that these patterns of hyperemia reflect increased perfusion to the bowel wall due to vasodilatation of mural vessels, including capillary congestion of the serosal surface. To our knowledge, these patterns have not been previously described with color Doppler US, and we suggest that they correlate with what has been described secondary to intestinal inflammation in pathologic specimens (21) and correspond to inflamed viable bowel in NEC.
The third feature, depicted with color Doppler US in 12 of our patients, was absence of color Doppler signals in the bowel wall, which was believed to reflect absence of perfusion to the involved loop. The absence of color Doppler signals correlated well with transmural bowel necrosis in histologic specimens obtained from laparotomy and autopsy or with pneumoperitoneum at abdominal radiography. One neonate who died of severe NEC did not undergo autopsy or have pneumoperitoneum. Color Doppler US enabled us to differentiate neonates with a single necrotic loop from those with multiple necrotic loops or diffuse bowel necrosis, and this differentiation allowed correlation with outcome in this series. Neonates in the former group had a good short-term outcome, whereas those in the latter group died within 24 hours of the study. Neither the patients in the control group nor the eight at risk for NEC had loops with absent color Doppler signals.
Our US findings match the pathologic stages of NEC. The hallmark finding of NEC is that of ischemic or coagulative necrosis. In the initial phases of NEC, an inflammatory process that leads to increased blood flow to the region occurs. This inflammatory process is reflected by the hyperemic appearances that we have demonstrated at color Doppler US. Bacteria penetrate the mucosal defense, and their byproducts of metabolism lead to the formation of intramural gas, which can be exquisitely depicted with gray-scale US. In the progression of NEC, platelet-activating factor produced by inflammatory cells and bacteria propagate the inflammatory cascade, mainly that of cytokines and complement leading to extensive transmural involvement (21,22). Eventually, there is compromise of the microvasculature such that ischemic changes to the tissue occur. Our color Doppler US findings of absent transmural blood flow are indicative of this later stage. Finally, the nonperfused bowel wall undergoes tissue thinning, which we have also shown at gray-scale US, and may perforate.
The prognosis for patients with NEC worsens once bowel perforation has occurred (6,7). The only universally accepted indication for a surgical procedure in NEC is the presence of perforation, which may manifest as free air and/or free fluid in the peritoneal cavity (1). Free air is traditionally diagnosed at abdominal radiography performed with a horizontal beam but may also be detected at US, as in the cases of four of our neonates with NEC. Perforation may, however, lead to an accumulation of free peritoneal fluid alone without free air (5). Although there are signs at abdominal radiography that may suggest the presence of free fluid, this is much more accurately depicted with US. The presence of intraperitoneal fluid at US, particularly if it is complex or echogenic, is suggestive of perforation (5). This finding, however, is nonspecific and may also be seen in intraperitoneal hemorrhage, with other causes of peritonitis, or in the presence of peritoneal dialysis catheters. Free intraperitoneal fluid was indeed present in three of our eight neonates at risk for NEC but with no evidence of bowel necrosis or perforation.
The results of our study and statistical analysis show that color Doppler US is more sensitive and specific than abdominal radiography in the detection of necrotic bowel in NEC. Of the 12 patients with absent color Doppler signals in single or multiple bowel loops, only four had pneumoperitoneum at abdominal radiography. Furthermore, of the remaining eight without pneumoperitoneum, three did not have complex free peritoneal fluid at US. The sensitivity of free air at abdominal radiography as a positive sign for severe NEC was 40% (four of 10 patients), which is significantly lower (P = .03) than the 100% (10 of 10 patients) sensitivity of absence of flow at color Doppler US. Furthermore, in eight neonates, the color Doppler US findings of necrotic bowel resulted in reclassification of the severity grade determined with the Bell staging system. The color Doppler US technique thus facilitated documentation of the presence of necrotic bowel, and the information provided assisted the surgical team and contributed to the more effective treatment of these ill infants.
Although the color Doppler US technique proved extremely valuable in this series of neonates who were suspected of having or proved to have NEC, our study had some limitations. Large amounts of bowel gas made evaluation of the bowel wall impossible, owing to artifacts, in two neonates. The number of neonates suspected of having or proved to have NEC was relatively small. This can be seen, for example, in the uncertainty of our estimate of positive predictive value for Bell stage III. Although the positive predictive value was 83% (10 of 12 patients), the relatively wide CI for this estimate (55%, 95%) suggests that a larger sample size would be needed to obtain a tighter CI. In addition, we did not evaluate interoperator variability. The same operator was involved in performing all of the gray-scale and color Doppler US examinations to ensure a standardized examination technique. It would have been unethical to subject these ill infants to repeat examinations performed by two blinded operators within a short period. Although the operator was blinded to the clinical stage, there was some bias due to apparent clinical status while performing the examination. Furthermore, there was a time interval between some of the US examinations and abdominal radiography in neonates with Bell stage I NEC—sometimes up to 12 hours. In all patients with Bell stages II and III, abdominal radiography was performed within less than 2 hours after US. In infants with low output states, the sensitivity of color Doppler US may need to be adjusted for detection of bowel wall perfusion; the use of parameters similar to those used with control subjects may not enable depiction of bowel wall perfusion.
Despite these limitations, US is an ideal modality for evaluating bowel necrosis because it is noninvasive, does not involve the use of ionizing radiation, can be performed readily at the bedside, and yields superior gray-scale and color Doppler US scans in infants who have a relative lack of subcutaneous and intraabdominal fat. The newer high-frequency linear-array transducers and Doppler technology facilitate imaging in this age group.
In summary, the results of this study show that color Doppler US findings are more accurate than clinical examination and abdominal radiography findings in the prediction of necrosis in neonates with NEC and therefore may alter clinical staging and management. The findings suggest that color Doppler US should be part of the standard evaluation of NEC, particularly in patients who do not respond to conventional medical treatment. Future studies with larger groups of patients are necessary to further evaluate this technique and to determine reproducibility and interoperator variability.
|
FOOTNOTES |
|---|
Authors stated no financial relationship to disclose.
Author contributions: Guarantor of integrity of entire study, R.F.; study concepts, R.F.; study design, R.F., A.D., P.S.B., D.E.M., J.H.K.; literature research, R.F., J.H.K.; clinical studies, R.F.; data acquisition, R.F., J.H.K., A.M.; data analysis/interpretation, D.E.M., R.F., C.S., G.T., A.D., A.M.M., P.S.B.; statistical analysis, G.T.; manuscript preparation, editing, revision/review, and final version approval, all authors; manuscript definition of intellectual content, R.F., A.D., G.T., J.H.K.
|
REFERENCES |
|---|
|
TOPABSTRACTINTRODUCTIONMATERIALS AND METHODS RESULTSDISCUSSIONREFERENCES |
|---|
This article has been cited by other articles:
|
|
 |
|
  C. A. Coursey, C. L. Hollingsworth, A. M. Gaca, C. Maxfield, D. DeLong, and G. Bisset III Radiologists' Agreement When Using a 10-Point Scale to Report Abdominal Radiographic Findings of Necrotizing Enterocolitis in Neonates and Infants Am. J. Roentgenol., July 1, 2008; 191(1): 190 - 197. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 G. B. Chavhan, D. A. Parra, A. Mann, and O. M. Navarro Normal Doppler Spectral Waveforms of Major Pediatric Vessels: Specific Patterns RadioGraphics, May 1, 2008; 28(3): 691 - 706. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. K. Saxena, A. Daneman, O. M. Navarro, M. Epelman, A. M. Moore, and J. T. Gerstle Significance of Absent Bowel Perfusion in Patients with Necrotizing Enterocolitis RadioGraphics, January 1, 2008; 28(1): 319 - 320. [Full Text] [PDF]
|
|
|
|
|
|
M. Epelman, A. Daneman, O. M. Navarro, I. Morag, A. M. Moore, J. H. Kim, R. Faingold, G. Taylor, and J. T. Gerstle Necrotizing Enterocolitis: Review of State-of-the-Art Imaging Findings with Pathologic Correlation RadioGraphics, March 1, 2007; 27(2): 285 - 305. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 N. Ueno, H. Kawamura, T. Hoshino, A. Kadowaki, and K. Nakamura Detection of alimentary tract hemorrhage on contrast-enhanced ultrasonography. J. Ultrasound Med., May 1, 2006; 25(5): 683 - 686. [Full Text] [PDF]
|
|
|
|
 |
|
 B. P. Wood Pediatric Imaging: Can We Improve Our Accuracy of Diagnosing NEC? AAP Grand Rounds, September 1, 2005; 14(3): 27 - 28. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
