Facilitating Radiograph Interpretation: Refined Generative Models for Precise Bone Suppression in Chest X-rays

Chen Z., Sun Y., Ge R., Qin W., Pan C., Deng W., Liu Z., Min W., Elazab A., Wan X., Wang C.

Liu Y., Zeng F., Ma M., Zheng B., Yun Z., Qin G., Yang W., Feng Q.

Bone suppression is to suppress the superimposed bone components over the soft tissues within the lung area of Chest X-ray (CXR), which is potentially useful for the subsequent lung disease diagnosis for radiologists, as well as computer-aided systems. Despite bone suppression methods for frontal CXRs being well studied, it remains challenging for lateral CXRs due to the limited and imperfect DES dataset containing paired lateral CXR and soft-tissue/bone images and more complex anatomical structures in the lateral view. In this work, we propose a bone suppression method for lateral CXRs by leveraging a two-stage distillation learning strategy and a specific data correction method. Specifically, a primary model is first trained on a real DES dataset with limited samples. The bone-suppressed results on a relatively large lateral CXR dataset produced by the primary model are improved by a designed gradient correction method. Secondly, the corrected results serve as training samples to train the distillated model. By automatically learning knowledge from both the primary model and the extra correction procedure, our distillated model is expected to promote the performance of the primary model while omitting the tedious correction procedure. We adopt an ensemble model named MsDd-MAP for the primary and distillated models, which learns the complementary information of Multi-scale and Dual-domain (i.e., intensity and gradient) and fuses them in a maximum-a-posteriori (MAP) framework. Our method is evaluated on a two-exposure lateral DES dataset consisting of 46 subjects and a lateral CXR dataset consisting of 240 subjects. The experimental results suggest that our method is superior to other competing methods regarding the quantitative evaluation metrics. Furthermore, the subjective evaluation by three experienced radiologists also indicates that the distillated model can produce more visually appealing soft-tissue images than the primary model, even comparable to real DES imaging for lateral CXRs.

Rani G., Misra A., Dhaka V.S., Buddhi D., Sharma R.K., Zumpano E., Vocaturo E.

The high transmission rate of COVID-19 and the lack of quick, robust, and intelligent systems for its detection have become a point of concern for the public, Government, and health experts worldwide. The study of radiological images is one of the fastest ways to comprehend the infectious spread and diagnose a patient. However, it is difficult to differentiate COVID-19 from other pneumonic infections. The purpose of this research is to provide an automatic, precise, reliable, robust, and intelligent assisting system ‘Covid Scanner’ for mass screening of COVID-19, Non-COVID Viral Pneumonia, and Bacterial Pneumonia from healthy chest radiographs. To train the proposed system, the authors of this research prepared novel a dataset called, “COVID-Pneumonia CXR”. The system is a coherent integration of bone suppression, lung segmentation, and the proposed classifier, ‘EXP-Net’. The system reported an AUC of 96.58% on the validation dataset and 96.48% on the testing dataset comprising chest radiographs. The results from the ablation study prove the efficacy and generalizability of the proposed integrated pipeline of models. To prove the system's reliability, the feature heatmaps visualized in the lung region were validated by radiology experts. Moreover, a comparison with the state-of-the-art models and existing approaches shows that the proposed system finds clearer demarcation between the highly similar chest radiographs of COVID-19 and Non-COVID viral pneumonia. The copyright of “Covid Scanner” is protected with registration number SW-13625/2020. The code for the models used in this research are publicly available at: https://github.com/Ankit-Misra/multi_modal_covid_detection/ . • Preparing a benchmarking dataset, “COVID Pneumonia CXR”, comprising of bone suppressed and lung segmented CXRs. The dataset is validated by radiology experts. • Utilizing the potential of the Generative Adversarial Networks (GANs) and Convolutional Neural Networks (CNN) for developing an automatic system for COVID-19 screening from non-COVID viral pneumonia, bacterial pneumonia, and healthy using chest radiographs. • Coherent integration of bone suppression, lung segmentation, and cGAN based classifier with a user friendly web application to develop the precise, low-cost, reliable, convenient, robust, and intelligent assisting system ‘COVID Scanner’ for mass screening of COVID-19. • Analyzing the impact of augmentation, bone suppression and lung segmentation on the performance of classifier. • Identifying the loss function useful for resolving the problem of class imbalance. • Validating the reliability of the model by generating feature heatmaps using GradCam++. • Validation of the screening system by the Radiology Experts involved in this research.

Rani G., Misra A., Dhaka V.S., Zumpano E., Vocaturo E.

• Bone shadow suppression proposed in this research is important for reducing the training time and improving the reliability of deep learning based classification models. • Bone shadow suppression without distorting the spatial features of an image play an important role in precise training of a deep learning model. • GAN based architectures have potential to retain the image quality while reconstruction. • Employing task-specific auxiliary loss functions provide an opportunity to optimize the performance of the GAN based architecture. Background and Objective: Chest radiographs (CXR) are in great demand for visualizing the pathology of the lungs. However, the appearance of bones in the lung region hinders the localization of any lesion or nodule present in the CXR. Thus, bone suppression becomes an important task for the effective screening of lung diseases. Simultaneously, it is equally important to preserve spatial information and image quality because they provide crucial insights on the size and area of infection, color accuracy, structural quality, etc. Many researchers considered bone suppression as an image denoising problem and proposed conditional Generative Adversarial Network-based (cGAN) models for generating bone suppressed images from CXRs. These works do not focus on the retention of spatial features and image quality. The authors of this manuscript developed the Spatial Feature and Resolution Maximization (SFRM) GAN to efficiently minimize the visibility of bones in CXRs while ensuring maximum retention of critical information. Method: This task is achieved by modifying the architectures of the discriminator and generator of the pix2pix model. The discriminator is combined with the Wasserstein GAN with Gradient Penalty to increase its performance and training stability. For the generator, a combination of different task-specific loss functions, viz., L1, Perceptual, and Sobel loss are employed to capture the intrinsic information in the image. Result: The proposed model reported as measures of performance a mean PSNR of 43.588, mean NMSE of 0.00025, mean SSIM of 0.989, and mean Entropy of 0.454 bits/pixel on a test size of 100 images. Further, the combination of δ = 10 4 , α = 1 , β = 10 , and γ = 10 are the hyperparameters that provided the best trade-off between image denoising and quality retention. Conclusion: The degree of bone suppression and spatial information preservation can be improved by adding the Sobel and Perceptual loss respectively. SFRM-GAN not only suppresses bones but also retains the image quality and intrinsic information. Based on the results of student’s t -test it is concluded that SFRM-GAN yields statistically significant results at a 0.95 level of confidence and shows its supremacy over the state-of-the-art models. Thus, it may be used for denoising and preprocessing of images.

Rajaraman S., Cohen G., Spear L., Folio L., Antani S.

Automatic detection of some pulmonary abnormalities using chest X-rays may be impacted adversely due to obscuring by bony structures like the ribs and the clavicles. Automated bone suppression methods would increase soft tissue visibility and enhance automated disease detection. We evaluate this hypothesis using a custom ensemble of convolutional neural network models, which we call DeBoNet, that suppresses bones in frontal CXRs. First, we train and evaluate variants of U-Nets, Feature Pyramid Networks, and other proposed custom models using a private collection of CXR images and their bone-suppressed counterparts. The DeBoNet, constructed using the top-3 performing models, outperformed the individual models in terms of peak signal-to-noise ratio (PSNR) (36.7977±1.6207), multi-scale structural similarity index measure (MS-SSIM) (0.9848±0.0073), and other metrics. Next, the best-performing bone-suppression model is applied to CXR images that are pooled from several sources, showing no abnormality and other findings consistent with COVID-19. The impact of bone suppression is demonstrated by evaluating the gain in performance in detecting pulmonary abnormality consistent with COVID-19 disease. We observe that the model trained on bone-suppressed CXRs (MCC: 0.9645, 95% confidence interval (0.9510, 0.9780)) significantly outperformed (p < 0.05) the model trained on non-bone-suppressed images (MCC: 0.7961, 95% confidence interval (0.7667, 0.8255)) in detecting findings consistent with COVID-19 indicating benefits derived from automatic bone suppression on disease classification. The code is available at https://github.com/sivaramakrishnan-rajaraman/Bone-Suppresion-Ensemble.

Bae K., Oh D.Y., Yun I.D., Jeon K.N.

To compare the effects of bone suppression imaging using deep learning (BSp-DL) based on a generative adversarial network (GAN) and bone subtraction imaging using a dual energy technique (BSt-DE) on radiologists' performance for pulmonary nodule detection on chest radiographs (CXRs).A total of 111 adults, including 49 patients with 83 pulmonary nodules, who underwent both CXR using the dual energy technique and chest CT, were enrolled. Using CT as a reference, two independent radiologists evaluated CXR images for the presence or absence of pulmonary nodules in three reading sessions (standard CXR, BSt-DE CXR, and BSp-DL CXR). Person-wise and nodule-wise performances were assessed using receiver-operating characteristic (ROC) and alternative free-response ROC (AFROC) curve analyses, respectively. Subgroup analyses based on nodule size, location, and the presence of overlapping bones were performed.BSt-DE with an area under the AFROC curve (AUAFROC) of 0.996 and 0.976 for readers 1 and 2, respectively, and BSp-DL with AUAFROC of 0.981 and 0.958, respectively, showed better nodule-wise performance than standard CXR (AUAFROC of 0.907 and 0.808, respectively; p ≤ 0.005). In the person-wise analysis, BSp-DL with an area under the ROC curve (AUROC) of 0.984 and 0.931 for readers 1 and 2, respectively, showed better performance than standard CXR (AUROC of 0.915 and 0.798, respectively; p ≤ 0.011) and comparable performance to BSt-DE (AUROC of 0.988 and 0.974; p ≥ 0.064). BSt-DE and BSp-DL were superior to standard CXR for detecting nodules overlapping with bones (p < 0.017) or in the upper/middle lung zone (p < 0.017). BSt-DE was superior (p < 0.017) to BSp-DL in detecting peripheral and sub-centimeter nodules.BSp-DL (GAN-based bone suppression) showed comparable performance to BSt-DE and can improve radiologists' performance in detecting pulmonary nodules on CXRs. Nevertheless, for better delineation of small and peripheral nodules, further technical improvements are required.

Xie J.

Chest X-ray is one of the main methods for screening chest diseases, which has the characteristics of low radiation dose, fast imaging and low cost. In order to better assist doctors in disease diagnosis, usually the X-ray bone suppression and organ segmentation are performed. Many research progress has been made in this field, but the accuracy of the above two tasks is still limited due to the inherent characteristics of medical images. Firstly, the shape of organs of different individuals varies greatly.So there are inevitable segmentation errors if the overall shape is not perceived. Generally, the boundary of organs is fuzzy, so it is prone to misclassification near the boundary. In addition, existing bone suppression methods still can't completely remove bone shadows. In this paper, we propose a deep learning model whose overall architecture is designed based on the pix2pix network. This model generates both bone suppression images and organ segmentation images.Aiming at the above three issues, we make some improvements. We innovatively use the Transformer structure to enhance the attention to the global context and enhance the perception of the overall shape of the organ in the feature extraction process. Also, we design a new loss function, which gives larger weight to the position error near the organ boundary in the later stage of network training. This loss function pays more attention to edge information and helps determine the position of organ boundaries. We evaluate the effectiveness of the model on the X-ray image dataset, and compare it with the latest algorithm comprehensively. We also evaluate the effectiveness of our improvements through ablation study which shows that our improvements are effective.

Thamilarasi V., Roselin R.

Abstract Automatic image segmentation and classification of medical images plays significant role in detection and diagnosis of various pathological process. Normally chest radiography is a basic representation to find many abnormalities present in the chest. Radiology services delayed due to proper detection, segmentation and classification of diseases. Automatic segmentation and classification of medical images improved both pathological and radiological process. In recent days the deep learning with CNN methods provides remarkable successes in medical image diagnosis with in time limit and with minimum cost. The proposed method handles CNN for automatic classification of lung chest x-ray images as normal and up normal. Applying these modern techniques to lung chest x-ray images face more challenges while using small dataset. For testing JSRT dataset used which contains 247 images. Preeminent performance achieved using 180 images of nodule and non-nodule images. This method produce expected classification accuracy with the help of faster computation of CNN within fraction of seconds and attain 86.67% in classification accuracy.

Zhou Z., Zhou L., Shen K.

PURPOSE The purpose of this essay is to improve computer-aided diagnosis of lung diseases by the removal of bone structures imagery such as ribs and clavicles, which may shadow a clinical view of lesions. This paper aims to develop an algorithm to suppress the imaging of bone structures within clinical x-ray images, leaving a residual portrayal of lung tissue; such that these images can be used to better serve applications, such as lung nodule detection or pathology based on the radiological reading of chest x rays. METHODS We propose a conditional Adversarial Generative Network (cGAN) (Mirza and Osindero, Conditional generative adversarial nets, 2014.) model, consisting of a generator and a discriminator, for the task of bone shadow suppression. The proposed model utilizes convolutional operations to expand the size of the receptive field of the generator without losing contextual information while downsampling the image. It is trained by enforcing both the pixel-wise intensity similarity and the semantic-level visual similarity between the generated x-ray images and the ground truth, via optimizing an L-1 loss of the pixel intensity values on the generator side and a feature matching loss on the discriminator side, respectively. RESULTS The framework we propose is trained and tested on an open-access chest radiograph dataset for benchmark. Results show that our model is capable of generating bone-suppressed images of outstanding quality with a limited number of training samples (N = 272). CONCLUSIONS Our approach outperforms current state-of-the-art bone suppression methods using x-ray images. Instead of simply downsampling images at different scales, our proposed method mitigates the loss of contextual information by utilizing dilated convolutions, which gains a noticeable quality improvement for the outputs. On the other hand, our experiment shows that enforcing the semantic similarity between the generated and the ground truth images assists the adversarial training process and achieves better perceptual quality.

Li H., Han H., Li Z., Wang L., Wu Z., Lu J., Zhou S.K.

There is clinical evidence that suppressing the bone structures in Chest X-rays (CXRs) improves diagnostic value, either for radiologists or computer-aided diagnosis. However, bone-free CXRs are not always accessible. We hereby propose a coarse-to-fine CXR bone suppression approach by using structural priors derived from unpaired computed tomography (CT) images. In the low-resolution stage, we use the digitally reconstructed radiograph (DRR) image that is computed from CT as a bridge to connect CT and CXR. We then perform CXR bone decomposition by leveraging the DRR bone decomposition model learned from unpaired CTs and domain adaptation between CXR and DRR. To further mitigate the domain differences between CXRs and DRRs and speed up the learning convergence, we perform all the aboved operations in Laplacian of Gaussian (LoG) domain. After obtaining the bone decomposition result in DRR, we upsample it to a high resolution, based on which the bone region in the original high-resolution CXR is cropped and processed to produce a high-resolution bone decomposition result. Finally, such a produced bone image is subtracted from the original high-resolution CXR to obtain the bone suppression result. We conduct experiments and clinical evaluations based on two benchmarking CXR databases to show that (i) the proposed method outperforms the state-of-the-art unsupervised CXR bone suppression approaches; (ii) the CXRs with bone suppression are instrumental to radiologists for reducing their false-negative rate of lung diseases from 15% to 8%; and (iii) state-of-the-art disease classification performances are achieved by learning a deep network that takes the original CXR and its bone-suppressed image as inputs.

Eslami M., Tabarestani S., Albarqouni S., Adeli E., Navab N., Adjouadi M.

Chest X-ray radiography is one of the earliest medical imaging technologies and remains one of the most widely-used for diagnosis, screening, and treatment follow up of diseases related to lungs and heart. The literature in this field of research reports many interesting studies dealing with the challenging tasks of bone suppression and organ segmentation but performed separately, limiting any learning that comes with the consolidation of parameters that could optimize both processes. This study, and for the first time, introduces a multitask deep learning model that generates simultaneously the bone-suppressed image and the organ-segmented image, enhancing the accuracy of tasks, minimizing the number of parameters needed by the model and optimizing the processing time, all by exploiting the interplay between the network parameters to benefit the performance of both tasks. The architectural design of this model, which relies on a conditional generative adversarial network, reveals the process on how the well-established pix2pix network (image-to-image network) is modified to fit the need for multitasking and extending it to the new image-to-images architecture. The developed source code of this multitask model is shared publicly on Github as the first attempt for providing the two-task pix2pix extension, a supervised/paired/aligned/registered image-to-images translation which would be useful in many multitask applications. Dilated convolutions are also used to improve the results through a more effective receptive field assessment. The comparison with state-of-the-art algorithms along with ablation study and a demonstration video 1 are provided to evaluate the efficacy and gauge the merits of the proposed approach.

Jha D., Smedsrud P.H., Riegler M.A., Johansen D., Lange T.D., Halvorsen P., D. Johansen H.

Accurate computer-aided polyp detection and segmentation during colonoscopy examinations can help endoscopists resect abnormal tissue and thereby decrease chances of polyps growing into cancer. Towards developing a fully automated model for pixel-wise polyp segmentation, we propose ResUNet++, which is an improved ResUNet architecture for colonoscopic image segmentation. Our experimental evaluations show that the suggested architecture produces good segmentation results on publicly available datasets. Furthermore, ResUNet++ significantly outperforms U-Net and ResUNet, two key state-of-the-art deep learning architectures, by achieving high evaluation scores with a dice coefficient of 81.33%, and a mean Intersection over Union (mIoU) of 79.27% for the Kvasir-SEG dataset and a dice coefficient of 79.55%, and a mIoU of 79.62% with CVC-612 dataset.

Wang C., Chen D., Hao L., Liu X., Zeng Y., Chen J., Zhang G.

Chest X-ray film is the most widely used and common method of clinical examination for pulmonary nodules. However, the number of radiologists obviously cannot keep up with this outburst due to the sharp increase in the number of pulmonary diseases, which increases the rate of missed diagnosis and misdiagnosis. The method based on deep learning is the most appropriate way to deal with such problems so far. The main research in this paper was using inception-v3 transfer learning model to classify pulmonary images, and finally to get a practical and feasible computer-aided diagnostic model. The computer-aided diagnostic model could improve the accuracy and rapidity of doctors in the diagnosis of thoracic diseases. In this experiment, we augmented the data of pulmonary images, then used the fine-tuned Inception-v3 model based on transfer learning to extract features automatically, and used different classifiers (Softmax, Logistic, SVM) to classify the pulmonary images. Finally, it was compared with various models based on the original Deep Convolution Neural Network (DCNN) model. The experiment proved that the experiment based on transfer learning was meaningful for pulmonary image classification. The highest sensitivity and specificity are 95.41% and 80.09% respectively in the experiment, and the better pulmonary image classification performance was obtained than other methods.

Irvin J., Rajpurkar P., Ko M., Yu Y., Ciurea-Ilcus S., Chute C., Marklund H., Haghgoo B., Ball R., Shpanskaya K., Seekins J., Mong D.A., Halabi S.S., Sandberg J.K., Jones R., et. al.

Large, labeled datasets have driven deep learning methods to achieve expert-level performance on a variety of medical imaging tasks. We present CheXpert, a large dataset that contains 224,316 chest radiographs of 65,240 patients. We design a labeler to automatically detect the presence of 14 observations in radiology reports, capturing uncertainties inherent in radiograph interpretation. We investigate different approaches to using the uncertainty labels for training convolutional neural networks that output the probability of these observations given the available frontal and lateral radiographs. On a validation set of 200 chest radiographic studies which were manually annotated by 3 board-certified radiologists, we find that different uncertainty approaches are useful for different pathologies. We then evaluate our best model on a test set composed of 500 chest radiographic studies annotated by a consensus of 5 board-certified radiologists, and compare the performance of our model to that of 3 additional radiologists in the detection of 5 selected pathologies. On Cardiomegaly, Edema, and Pleural Effusion, the model ROC and PR curves lie above all 3 radiologist operating points. We release the dataset to the public as a standard benchmark to evaluate performance of chest radiograph interpretation models.

Zhou B., Lin X., Eck B., Hou J., Wilson D.

Dual-energy (DE) chest radiographs provide greater diagnostic information than standard radiographs by separating the image into bone and soft tissue, revealing suspicious lesions which may otherwise be obstructed from view. However, acquisition of DE images requires two physical scans, necessitating specialized hardware and processing, and images are prone to motion artifact. Generation of virtual DE images from standard, single-shot chest radiographs would expand the diagnostic value of standard radiographs without changing the acquisition procedure. We present a Multi-scale Conditional Adversarial Network (MCA-Net) which produces high-resolution virtual DE bone images from standard, single-shot chest radiographs. Our proposed MCA-Net is trained using the adversarial network so that it learns sharp details for the production of high-quality bone images. Then, the virtual DE soft tissue image is generated by processing the standard radiograph with the virtual bone image using a cross projection transformation. Experimental results from 210 patient DE chest radiographs demonstrated that the algorithm can produce high-quality virtual DE chest radiographs. Important structures were preserved, such as coronary calcium in bone images and lung lesions in soft tissue images. The average structure similarity index and the peak signal to noise ratio of the produced bone images in testing data were 96.4 and 41.5, which are significantly better than results from previous methods. Furthermore, our clinical evaluation results performed on the publicly available dataset indicates the clinical values of our algorithms. Thus, our algorithm can produce high-quality DE images that are potentially useful for radiologists, computer-aided diagnostics, and other diagnostic tasks.