Integrating Self-Supervised Learning with Nonlinear Classifiers in Lightweight Swin Transformer for X-Ray Image Classification
Abstract
In this paper, we present a new approach about the integration Self-Supervised Learning with nonlinear
Classifiers in Lightweight Swin Transformer (SSLnC-LSwinT) for improving performance of X-ray image classification. Our approach leverages unlabeled data to address the issue of labeled data scarcity in the medical field by using self supervised learning (SSL) to extract features. One of our key contributions is the introduction of the Lightweight SwinT architecture, a more lightweight variant of SwinT, designed to enhance computational efficiency, reduce model complexity, and shorten training time. To further improve classification efficiency, we propose the integration of a nonlinear classifier instead of a linear classifier in Lightweight SwinT. The experimental results underscore our contributions, demonstrating significant reductions in model training time and notable improvements in classification performance. Our proposed method, which integrates SSL based on LSwinT with a nonlinear LightGBM classifier, achieves an accuracy of up to 87%, improving by 1.8% over the non-LightGBM SwinT version and reducing training time significantly (3:23:00 vs. 7:37:29) compared to the original SwinT architecture.
References
S. K. Zhou, H. Greenspan, and D. Shen, Deep learning for medical image analysis. Academic Press, 2023.
K. Sailunaz, T. ¨ Ozyer, J. Rokne, and R. Alhajj, “A survey of machine learning-based methods for covid-19 medical image analysis,” Medical & Biological Engineering & Computing, vol. 61, no. 6, pp. 1257–1297, 2023.
A. Dosovitskiy et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” 2021. [Online]. Available: https://arxiv.org/abs/2010.11929
Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” 2021. [Online]. Available: https://arxiv.org/abs/2103.14030
K. He, C. Gan, Z. Li, I. Rekik, Z. Yin, W. Ji, Y. Gao, Q. Wang, J. Zhang, and D. Shen, “Transformers in medical
image analysis,” Intelligent Medicine, vol. 3, no. 1, pp. 59 78, 2023.
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9729–9738.
X. Chen, H. Fan, R. Girshick, and K. He, “Improved base lines with momentum contrastive learning,” arXiv preprint arXiv:2003.04297, 2020.
X. Chen, S. Xie, and K. He, “An empirical study of training self-supervised vision transformers,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 9640–9649.
Z. Xie, Y. Lin, Z. Yao, Z. Zhang, Q. Dai, Y. Cao, and H. Hu, “Self-supervised learning with swin transformers,”
[Online]. Available: https://arxiv.org/abs/2105.04553
S.-C. Huang, A. Pareek, M. Jensen, M. P. Lungren, S. Yeung, and A. S. Chaudhari, “Self-supervised learning for medical image classification: a systematic review and implementation guidelines,” NPJ Digital Medicine, vol. 6, no. 1, p. 74, 2023.
M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in
self-supervised vision transformers,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 9650–9660.
H. Sowrirajan, J. Yang, A. Y. Ng, and P. Rajpurkar, “Moco pretraining improves representation and transferability of chest x-ray models,” in Medical Imaging with Deep Learning. PMLR, 2021, pp. 728–744.
K. Cho et al., “Chess: Chest x-ray pre-trained model via self supervised contrastive learning,” Journal of Digital Imaging, vol. 36, no. 3, pp. 902–910, 2023.
P. Rajpurkar and et al., “Deep learning for chest radiograph diagnosis: A retrospective comparison of the chexnext algorithm to practicing radiologists,” PLoS medicine, vol. 15, no. 11, p. e1002686, 2018.
K.-C. Chen and et al., “Diagnosis of common pulmonary diseases in children by x-ray images and deep learning,” Scientific reports, vol. 10, no. 1, p. 17374, 2020.
Y. Jin, H. Lu, W. Zhu, and W. Huo, “Deep learning based classification of multi-label chest x-ray images via dual weighted metric loss,” Computers in Biology and Medicine, vol. 157, p. 106683, 2023.
M. Nahiduzzaman and et al., “Parallel cnn-elm: A multiclass classification of chest x-ray images to identify seventeen lung diseases including covid-19,” Expert Systems with Applications, vol. 229, p. 120528, 2023.
M. Kaya and M. Eris, “D3senet: A hybrid deep feature extraction network for covid-19 classification using chest x-ray images,” Biomedical signal processing and control, vol. 82, p. 104559, 2023.
T.-T. Vo and T.-N. Do, “Improving chest x-ray image classification via integration of self-supervised learning and machine learning algorithms.” Journal of Information & Communication Convergence Engineering, vol. 22, no. 2, 2024.
Z. Ullah, M. Usman, and J. Gwak, “Mtss-aae: Multi-task semi-supervised adversarial autoencoding for covid-19 detection based on chest x-ray images,” Expert Systems with Applications, vol. 216, p. 119475, 2023.
S. U. Amin, S. Taj, A. Hussain, and S. Seo, “An automated chest x-ray analysis for covid-19, tuberculosis, and pneumonia employing ensemble learning approach,” Biomedical Signal Processing and Control, vol. 87, p. 105408, 2024.
S. Taslimi, S. Taslimi, N. Fathi, M. Salehi, and M. H. Rohban, “Swinchex: Multi-label classification on chest x-ray images with transformers,” arXiv preprint arXiv:2206.04246, 2022. [Online]. Available: https://arxiv.org/abs/2206.04246
G. I. Okolo, S. Katsigiannis, and N. Ramzan, “Ievit: An enhanced vision transformer architecture for chest x-ray image classification,” Computer Methods and Programs in Biomedicine, vol. 226, p. 107141, 2022.
J. Ko, S. Park, and H. G. Woo, “Optimization of vision transformer-based detection of lung diseases from chest x-ray images,” BMC Medical Informatics and Decision Making, vol. 24, no. 1, p. 191, 2024.
P. Komorowski, H. Baniecki, and P. Biecek, “Towards evaluating explanations of vision transformers for medical imaging,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 3726 3732.
L. Huang, J. Ma, H. Yang, and Y. Wang, “Research and implementation of multi-disease diagnosis on chest x-ray based on vision transformer,” Quantitative Imaging in Medicine and Surgery, vol. 14, no. 3, p. 2539, 2024.
C. C. Ukwuoma et al., “A hybrid explainable ensemble transformer encoder for pneumonia identification from chest x-ray images,” Journal of Advanced Research, vol. 48, pp. 191–211, 2023.
B. VanBerlo, J. Hoey, and A. Wong, “A survey of the impact of self-supervised pretraining for diagnostic tasks
in medical x-ray, ct, mri, and ultrasound,” BMC Medical Imaging, vol. 24, no. 1, p. 79, 2024.
E. Tiu, E. Talius, P. Patel, C. P. Langlotz, A. Y. Ng, and P. Rajpurkar, “Expert-level detection of pathologies from
unannotated chest x-ray images via self-supervised learning,” Nature Biomedical Engineering, vol. 6, no. 12, pp. 1399 1406, 2022.
M. Shakouri, F. Iranmanesh, and M. Eftekhari, “Dino-cxr: A self supervised method based on vision transformer for chest x-ray classification,” in International Symposium on Visual Computing. Springer, 2023, pp. 320–331.
J. Yao, X. Wang, Y. Song, H. Zhao, J. Ma, Y. Chen, W. Liu, and B. Wang, “Eva-x: A foundation model for
general chest x-ray analysis with self-supervised learning,” 2024. [Online]. Available: https://arxiv.org/abs/2405.05237
G. Li, R. Togo, T. Ogawa, and M. Haseyama, “Covid-19 detection based on self-supervised transfer learning using chest x-ray images,” International Journal of Computer Assisted Radiology and Surgery, vol. 18, no. 4, pp. 715 722, 2023.
J.-B. Grill et al., “Bootstrap your own latent: A new approach to self-supervised learning,” 2020. [Online].
Available: https://arxiv.org/abs/2006.07733
G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, “Lightgbm: A highly efficient gradient
boosting decision tree,” Advances in neural information processing systems, vol. 30, 2017.
T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 2016, pp. 785–794.
L. Prokhorenkova, G. Gusev, A. Vorobev, A. V. Dorogush, and A. Gulin, “Catboost: unbiased boosting with categorical features,” Advances in neural information processing systems, vol. 31, 2018.
F. Pedregosa et al., “Scikit-learn: Machine learning in python,” the Journal of machine Learning research, vol. 12, pp. 2825–2830, 2011.
A. Paszke et al., “Pytorch: An imperative style, high performance deep learning library,” 2019. [Online].
Available: https://arxiv.org/abs/1912.01703
