viTBI-BERT: A Vietnamese Language Model for Prediction of Traumatic Brain Injury
Abstract
Artificial Intelligence (AI) is increasingly utilized for disease prediction by analyzing large and complex datasets. Traditionally, AI-based traumatic brain injury (TBI) prediction relies on multimodal data, including structured
information like test results and unstructured data such as CT and/or MRI scans. However, physician conclusions in text form at admission and discharge represent an underutilized source of valuable insights into a patient’s condition. This paper proposes a novel approach to classifying TBI severity using these physicians’ written conclusions. We introduce viTBI-BERT, a model built on the ViHealthBERT backbone, incorporating pre-processing techniques such as acronym normalization, special character removal, word segmentation, and textual data augmentation. Following pre-processing, the data is passed through the pre-trained ViHealthBERT model, augmented with a fully connected layer and a softmax layer for classification. Evaluated on three sets of medical notes from a self-collected dataset of 503 Vietnamese patients, viTBI-BERT achieved a sensitivity of 71% in classifying four levels of injury severity. These findings demonstrate the potential of language based analysis, which, when integrated with structured clinical and subclinical data, could further improve TBI classification outcomes.
References
H. Khalili, M. Rismani, M. A. Nematollahi, M. S. Masoudi, A. Asadollahi, R. Taheri, H. Pourmontaseri, A. Valibeygi, M. Roshanzamir, R.Alizadehsani et al., “Prognosis prediction in traumatic brain injurypatients using machine learning algorithms,” Scientific reports, vol.13,no.1,p.960,2023.
D. Yeboah, H. Nguyen, D. B. Hier, G. R. Olbricht, and T. Obafemi Ajayi, “A deep learning model to predict traumatic brain injury severity and outcome from mr images,” in 2021 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB), Oct 2021, pp. 1–6.
M. P. Nguyen, V. H. Tran, V. Hoang, T. D. Huy, T. H. Bui, and S. Q. H. Truong, “ViHealthBERT: Pre-trained Language Models for Vietnamese in Health Text Mining,” 13th Edition of its Language Resources and Evaluation Conference, 2022.
K. Matsuo, H. Aihara, T. Nakai, A. Morishita, Y. Tohma, and E. Kohmura, “Machine learning to predict in-hospital morbidity and mortality after traumatic brain injury,” Journal of Neurotrauma, vol. 37, no. 1, pp. 202–210, 2020. [Online]. Available: https://doi.org/10.1089/neu.2018.6276
R. Bruschetta, G. Tartarisco, L. F. Lucca, E. Leto, M. Ursino, P. Tonin, G. Pioggia, and A. Cerasa, “Predicting outcome of traumatic brain injury: Is machine learning the best way?” Biomedicines, vol. 10, no. 3, 2022. [Online]. Available: https: //www.mdpi.com/2227-9059/10/3/686
S. M. Adil, C. Elahi, D. N. Patel, A. Seas, P. I. Warman, A. T. Fuller, M. M. Haglund, and T. W. Dunn, “Deep learning to predict traumatic brain injury outcomes in the low-resource setting,” World Neurosurgery, vol. 164, pp. e8–e16, 2022.
M. Pease, D. Arefan, J. Barber, E. Yuh, A. Puccio, K. Hochberger, E. Nwachuku, S. Roy, S. Casillo, N. Temkin, D. O. Okonkwo, S. Wu, , N. Badjatia, Y. Bodien, A.-C. Duhaime, V. R. Feeser, A. R. Ferguson, B. Foreman, R. Gardner, S. Gopinath, C. D. Keene, C. Madden, M. McCrea, P. Mukherjee, L. B. Ngwenya, D. Schnyer, S. Taylor, and J. K. Yue, “Outcome prediction in patients with severe traumatic brain injury using deep learning from head ct scans,” Radiology, vol. 304, no. 2, pp. 385–394, 2022. [Online]. Available: https://doi.org/10.1148/radiol.212181
S. Ding, J. Ye, X. Hu, and N. Zou, “Distilling the knowledge from large-language model for health event prediction,” medRxiv, pp. 2024–06, 2024.
M. Jin, Q. Yu, D. Shu, C. Zhang, L. Fan, W. Hua, S. Zhu, Y. Meng, Z. Wang, M. Du et al., “Health-llm: Personalized
retrieval- augmented disease prediction system,” arXiv preprint arXiv:2402.00746, 2024.
L. Rasmy, Y. Xiang, Z. Xie, C. Tao, and D. Zhi, “Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction,” NPJ digital medicine, vol. 4, no. 1, p. 86, 2021.
S. Hegselmann, A. Buendia, H. Lang, M. Agrawal, X. Jiang, and D. Sontag, “Tabllm: Few-shot classification of tabular data with large language models,” in International Conference on Artificial Intelligence and Statistics. PMLR, 2023, pp. 5549–5581.
Z. Li, Y. Li, Q. Li, P. Wang, D. Guo, L. Lu, D. Jin, Y. Zhang, and Q. Hong, “Lvit: language meets vision transformer in medical image segmentation,” IEEE transactions on medical imaging, 2023.
C. Le Barbey, “Evaluation of artificial language model chatgpt4 to predict 6-month outcome after traumatic brain injury,” Ph.D. dissertation, 2023.
C. Vo, T. Cao, and B. Ho, “Abbreviation detection in vietnamese clinical texts,” VNU Journal of Science: Computer Science and Communication Engineering, vol. 34, no. 2, 2018. [Online]. Available: //jcsce.vnu.edu.vn/index.php/jcsce/article/view/211
A. Tabassum and D. R. R. Patil, “A survey on text pre-processing & feature extraction techniques in natural language processing,” 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID: 235211496
D. Q. Nguyen, D. Q. Nguyen, T. Vu, M. Dras, and M. Johnson, “A fast and accurate vietnamese word segmenter,” CoRR, vol. abs/1709.06307, 2017. [Online]. Available: http://arxiv.org/abs/1709. 06307
T. Vu, D. Q. Nguyen, D. Q. Nguyen, M. Dras, and M. Johnson, “VnCoreNLP: A Vietnamese natural language processing toolkit,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations. Association for Computational Linguistics, 2018, pp. 56–60. [Online]. Available: https://aclanthology.org/N18-5012
J. Wei and K. Zou, “EDA: Easy data augmentation techniques for boosting performance on text classification tasks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, nov 2019, pp. 6382–6388. [Online]. Available: https://aclanthology.org/D19-1670
T. P. Nguyen, V.-L. Pham, H.-A. Nguyen, H.-H. Vu, N.-A. Tran, and T.-T.-H. Truong, “A two-phase approach for building Vietnamese WordNet,” in Proceedings of the 8th Global WordNet Conference (GWC). Global Wordnet Association, 27–30 2016, pp. 261–266. [Online]. Available: https://aclanthology.org/2016.gwc-1.38
M. Xia, X. Kong, A. Anastasopoulos, and G. Neubig, “Generalized data augmentation for low-resource translation,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, jul 2019, pp. 5786–5796. [Online]. Available: https://aclanthology.org/P19-1579
A. Sugiyama and N. Yoshinaga, “Data augmentation using back-translation for context-aware neural machine translation,” in Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019). Association for Computational Linguistics, nov 2019, pp. 35–44. [Online]. Available: https://aclanthology.org/D19-6504
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in North American Chapter of the Association for Computational Linguistics, 2019. [Online]. Available: https: //api.semanticscholar.org/CorpusID:52967399
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2023. [Online]. Available: https://arxiv.org/abs/1706.03762
M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” in International conference on machine learning. PMLR, 2017, pp. 3319–3328.
Z. Zhao, R. Anand, and M. Wang, “Maximum relevance and mini mum redundancy feature selection methods for a marketing machine learning platform,” in 2019 IEEE International Conference on Data Science and Advanced Analytics (DSAA), 2019, pp. 442–452.
F. Gaillard, “Rotterdam CT score of traumatic brain injury Radiology Reference Article | Radiopaedia.org — radiopaedia.org,” https://radiopaedia.org/articles/rotterdam-ct-score-of-traumatic-brain-injury.
N. Kokhlikyan, V. Miglani, M. Martin, E. Wang, B. Alsallakh, J. Reynolds, A. Melnikov, N. Kliushkina, C. Araya, S. Yan, and O. Reblitz-Richardson, “Captum: A unified and generic model interpretability library for pytorch,” 2020.
Q. T. Nguyen, T. L. Nguyen, N. H. Luong, and Q. H. Ngo, “Fine-tuning BERT for sentiment analysis of vietnamese reviews,” CoRR, vol. abs/2011.10426, 2020. [Online]. Available: https://arxiv.org/abs/2011.10426
D. Q. Nguyen and A. T. Nguyen, “Phobert: Pre-trained language models for vietnamese,” CoRR, vol. abs/2003.00744, 2020. [Online]. Available: https://arxiv.org/abs/2003.00744
L. Phan, T. Dang, H. Tran, T. H. Trinh, V. Phan, L. D. Chau, and M.-T. Luong, “Enriching biomedical knowledge for low-resource language through large-scale translation,” 2022. [Online]. Available: https://arxiv.org/abs/2210.05598
