DaNangVMD: Vietnamese Speech Mispronunciation Detection
DaNangVMD: Nhận diện phát âm sai tiếng Việt
Abstract
Automatic Speech Recognition, also known as ASR, has grown exponentially over the past decade and is used
to recognize and translate human speech into readable text automatically. However, Vietnamese Speech Recognition faces critical challenges such as frequent mispronunciations as well as a huge variant in Vietnamese speech. In this work, we dive into the difficult challenge of Mispronunciation Detection (MD) in the Vietnamese language. As such a tonal language, Vietnamese is not only based on consonants and vowels but also on variations in pitch or tone during pronunciation. In this paper, we propose DaNangVMD model for detecting mispronunciations in Vietnamese speech based on the audio speech and canonical transcript. By leveraging multi-head attention-based multimodal representation from the embeddings of the phonetic encoder and linguistic encoder, DaNangVMD aims to provide a robust solution for accurate mispronunciation
detection and diagnosis. Throughout the extensive evaluation, the proposed DaNangVMD exhibits superior performances rather than that of the PAPL baseline models by 15% in F1 score and 13% in accuracy.
References
Wai-Kim Leun. “CNN-RNN-CTC BASED ENDTO-END MISPRONUNCIATION DETECTION AND DIAGNOSIS.” Department of Systems Engineering and Engineering Management, https://www1.se.cuhk.edu.hk/ hccl/publications/pub/ICASSP2019-mdd.pdf. Accessed 22 November 2023.
Kun Li, and Helen Meng. “Mispronunciation Detection and Diagnosis in L2 English Speech Using MultiDistribution Deep Neural Networks.” Human-Computer Communications Laboratory Department of System Engineering and Engineering Management The Chinese University of Hong Kong, Hong Kong SAR, China, 16 June 2023, https://www1.se.cuhk.edu.hk/ hccl/publications/pub/2014 PID3298385 LK.pdf. Accessed 22 November 2023.
Yiqing Feng, et al. “SED-MDD: Towards Sentence Dependent End-To-End Mispronunciation Detection and Diagnosis.” 16 June 2023, https://ieeexplore.ieee.org/document/9052975. Accessed 24 November 2023.
Kaiqi Fu. Jones Lin, Dengfeng Ke, Yanlu Xie, Jinsong Zhang, Binghuai Lin “A Full Text-Dependent End to End Mispronunciation Detection and Diagnosis with Easy Data Augmentation Techniques.” arXiv, 17 April 2021, https://arxiv.org/abs/2104.08428. Accessed 24 November 2023.
Wenxuan Ye, et al. “An Approach to Mispronunciation Detection and Diagnosis with Acoustic, Phonetic and Linguistic (APL) Embeddings.” arXiv, 14 October 2021, https://arxiv.org/abs/2110.07274. Accessed 24 November 2023.
Huu Tuong Tu, et al. “Mispronunciation detection and diagnosis model for tonal language, applied to Vietnamese.” [Dublin, Ireland], 20-24 8 2023, https://www.isca-speech.org/archive/pdfs/interspeech2023/huu23 interspeech.pdf.
Thi Thu Trang Nguyen. HMM-based Vietnamese TextTo-Speech: Prosodic Phrasing Modeling, Corpus Design System Design, and Evaluation, 22 January 2016, https://theses.hal.science/tel-01260884/document. Accessed 24 November 2023.
Maxprotect, https://www.maxprotect.com/. Last accessed 21 Feb. 2024
M. Hammami, Y. Chahir and L. Chen, “WebGuard: Web based adult content detection and filtering system," In
Proceedings IEEE/WIC International Conference on Web Intelligence (WI 2003), Halifax, NS, Canada, 2003, pp. 574-578.
Hu, Weiming, Haiqiang Zuo, Ou Wu, Yunfei Chen, Zhongfei Zhang, and David Suter. “Recognition of adult images, videos, and web page bags." ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 7, no. 1 (2011): 1-24.
Sharma, Preeti, Manoj Kumar, and Hitesh Sharma. “Comprehensive analyses of image forgery detection methods from traditional to deep learning approaches: an evaluation." Multimedia Tools and Applications 82, no. 12 (2023): 18117-18150.
Soliman, Mohamed Mostafa, Mohamed Hussein Kamal, Mina Abd El-Massih Nashed, Youssef Mohamed Mostafa, Bassel Safwat Chawky, and Dina Khattab. “Violence recognition from videos using deep learning techniques." In 2019 Ninth International Conference on Intelligent Computing and Information Systems (ICICIS), pp. 80-85. IEEE, 2019.
Human Action Recognition (HAR) Dataset. https://www.kaggle.com/datasets/meetnagadia/humanaction-recognition-har-dataset. Last accessed 21 Feb. 2024
Caruana, Rich. “Multitask learning." Machine learning 28 (1997): 41-75.
M. Ravanelli, T. Parcollet, Y. Bengio, "The PyTorch-Kaldi Speech Recognition Toolkit", https://arxiv.org/abs/1811.07453. Accessed 19 Nov 2018.
