Tag infilling for Vietnamese administrative forms: Leveraging synthetic datasets and domain-specific modelling
Keywords:
administrative documents, large language model, masked language model, tag infilling, text infilling
Abstract
Automatic identification of administrative tags is vital for automating administrative form processing. Despite its importance, this task remains underexplored in low-resource languages and domain-specific settings, especially in the context of Vietnamese government documents. In this article, we introduce a comprehensive and scalable pipeline for assisting constrained administrative tag infilling in Vietnamese administrative forms, comprising synthetic dataset construction, domain-adapted model training, and robust evaluation strategies. This forms a complete pipeline that predicts the appropriate administrative tag from a predefined tag set for a variety of forms generated from authentic Vietnamese administrative documents. Using a large-scale dataset of 50,000 synthetic Vietnamese administrative forms annotated with 425 distinct tags and structured into five levels of complexity, our experiments show that domain-adapted Transformer-based models consistently outperform general-purpose zero-shot large language models on seen data and generalise well to unseen inputs. This highlights the potential of our approach to achieve high accuracy with substantially smaller models, offering a more practical and efficient solution for processing Vietnamese administrative forms.
References
[1] A.R. Salama, Ö. Alaçam, W. Menzel (2018), “Text completion using context-integrated dependency parsing”, Proceedings of The Third Workshop on Representation Learning for NLP, pp.41-49, DOI: 10.18653/v1/W18-3005.
[2] P. Mirowski, A. Vlachos (2015), “Dependency recurrent neural language models for sentence completion”, Proceedings of The 53rd Annual Meeting of The Association for Computational Linguistics and The 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp.511-517, DOI: 10.3115/v1/P15-2084.
[3] M. Zhang, M. Huang, R. Shi, et al. (2024), “Calibrating the confidence of large language models by eliciting fidelity”, Proceedings of The 2024 Conference on Empirical Methods in Natural Language Processing, pp.2959-2979, DOI: 10.18653/v1/2024.emnlp-main.173.
[4] J. Leng, C. Huang, B. Zhu, et al. (2025), “Taming overconfidence in LLMs: Reward calibration in RLHF”, 13th International Conference on Learning Representations (ICLR 2025), https://proceedings.iclr.cc/paper_files/paper/2025/file/29fb6e1456b3d8b57ede5c45aa2c6537-Paper-Conference.pdf, accessed 20 September 2026.
[5] C. Xu, B. Wen, B. Han, et al. (2025), “Do language models mirror human confidence? Exploring psychological insights to address overconfidence in LLMs”, Findings of The Association for Computational Linguistics: ACL 2025, pp.25655-25672, DOI: 10.18653/v1/2025.findings-acl.1316.
[6] A. Hamdi, M. Coustaty, A. Joseph, et al. (2018), “Feature selection for document flow segmentation”, 2018 13th IAPR International Workshop on Document Analysis Systems (DAS), pp.245-250, DOI: 10.1109/DAS.2018.66.
[7] S. Biswas, P. Riba, J. Lladós, et al. (2021), “Graph-based deep generative modelling for document layout generation”, Document Analysis and Recognition - ICDAR 2021 Workshops, pp.525-537, DOI: 10.1007/978-3-030-86159-9_38.
[8] X. Ling, M. Gao, D. Wang (2020), “Intelligent document processing based on RPA and machine learning”, 2020 Chinese Automation Congress (CAC), pp.1349-1353, DOI: 10.1109/CAC51589.2020.9326579.
[9] Z.A. Nazi, M.R. Hossain, F.A. Mamun (2025), “Evaluation of open and closed-source LLMs for low-resource language with zero-shot, few-shot, and chain-of-thought prompting”, Natural Language Processing Journal, 10, DOI: 10.1016/j.nlp.2024.100124.
[10] H. Gurdil, Y.B. Soguksu, S. Salihoglu, et al. (2025), “Integration of artificial intelligence in educational measurement: Efficacy of ChatGPT in data generation within the scope of item response theory”, Trakya Eğitim Dergisi, DOI:10.24315/tred.1509299 (in Turkish).
[11] H. Tari, M.D. Khan, J. Rutten, et al. (2024), “Leveraging GPT for the generation of multi-platform social media datasets for research”, HT’24: Proceedings of The 35th ACM Conference on Hypertext and Social Media, pp.337-343, DOI: 10.1145/3648188.3675153.
[12] S.A. Lehr, A. Caliskan, S. Liyanage, et al. (2024), “ChatGPT as research scientist: Probing GPT’s capabilities as a research librarian, research ethicist, data generator, and data predictor”, Proceedings of The National Academy of Sciences, 121(35), DOI: 10.1073/pnas.2404328121.
[13] K. Song, X. Tan, T. Qin, et al. (2019), “MASS: Masked sequence to sequence pre-training for language generation”, Proceedings of The 36th International Conference on Machine Learning (ICML), PMLR 97, pp.5926-5936.
[14] M. Lewis, Y. Liu, N. Goyal, et al. (2020), “BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension”, Proceedings of The 58th Annual Meeting of The Association for Computational Linguistics, pp.7871-7880, DOI: 10.18653/v1/2020.acl-main.703.
[15] W. Zhu, Z. Hu, E. Xing (2019), “Text infilling”, ArXiv Preprint, DOI: 10.48550/arXiv.1901.00158.
[16] C. Donahue, M. Lee, P. Liang (2020), “Enabling language models to fill in the blanks”, Proceedings of The 58th Annual Meeting of The Association for Computational Linguistics, pp.2492-2501, DOI: 10.18653/v1/2020.acl-main.225.
[17] D. Li, J. You, K. Funakoshi, et al. (2022), “A-TIP: Attribute-aware text infilling via pre-trained language model”, Proceedings of The 29th International Conference on Computational Linguistics, pp.5857-5869.
[18] H.N. Duc (2003), Vietnamese Dictionary, https://github.com/undertheseanlp/dictionary/tree/hongocduc (in Vietnamese).
[19] T. Shin, Y. Razeghi, R.L. Logan IV, et al. (2020), “AutoPrompt: Eliciting knowledge from language models with automatically generated prompts”, Proceedings of The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.4222-4235, DOI: 10.18653/v1/2020.emnlp-main.346.
[20] V. Sanh, L. Debut, J. Chaumond, et al. (2019), “DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter”, 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing @ NeurIPS 2019.
[21] H.T. Nguyen, M.K. Phi, X.B. Ngo, et al. (2024), “Attentive deep neural networks for legal document retrieval”, Artificial Intelligence and Law, 32(1), pp.57-86, DOI: 10.1007/s10506-022-09341-8.
[2] P. Mirowski, A. Vlachos (2015), “Dependency recurrent neural language models for sentence completion”, Proceedings of The 53rd Annual Meeting of The Association for Computational Linguistics and The 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp.511-517, DOI: 10.3115/v1/P15-2084.
[3] M. Zhang, M. Huang, R. Shi, et al. (2024), “Calibrating the confidence of large language models by eliciting fidelity”, Proceedings of The 2024 Conference on Empirical Methods in Natural Language Processing, pp.2959-2979, DOI: 10.18653/v1/2024.emnlp-main.173.
[4] J. Leng, C. Huang, B. Zhu, et al. (2025), “Taming overconfidence in LLMs: Reward calibration in RLHF”, 13th International Conference on Learning Representations (ICLR 2025), https://proceedings.iclr.cc/paper_files/paper/2025/file/29fb6e1456b3d8b57ede5c45aa2c6537-Paper-Conference.pdf, accessed 20 September 2026.
[5] C. Xu, B. Wen, B. Han, et al. (2025), “Do language models mirror human confidence? Exploring psychological insights to address overconfidence in LLMs”, Findings of The Association for Computational Linguistics: ACL 2025, pp.25655-25672, DOI: 10.18653/v1/2025.findings-acl.1316.
[6] A. Hamdi, M. Coustaty, A. Joseph, et al. (2018), “Feature selection for document flow segmentation”, 2018 13th IAPR International Workshop on Document Analysis Systems (DAS), pp.245-250, DOI: 10.1109/DAS.2018.66.
[7] S. Biswas, P. Riba, J. Lladós, et al. (2021), “Graph-based deep generative modelling for document layout generation”, Document Analysis and Recognition - ICDAR 2021 Workshops, pp.525-537, DOI: 10.1007/978-3-030-86159-9_38.
[8] X. Ling, M. Gao, D. Wang (2020), “Intelligent document processing based on RPA and machine learning”, 2020 Chinese Automation Congress (CAC), pp.1349-1353, DOI: 10.1109/CAC51589.2020.9326579.
[9] Z.A. Nazi, M.R. Hossain, F.A. Mamun (2025), “Evaluation of open and closed-source LLMs for low-resource language with zero-shot, few-shot, and chain-of-thought prompting”, Natural Language Processing Journal, 10, DOI: 10.1016/j.nlp.2024.100124.
[10] H. Gurdil, Y.B. Soguksu, S. Salihoglu, et al. (2025), “Integration of artificial intelligence in educational measurement: Efficacy of ChatGPT in data generation within the scope of item response theory”, Trakya Eğitim Dergisi, DOI:10.24315/tred.1509299 (in Turkish).
[11] H. Tari, M.D. Khan, J. Rutten, et al. (2024), “Leveraging GPT for the generation of multi-platform social media datasets for research”, HT’24: Proceedings of The 35th ACM Conference on Hypertext and Social Media, pp.337-343, DOI: 10.1145/3648188.3675153.
[12] S.A. Lehr, A. Caliskan, S. Liyanage, et al. (2024), “ChatGPT as research scientist: Probing GPT’s capabilities as a research librarian, research ethicist, data generator, and data predictor”, Proceedings of The National Academy of Sciences, 121(35), DOI: 10.1073/pnas.2404328121.
[13] K. Song, X. Tan, T. Qin, et al. (2019), “MASS: Masked sequence to sequence pre-training for language generation”, Proceedings of The 36th International Conference on Machine Learning (ICML), PMLR 97, pp.5926-5936.
[14] M. Lewis, Y. Liu, N. Goyal, et al. (2020), “BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension”, Proceedings of The 58th Annual Meeting of The Association for Computational Linguistics, pp.7871-7880, DOI: 10.18653/v1/2020.acl-main.703.
[15] W. Zhu, Z. Hu, E. Xing (2019), “Text infilling”, ArXiv Preprint, DOI: 10.48550/arXiv.1901.00158.
[16] C. Donahue, M. Lee, P. Liang (2020), “Enabling language models to fill in the blanks”, Proceedings of The 58th Annual Meeting of The Association for Computational Linguistics, pp.2492-2501, DOI: 10.18653/v1/2020.acl-main.225.
[17] D. Li, J. You, K. Funakoshi, et al. (2022), “A-TIP: Attribute-aware text infilling via pre-trained language model”, Proceedings of The 29th International Conference on Computational Linguistics, pp.5857-5869.
[18] H.N. Duc (2003), Vietnamese Dictionary, https://github.com/undertheseanlp/dictionary/tree/hongocduc (in Vietnamese).
[19] T. Shin, Y. Razeghi, R.L. Logan IV, et al. (2020), “AutoPrompt: Eliciting knowledge from language models with automatically generated prompts”, Proceedings of The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.4222-4235, DOI: 10.18653/v1/2020.emnlp-main.346.
[20] V. Sanh, L. Debut, J. Chaumond, et al. (2019), “DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter”, 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing @ NeurIPS 2019.
[21] H.T. Nguyen, M.K. Phi, X.B. Ngo, et al. (2024), “Attentive deep neural networks for legal document retrieval”, Artificial Intelligence and Law, 32(1), pp.57-86, DOI: 10.1007/s10506-022-09341-8.
Published
2026-11-02
Section
Regular Articles
