An intermediate architecture for secure multimodal search over public web sources and internal scanned documents in Vietnamese organisations
Abstract
Organisations need search over both public web sources and internal repositories, including scanned portable document format (PDF) files that lack native text and must be retrieved under authorisation constraints. This paper specifies an intermediate architecture for secure multimodal search in Vietnamese organisations. The architecture provides a required integration layer between conventional enterprise keyword search and research-grade multimodal retrieval, rather than introducing a new encoder or retrieval model. The contract has four joints: unified public and internal sources, dual text-page representation of scanned archives, access control as a pre-ranking bitset constraint, and auditable hybrid fusion. Elasticsearch is the execution core, not the contribution. Encoder, optical character recognition (OCR), and late-interaction models remain replaceable. The architecture also defines an evaluation protocol covering Precision@k, nDCG@k, MRR, retrieval latency, OCR coverage, and authorisation completeness. As an architecture-focused study, the paper establishes these evaluation dimensions rather than reporting completed experimental results. The proposed design provides a structured basis for implementing secure multimodal retrieval with component replaceability, traceability, and access-control enforcement across heterogeneous information sources.
References
[2] J. Lin, R. Nogueira, A. Yates (2021), “Pretrained transformers for text ranking: BERT and beyond”, Synthesis Lectures on Human Language Technologies, 14(4), pp.1- 325, DOI: 10.2200/S01123ED1V01Y202108HLT053.
[3] J. Wang, X. Yi, R. Guo, et al. (2021), “Milvus: A purpose-built vector data management system”, Proc. SIGMOD, pp.2614-2627, DOI: 10.1145/3448016.3457550.
[4] U. Kruschwitz, C. Hull (2017), “Searching the enterprise”, Foundations and Trends in Information Retrieval, 11(1), pp.1-142, DOI: 10.1561/1500000053.
[5] O. Khattab, M. Zaharia (2020), “ColBERT: Efficient and effective passage search via contextualized late interaction over BERT”, Proc. SIGIR, pp.39-48, DOI: 10.1145/3397271.3401075.
[6] N. Reimers, I. Gurevych (2019), “Sentence-BERT: Sentence embeddings using Siamese BERT-networks”, Proc. EMNLP-IJCNLP, pp.3982-3992, DOI: 10.18653/v1/D19-1410.
[7] J. Johnson, M. Douze, H. Jégou (2021), “Billion-scale similarity search with GPUs”, IEEE Transactions on Big Data, 7(3), pp.535-547, DOI: 10.1109/TBDATA.2019.2921572.
[8] R. Bost, B. Minaud, O. Ohrimenko, et al. (2017), “Forward and backward private searchable symmetric encryption”, Proc. CCS, pp. 1465-1482, DOI: 10.1145/3133956.3134014.
[9] M. Li, T. Lv, J. Chen, et al. (2023), “TrOCR: Transformer-based optical character recognition with pre-trained models”, Proc. AAAI, 37(11), pp.13094-13102, DOI: 10.1609/aaai.v37i11.26538.
[10] D.Q. Nguyen, A.T. Nguyen (2020), “PhoBERT: Pre-trained language models for Vietnamese”, Findings of EMNLP, pp.1037-1042, DOI: 10.18653/v1/2020.findingsemnlp.92.
[11] Y. Xu, M. Li, L. Cui, et al. (2021), “LayoutLMv2: Multi-modal pre-training for visually-rich document understanding”, Proc. ACL, pp.2579-2591, DOI: 10.18653/v1/2021.acl-long.201.
[12] L. Phan, H. Tran, T. Nguyen, et al. (2022), “ViT5: Pretrained text-to-text transformer for Vietnamese language generation”, Proc. NAACL-HLT, pp.136-147, DOI: 10.18653/v1/2022.naacl-main.10.
[13] V. Karpukhin, B. Oguz, S. Min, et al. (2020), “Dense passage retrieval for open-domain question answering”, Proc. EMNLP, pp.6769-6781, DOI: 10.18653/v1/2020.emnlp-main.550.
[14] X. Zhang, G. Flumeri, A. Vozzi, et al. (2026), “Multimodal assessment of mental states and visual search for a user-centred design of semantic web platforms”, Applied Sciences, 16(10), DOI: 10.3390/app16104756.
[15] A. Radford, J.W. Kim, C. Hallacy, et al. (2021), “Learning transferable visual models from natural language supervision”, Proc. ICML, pp.8748-8763, DOI: 10.48550/arXiv.2103.00020.
[16] Y. Huang, T. Lv, L. Cui, et al. (2022), “LayoutLMv3: Pre-training for document AI with unified text and image masking”, Proc. ACM Multimedia, pp.4083- 4091, DOI: 10.1145/3503161.3548112.
[17] G. Kim, T. Hong, M. Yim, et al. (2022), “OCR-free document understanding multimodal transformer”, Proc. ECCV, pp.498-517, DOI: 10.1007/978-3-031-19815- 1_29. [18] S. Appalaraju, B. Jasani, B.U. Singh, et al. (2021), “DocFormer: End-to-end transformer for document understanding”, Proc. ICCV, pp.993-1003, DOI: 10.1109/ICCV48922.2021.00104.
[19] S. Robertson, H. Zaragoza (2009), “The probabilistic relevance framework: BM25 and beyond”, Foundations and Trends in Information Retrieval, 3(4), pp.333-389, DOI: 10.1561/1500000019.
[20] C.D. Manning, P. Raghavan, H. Schütze (2008), Introduction to Information Retrieval, Cambridge University Press, 506pp, DOI: 10.1017/CBO9780511809071.
[21] J. Chen, S. Xiao, P. Zhang, et al. (2024), “BGE M3-Embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation”, arXiv preprint, DOI: 10.48550/arXiv.2402.03216.
[22] M. Faysse, H. Sibille, T. Wu, et al. (2025), “ColPali: Efficient document retrieval with vision language models”, Proc. ICLR, DOI: 10.48550/arXiv.2407.01449.
[23] A. Radford, J.W. Kim, T. Xu, et al. (2023), “Robust speech recognition via large-scale weak supervision”, Proc. ICML, 202, pp.28492-28518.
[24] C. Büttcher, C.L.A. Clarke, G.V. Cormack (2010), Information Retrieval: Implementing and Evaluating Search Engines, MIT Press, 632pp.
[25] Y. Luan, J. Eisenstein, K. Toutanova, et al. (2021), “Sparse, dense, and attentional representations for text retrieval”, Transactions of The Association for Computational Linguistics, 9, pp.329-345, DOI: 10.1162/tacl_a_00369.
[26] Elastic (2026), “Hybrid search”, Elasticsearch Documentation, https://www.elastic.co/docs/solutions/search/hybrid-search, accessed 28 September 2026.
[27] Elastic (2026), “Reciprocal rank fusion”, Elasticsearch Reference, https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion, accessed 28 September 2026.
