Pre-trained BERT Architecture Analysis for Indonesian Question Answer Model

Authors

  • Sudianto Sudianto Institut Teknologi Telkom Purwokerto

DOI:

https://doi.org/10.37385/jaets.v6i1.4746

Keywords:

BERT, Chatbot, Indonesian, NLP, Transfer Learning, Question Answer

Abstract

Developing a question-and-answer system in Natural Language Processing (NLP) has become a major concern in the Indonesian language context. One of the main challenges in developing a question-and-answer system is the limited dataset, which can cause instability in system performance. The limitations of the dataset make it difficult for the question-and-answer model to understand and answer questions well. The proposed solution uses Transfer Learning with pre-trained models such as BERT. This research aims to analyze the performance of the BERT model, which has been adapted for question-and-answer tasks in Indonesian. The BERT model uses an Indonesian language dataset adapted specifically for question-and-answer tasks. A customization approach tunes BERT parameters according to the given training data. The results obtained; the model is improved by minimizing the loss function. Evaluation of the trained model shows that the best validation loss is 0.00057 after 150 epochs. In addition, through in-depth evaluation of the similarity of question texts, the BERT model can answer questions measurably, according to existing knowledge in the dataset. 

Downloads

Download data is not yet available.

References

Alotaibi, R., Ali, A., Alharthi, H., & Almehamdi, R. (2020). AI Chatbot for Tourist Recommendations: A Case Study in the City of Jeddah, Saudi Arabia. International Journal of Interactive Mobile Technologies (iJIM), 14(19), Article 19. https://doi.org/10.3991/ijim.v14i19.17201

Arnumukti, M. L., Sudianto, S., & Athiyah, U. (2023). Product Layout Recommendations based on Customer Behavior and Data Mining. 2023 IEEE International Conference on Communication, Networks and Satellite (COMNETSAT), 330–334. https://doi.org/10.1109/COMNETSAT59769.2023.10420563

Cahyawijaya, S., Winata, G. I., Wilie, B., Vincentio, K., Li, X., Kuncoro, A., Ruder, S., Lim, Z. Y., Bahar, S., Khodra, M. L., Purwarianti, A., & Fung, P. (2021). IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation.

Clark, J. H., Choi, E., Collins, M., Garrette, D., Kwiatkowski, T., Nikolaev, V., & Palomaki, J. (2020). TYDI QA: A Benchmark for Information-Seeking Question Answering in in Typologically Diverse Languages. Transactions of the Association for Computational Linguistics, 8, 454–470. https://doi.org/10.1162/tacl_a_00317

Cortes, E. G., Woloszyn, V., Barone, D., Möller, S., & Vieira, R. (2022). A systematic review of question answering systems for non-factoid questions. Journal of Intelligent Information Systems, 58(3), 453–480. https://doi.org/10.1007/s10844-021-00655-8

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv:1810.04805). arXiv. https://doi.org/10.48550/arXiv.1810.04805

Dikti, K. M. (2020). Buku Panduan Merdeka Belajar—Kampus Merdeka.

Galassi, A., Lippi, M., & Torroni, P. (2021). Attention in Natural Language Processing. IEEE Transactions on Neural Networks and Learning Systems, 32(10), 4291–4308. https://doi.org/10.1109/TNNLS.2020.3019893

Imamura, K., & Sumita, E. (2019). Recycling a Pre-trained BERT Encoder for Neural Machine Translation. In A. Birch, A. Finch, H. Hayashi, I. Konstas, T. Luong, G. Neubig, Y. Oda, & K. Sudoh (Eds.), Proceedings of the 3rd Workshop on Neural Generation and Translation (pp. 23–31). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-5603

Indah Rahajeng, M., & Purwarianti, A. (2021). Indonesian Question Answering System for Factoid Questions using Face Beauty Products Knowledge Graph. Jurnal Linguistik Komputasional (JLK), 4(2), 59. https://doi.org/10.26418/jlk.v4i2.62

Jhaerol, M. R., & Sudianto, S. (2023). Implementation of Chatbot for Merdeka Belajar Kampus Merdeka Program using Long Short-Term Memory. Jurnal Nasional Pendidikan Teknik Informatika?: JANAPATI, 12(2), Article 2. https://doi.org/10.23887/janapati.v12i2.58794

Koto, F., Rahimi, A., Lau, J. H., & Baldwin, T. (2020). IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP. Proceedings of the 28th International Conference on Computational Linguistics, 757–770. https://doi.org/10.18653/v1/2020.coling-main.66

Kovaleva, O., Romanov, A., Rogers, A., & Rumshisky, A. (2019). Revealing the Dark Secrets of BERT (arXiv:1908.08593). arXiv. http://arxiv.org/abs/1908.08593

Lasama, J., Sudianto, S., Ramadhani, R., Hilmawan, M. D., Aldean, M. Y., & Satria, M. A. H. (2024). English Indonesia-Chan: OPUS-MT Powered Chatbot. Jurnal Teknik Elektro dan Komputasi (ELKOM), 6(1), 105–111. https://doi.org/10.32528/elkom.v6i1.18613

Lee, J.-S., & Hsiang, J. (2020). Patent classification by fine-tuning BERT language model. World Patent Information, 61, 101965. https://doi.org/10.1016/j.wpi.2020.101965

Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, L., & Shazeer, N. (2018). Generating Wikipedia by Summarizing Long Sequences (arXiv:1801.10198). arXiv. https://doi.org/10.48550/arXiv.1801.10198

Lubis, A., & Sumartono, I. (2023). Implementasi Layanan Akademik Berbasis Chatbot untuk Meningkatkan Interaksi Mahasiswa. 3(5).

Nasa-Ngium, P., Nuankaew, W. S., & Nuankaew, P. (2023). Analyzing and Tracking Student Educational Program Interests on Social Media with Chatbots Platform and Text Analytics. International Journal of Interactive Mobile Technologies (iJIM), 17(05), Article 05. https://doi.org/10.3991/ijim.v17i05.31593

Naufal, A. B., Sudianto, S., & Fachri, M. A. A. (2023). Implementation of Chatbot System on Tourism Objects in Banyumas Regency with AIML and Chatterbot. 5.

Nguyen, T. T., Le, A. D., Hoang, H. T., & Nguyen, T. (2021). NEU-chatbot: Chatbot for admission of National Economics University. Computers and Education: Artificial Intelligence, 2. https://doi.org/10.1016/j.caeai.2021.100036

Pratama, T., & Rjito, S. (2021). IndoXLNet: Pre-Trained Language Model for Bahasa Indonesia. International Journal of Engineering Trends and Technology, 70(5), 367–381. https://doi.org/10.14445/22315381/IJETT-V70I5P240

Schwartz, R., Dodge, J., Smith, N. A., & Etzioni, O. (2020). Green AI. Communications of the ACM, 63(12), 54–63. https://doi.org/10.1145/3381831

Sudianto, Herdiyeni, Y., & Prasetyo, L. B. (2023a). Machine learning for sugarcane mapping based on segmentation in cloud platform. AIP Conference Proceedings, 2482(1), 020001. https://doi.org/10.1063/5.0132180

Sudianto, S., Herdiyeni, Y., & Prasetyo, L. B. (2023b). Classification of Sugarcane Area Using Landsat 8 and Random Forest based on Phenology Knowledge. JOIV?: International Journal on Informatics Visualization, 7(3–2), Article 3–2. https://doi.org/10.30630/joiv.7.3-2.1401

Sudianto, S., Herdiyeni, Y., & Prasetyo, L. B. (2023c). Early Warning for Sugarcane Growth using Phenology-Based Remote Sensing by Region. International Journal of Advanced Computer Science and Applications, 14(2). https://doi.org/10.14569/IJACSA.2023.0140259

Wilie, B., Vincentio, K., Winata, G. I., Cahyawijaya, S., Li, X., Lim, Z. Y., Soleman, S., Mahendra, R., Fung, P., Bahar, S., & Purwarianti, A. (2020). IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding.

Downloads

Published

2024-12-15

How to Cite

Sudianto, S. (2024). Pre-trained BERT Architecture Analysis for Indonesian Question Answer Model. Journal of Applied Engineering and Technological Science (JAETS), 6(1), 60–68. https://doi.org/10.37385/jaets.v6i1.4746